When finding out FreeBSD's network tuning, any articles say “Optimise the random harvest.” I wondered why random number is related even though I looked at the network, but FreeBSD Handbook made sense to me: random(4)とrandom_harvest(9), random_harvest(9). This information is as of FreeBSD 12.1-RELEASE.

First about the random harvest.

FreeBSD has a special file /dev/random to get random numbers. It is normally PRN (Pseudo Random Number) generator, thus it is nothing but return definite numbers generated by formula which looks like random. Maintaining randomness is important to keep good entropy state. FreeBSD seems to call this mechanism “random harvest.” I think this naming pretty fits in the behaviour which growing good entropy and using it.

The entropy source is controlled by the kern.random.harvest.mask kernel variable.

This value is bit field which each bit corresponds to a source, “1” means the source is enabled, “0” means disabled. Aliased “mask” variables, “mask_symbolic” and “mask_bin” should be useful rather than unfriendly decimal value. Their values are below on my PC:

$ sysctl kern.random
kern.random.fortuna.minpoolsize: 64
kern.random.harvest.mask_symbolic: PURE_RDRAND,[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
kern.random.harvest.mask_bin: 00000010000000111111111
kern.random.harvest.mask: 66047
kern.random.random_sources: 'Intel Secure Key RNG'

As you see, “mask_bin” is binary representation, and “mask_symbolic” is more human readable representation of the mask value which contains all available entropy sources on the system. The source enclosed in square brackets means disabled source. As of FreeBSD 12.1-RELEASE, 24 entropy sources are defined at sys/sys/random.h.

In my case, NET_ETHER is used as one of entropy sources. They say the reason why it affects network performance is because of race conditions on a lock between the harvesting and network communicating. The affect is said not to be negligible especially on high speed network like over 10Gb/s. I see how it is.

• random(4)
• random_harvest(9)
• Tuning FreeBSD for routing and firewalling(PDF)

The X10DRi which is a one of Supermicro's mobo seems to be able to boot from NVMe devices.

You may think “why now do you care that?”, but the mobo has no M.2 slots, and I haven't seen the official specific announcement mentioned to it. Moreover, I couldn't understand the capability at all when I'd read their FAQs because there are some confusing answers about their similar products.

• Q: Will X10DRI-T support PCI-E NVMe boot? I am using Samsung 970 pro.
  • A: No.
• Q: Does the X10DAi motherboard support to boot from an M.2 device installed on an AOC-SLG3-2M2?
  • A: Not available. Needs OptionROM.
• Q: Can a PCI-E NVMe add on card which does not have onboard Option ROM be used as a boot devices on the X10DAi?
  • A: No. Needs OptionROM.
• Q: Does X10DAi support NVMe PCIe SSD as a boot device?
  • A: Yes. Bootable with UEFI mode.
    • NOTE: As it is “No.” The answer implies the mobo needs OptionROM.

Like these. Well, it is safe to say it can boot from NVMe with a Option ROM.

I assumed the X10DRi can do it natively because it is for enterprise use which is produced by established Supermicro, so I had bought U.2 SSD Samsung PM983 without thinking too much days ago. Be that as it may, according to later survey, I found it seemed to install the Option ROM into PM983 using Samsung DC Toolkit, or in the worst case, I thought it would work itself out somehow with Clover EFI bootloader.

That's way I installed the SSD into the mobo, then it could normally work out without any problems.

I tried to disable its Option ROM, but it is still recognised in BIOS, and the machine can boot from it, so the mobo natively supports the NVMe boot in UEFI mode, I guess.

This result is under condition as BIOS revision 3.1b. My apologies if it doesn't work.

Samsung MZQLB960HAJR-00007 came up across the water. It is one of the PM983 series SSD for enterprise use that has U.2 connector and 960GB capacity. Now I haven't been satisfied with present SATA SSD on my server because I upgraded my LAN to 40GbE, so I couldn't help but get it.

Actually recent consumer M.2 SSDs are faster than the PM983 in term of simply comparison, but I wanted a reliability-oriented product to protect my data or avoid unreasonable troubles. Then, I had found seller tested used one approximate at 10000 yen on ebay, so I had to order it. I could also get a brand-new PM963 or PM953 series at the same price, but their performances are distinct different with PM983. I think it was pretty good deal. The specs is bellow:

• Sequential access speed
  • Read 3000MB/s
  • Write 1050MB/s
• Random access speed
  • Read 400000 IOPS
  • Write 40000 IOPS
• 1.3 DWPD (TBW=1366TB)
• TLC (3bit MLC VNAND)

I'd ordered the SSDs on 2nd September, and received them on 14th. They looked good. Their written bytes were less than 1TB. I can say the overall quality is good.

1GB		AS SSD Benchmark 2.0.6485.19676 ------------------------------ Name: SAMSUNG MZQLB960HAJR-00007 Firmware: EDA5202Q Controller: stornvme Offset: 16384 K - OK Size: 894.25 GB Date: 2019/09/15 22:03:59 Note: MZQLB960HAJR-00007 ------------------------------ Sequential: ------------------------------ Read: 2569.63 MB/s Write: 1294.86 MB/s ------------------------------ 4K: ------------------------------ Read: 41.92 MB/s Write: 113.66 MB/s ------------------------------ 4K-64Threads: ------------------------------ Read: 1365.31 MB/s Write: 1254.64 MB/s ------------------------------ Access Times: ------------------------------ Read: 0.036 ms Write: 0.034 ms ------------------------------ Score: ------------------------------ Read: 1664 Write: 1498 Total: 3951 ------------------------------	AS SSD Benchmark 2.0.6485.19676 ------------------------------ Name: SAMSUNG MZQLB960HAJR-00007 Firmware: EDA5202Q Controller: stornvme Offset: 16384 K - OK Size: 894.25 GB Date: 2019/09/15 22:04:22 Note: MZQLB960HAJR-00007 ------------------------------ 16MB ------------------------------ Read: 160.60 iops Write: 80.93 iops ------------------------------ 4K: ------------------------------ Read: 10731 iops Write: 29098 iops ------------------------------ 4K-64Threads: ------------------------------ Read: 349520 iops Write: 321189 iops ------------------------------ 512B ------------------------------ Read: 27858 iops Write: 29645 iops ------------------------------ Score: ------------------------------ Read: 1664 Write: 1498 Total: 3951 ------------------------------
10GB		AS SSD Benchmark 2.0.6485.19676 ------------------------------ Name: SAMSUNG MZQLB960HAJR-00007 Firmware: EDA5202Q Controller: stornvme Offset: 16384 K - OK Size: 894.25 GB Date: 2019/09/15 22:04:08 Note: MZQLB960HAJR-00007 ------------------------------ Sequential: ------------------------------ Read: 2857.39 MB/s Write: 1275.96 MB/s ------------------------------ 4K: ------------------------------ Read: 41.43 MB/s Write: 114.74 MB/s ------------------------------ 4K-64Threads: ------------------------------ Read: 955.91 MB/s Write: 687.14 MB/s ------------------------------ Access Times: ------------------------------ Read: 0.038 ms Write: 0.344 ms ------------------------------ Score: ------------------------------ Read: 1283 Write: 929 Total: 2796 ------------------------------	AS SSD Benchmark 2.0.6485.19676 ------------------------------ Name: SAMSUNG MZQLB960HAJR-00007 Firmware: EDA5202Q Controller: stornvme Offset: 16384 K - OK Size: 894.25 GB Date: 2019/09/15 22:04:43 Note: MZQLB960HAJR-00007 ------------------------------ 16MB ------------------------------ Read: 178.59 iops Write: 79.75 iops ------------------------------ 4K: ------------------------------ Read: 10607 iops Write: 29373 iops ------------------------------ 4K-64Threads: ------------------------------ Read: 244714 iops Write: 175909 iops ------------------------------ 512B ------------------------------ Read: 26622 iops Write: 2910 iops ------------------------------ Score: ------------------------------ Read: 1283 Write: 929 Total: 2796 ------------------------------

A cache capacity on the SSD is unknown.

Its speed was stable in 1GB test as far as I tested it several times with CrystalDiskMark. However, in 32GB test, there were variation result particularly on write speed. It was because of a cache overflow, I guess. It slowed down to 60MB/s in the worst case of a 4K1T writing test.

For your information, a result the SSD connected with PCIe 2.0x4 is below:

Initially I didn't know the reason why the SSD was slow because I didn't notice it was connected with that.

MikroTik CRS326-24S+2Q+RM just arrived at home! It is the world cheapest 40GbE capable Ethernet switch out of legally distributed ones as of 9/4/2019, I guess. That's our MikroTik!　But then, I got their product for the first time.

I suppose the guys come to see this page are aware of it, but CRS326-24S+2Q+RM has 10GbE capable 24 SFP+ ports and additionally two 40GbE QSFP+ ports, and its price is only 499 USD. What a crazy stuff it is! Though the price as worth is pretty expensive if I though with a calm mind, it is a unprecedented price-slashing considering its performance. Moreover, it is sold at 384 USD in EuroDK. I can't believe it.

That's why I couldn't help buying it. I had ordered it on 22nd August (JST), then it delivered on 26th quickly like 40GbE, but I couldn't receive it until the weekend. A total costs was approximate 50,000 yen including shipping, taxes and etc.

I tentatively test a 40GbE connection below environment in figure because of some reasons. It marked 28Gbps by iperf3 with MTU=1500.

It seems a switching chip is Marvell 98DX8332. This switch might also support 2.5GbE and 5GbE although MikroTik doesn't clearly say, because the chip support them and there are their check boxes on the switch's config screen…

The switch supports dual boot of RouterOS and SwOS, so it can do L3 processing to some extent. Having said that, the processes which aren't handled by the chip are processed by CPU, and what is worse, an internal bandwidth between them is only 1Gbps. Therefore, we had better use it as plain L2 switch, or should keep lightweight L3 usage such as connecting to WAN with simple packet filtering. Otherwise its valuable performance will be completely dead. Fortunately it has enough 10GbE ports, so I think fit to pass on the L3 processes to a dedicated router with a LAG. AT-x510-28GTX which is a L3 switch I got some time ago will have tears of joy for this application.

Following table shows rough power consumptions:

Cooling fans were pretty noisy in factory default RouterOS version. I updated it to version 6.45.4, now they get quieter than before. The OS aggressively controls their speed between zero to full blast depending on the internal temperature. I think these specs are very kindly for common home use. Thank you, MikroTik. Well, when the fans is running, they cause a certain level of noise. It seems themselves' operating noise is louder rather than wind noise. It is acceptable for me, but not for someone, I guess.

It seems to fail to install Visual Studio 2013 in case Windows' system font is changed to “Noto Sans CJK.”

I ran vs_community.exe which is an installer of VS2013, then “Visual Studio” logo appeared on screen. After thant, the installer window showed, but it was black window and disappeared on its own about a minute later.

Once I tried to use an compatibility option on the installer's property, the window showed correctly, but I still couldn't install it because of the compatibility engine blah-blah-blah. Well, that is a well-known problem, in a manner, that is nomarl behaviour.

I checked an installer log file named “dd_vs_community_datetime.log” in %TEMP% directory. I found a weird message:
「[4F88:2824][2019-08-17T11:40:29]e000: MUX: ERROR: 'file:///C:/WINDOWS/FONTS/NotoSansCJKjp-Regular.otf' ファイルは、予測されるファイル形式の仕様に準拠していません。」

Its whole sentence is below. The installer obviously crashed at instantiating the font.

[4F88:2824][2019-08-17T11:40:29]e000: MUX:  ERROR: 'file:///C:/WINDOWS/FONTS/NotoSansCJKjp-Regular.otf' ファイルは、予測されるファイル形式の仕様に準拠していません。
[4F88:2824][2019-08-17T11:40:29]e000: MUX:  Stack:    場所 Adobe.CffRasterizer.OTFRasterizer.MapErrorCode(AdobeErrorCode error)
   場所 Adobe.CffRasterizer.OTFRasterizer.NewFont(UnmanagedMemoryStream fontFileStream, Uri sourceUri, Int32 faceIndex)
   場所 MS.Internal.FontFace.TrueTypeFontDriver.ReadCFFMetrics(FontFaceLayoutInfo cache, Boolean vmtxPresent, UInt16 typoAscent, UInt16 typoDescent)
   場所 MS.Internal.FontFace.TrueTypeFontDriver.ReadGlyfMetrics(FontFaceLayoutInfo cache, UInt16 indexToLocFormat, Boolean vmtxPresent, UInt16 typoAscent, UInt16 typoDescent)
   場所 MS.Internal.FontFace.TrueTypeFontDriver.ReadAdvances(FontFaceLayoutInfo cache, CheckedPointer hmtxTable, UInt16 numberOfMetrics, UInt16 indexToLocFormat, UInt16 typoAscent, UInt16 typoDescent)
   場所 MS.Internal.FontFace.TrueTypeFontDriver.GetLayoutFontFaceInfo(FontFaceLayoutInfo cache)
   場所 MS.Internal.FontCache.FontFaceLayoutInfo.MS.Internal.FontCache.IFontCacheElement.AddToCache(CheckedPointer newPointer, ElementCacher cacher)
   場所 MS.Internal.FontCache.HashTable.Lookup(IFontCacheElement e, Boolean add)
   場所 MS.Internal.FontCache.CacheManager.Lookup(IFontCacheElement e)
   場所 System.Windows.Media.GlyphTypeface.Initialize(Uri typefaceSource, StyleSimulations styleSimulations, Boolean fromPublic)
   場所 MS.Internal.FontCache.CachedFontFace.CreateGlyphTypeface()
   場所 MS.Internal.FontFace.PhysicalFontFamily.GetGlyphTypeface(FontStyle style, FontWeight weight, FontStretch stretch)
   場所 MS.Internal.FontFace.PhysicalFontFamily.MS.Internal.FontFace.IFontFamily.GetTypefaceMetrics(FontStyle style, FontWeight weight, FontStretch stretch)
   場所 System.Windows.Media.Typeface.ConstructCachedTypeface()
   場所 System.Windows.Media.Typeface.get_CachedTypeface()
   場所 System.Windows.Media.Typeface.CheckFastPathNominalGlyphs(CharacterBufferRange charBufferRange, Double emSize, Double widthMax, Boolean keepAWord, Boolean numberSubstitution, CultureInfo cultureInfo, Int32& stringLengthFit)
   場所 MS.Internal.TextFormatting.SimpleRun.CreateSimpleTextRun(CharacterBufferRange charBufferRange, TextRun textRun, TextFormatterImp formatter, Int32 widthLeft, Boolean emergencyWrap)
   場所 MS.Internal.TextFormatting.SimpleRun.Create(FormatSettings settings, CharacterBufferRange charString, TextRun textRun, Int32 runLength, Int32 widthLeft)
   場所 MS.Internal.TextFormatting.SimpleRun.Create(FormatSettings settings, Int32 cp, Int32 cpFirst, Int32 widthLeft, Int32 widthMax)
   場所 MS.Internal.TextFormatting.SimpleTextLine.Create(FormatSettings settings, Int32 cpFirst, Int32 paragraphWidth)
   場所 MS.Internal.TextFormatting.TextFormatterImp.FormatLineInternal(TextSource textSource, Int32 firstCharIndex, Int32 lineLength, Double paragraphWidth, TextParagraphProperties paragraphProperties, TextLineBreak previousLineBreak, TextRunCache textRunCache)
   場所 MS.Internal.TextFormatting.TextFormatterImp.FormatLine(TextSource textSource, Int32 firstCharIndex, Double paragraphWidth, TextParagraphProperties paragraphProperties, TextLineBreak previousLineBreak, TextRunCache textRunCache)
   場所 System.Windows.Controls.TextBlock.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 System.Windows.Controls.StackPanel.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 System.Windows.Controls.Grid.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 System.Windows.Controls.Grid.MeasureCell(Int32 cell, Boolean forceInfinityV)
   場所 System.Windows.Controls.Grid.MeasureCellsGroup(Int32 cellsHead, Size referenceSize, Boolean ignoreDesiredSizeU, Boolean forceInfinityV)
   場所 System.Windows.Controls.Grid.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 System.Windows.Controls.Grid.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 MS.Internal.Helper.MeasureElementWithSingleChild(UIElement element, Size constraint)
   場所 System.Windows.Controls.ContentPresenter.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 System.Windows.Controls.Border.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 System.Windows.Controls.Control.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 System.Windows.Controls.Decorator.MeasureOverride(Size constraint)
   場所 System.Windows.Documents.AdornerDecorator.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 System.Windows.Controls.Grid.MeasureOverride(Size constraint)
   場所 System.Windows.FrameworkElement.MeasureCore(Size availableSize)
   場所 System.Windows.UIElement.Measure(Size availableSize)
   場所 System.Windows.ContextLayoutManager.UpdateLayout()
   場所 System.Windows.ContextLayoutManager.UpdateLayoutCallback(Object arg)
   場所 System.Windows.Media.MediaContext.InvokeOnRenderCallback.DoWork()
   場所 System.Windows.Media.MediaContext.FireInvokeOnRenderCallbacks()
   場所 System.Windows.Media.MediaContext.RenderMessageHandlerCore(Object resizedCompositionTarget)
   場所 System.Windows.Media.MediaContext.RenderMessageHandler(Object resizedCompositionTarget)
   場所 System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Boolean isSingleParameter)
   場所 System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Boolean isSingleParameter, Delegate catchHandler)
   場所 System.Windows.Threading.Dispatcher.WrappedInvoke(Delegate callback, Object args, Boolean isSingleParameter, Delegate catchHandler)
   場所 System.Windows.Threading.DispatcherOperation.InvokeImpl()
   場所 System.Windows.Threading.DispatcherOperation.InvokeInSecurityContext(Object state)
   場所 System.Threading.ExecutionContext.runTryCode(Object userData)
   場所 System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   場所 System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   場所 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   場所 System.Windows.Threading.DispatcherOperation.Invoke()
   場所 System.Windows.Threading.Dispatcher.ProcessQueue()
   場所 System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   場所 MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   場所 MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   場所 System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Boolean isSingleParameter)
   場所 System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Boolean isSingleParameter, Delegate catchHandler)
   場所 System.Windows.Threading.Dispatcher.WrappedInvoke(Delegate callback, Object args, Boolean isSingleParameter, Delegate catchHandler)
   場所 System.Windows.Threading.Dispatcher.InvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Boolean isSingleParameter)
   場所 System.Windows.Threading.Dispatcher.Invoke(DispatcherPriority priority, Delegate method, Object arg)
   場所 MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
   場所 MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
   場所 System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
   場所 System.Windows.Threading.Dispatcher.PushFrame(DispatcherFrame frame)
   場所 System.Windows.Threading.Dispatcher.Run()
   場所 Microsoft.Devdiv.Bootstrapper.ManagedUx.RunUI(ViewModelCommonUi viewModel)
   場所 Microsoft.Devdiv.Bootstrapper.ManagedUx.InternalRun()
   場所 Microsoft.Devdiv.Bootstrapper.ManagedUx.Run()
   場所 System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   場所 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   場所 System.Threading.ThreadHelper.ThreadStart()

I got Windows' font back to default one, then the installer ran correctly although I didn't expect that.

I guess the problem may be caused by OTF font rather than Noto font. Be that as it may, I'm relieved to succeed the install.