FreeBSDのnvmecontrolコマンドでNVMeデバイスをフォーマットしてみた記録。
別マシンでは読み込み3GB/sほど出るSSDがFreeBSDマシンでは1.6GB/sほどしか出ず、初期化したら速度回復すんじゃね？という安直な気持ちでformatしてみたら、色々と興味深かったのでメモがてら残しておく。
なお、初期化してみたけど速度は回復しなかった。というか、1.6GB/sはPCIe 2.0×4接続時の速度上限っぽい雰囲気なんだけど、間違いなくPCIe 3.0×4接続になってるし、PCIeスロット変えても変化無いしだし謎。
（2019-10-06 追記）
その後、どう足掻いても1.6GB/sを超えなかった方は、実際に使いだしたら2GB/s超えるようになったり、逆に元から3GB/s出てた方は遅くなったりして、NVMeなんもわからん。そもそもnvmecontrolのベンチマーク機能は、特定の領域をひたすらアクセスするだけの原始的なものらしく、結果に一喜一憂しても仕方ないと思い至った。参考値として見るに留めるのが吉。
> sudo smartctl -a /dev/nvme1 smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.0-RELEASE-p4 amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: SAMSUNG MZQLB960HAJR-00007 Serial Number: S437NY0KC00XYZ Firmware Version: EDA5202Q PCI Vendor/Subsystem ID: 0x144d IEEE OUI Identifier: 0x002538 Total NVM Capacity: 960,197,124,096 [960 GB] Unallocated NVM Capacity: 0 Controller ID: 4 Number of Namespaces: 1 Namespace 1 Size/Capacity: 960,197,124,096 [960 GB] Namespace 1 Utilization: 1,232,384,000 [1.23 GB] Namespace 1 Formatted LBA Size: 512 Local Time is: Thu Sep 19 00:38:43 2019 JST Firmware Updates (0x17): 3 Slots, Slot 1 R/O, no Reset required Optional Admin Commands (0x000f): Security Format Frmw_DL NS_Mngmt Optional NVM Commands (0x001f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Maximum Data Transfer Size: 512 Pages Warning Comp. Temp. Threshold: 87 Celsius Critical Comp. Temp. Threshold: 88 Celsius Namespace 1 Features (0x02): NA_Fields Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 10.60W - - 0 0 0 0 0 0 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 0 1 - 4096 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 41 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 36,935,426 [18.9 TB] Data Units Written: 4,227,826 [2.16 TB] Host Read Commands: 2,288,568,067 Host Write Commands: 96,108,724 Controller Busy Time: 241 Power Cycles: 417 Power On Hours: 2,260 Unsafe Shutdowns: 395 Media and Data Integrity Errors: 0 Error Information Log Entries: 11 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 41 Celsius Temperature Sensor 2: 48 Celsius Temperature Sensor 3: 57 Celsius Error Information (NVMe Log 0x01, max 64 entries) No Errors Logged
今回初めて知ったんだけど、NVMeではnamespaceなる仕組みでデバイスを論理的に分割できるそうだ。要はパーティションのようなものだが、namespace毎に論理セクタサイズを変えたりできるらしい。
S.M.A.R.Tの中で気になったのは「Namespace 1 Utilization」という項目。名前の通り、当該namespaceの使用量を表している。デバイス側で使用量を持ってる事に驚いた。ファイルシステム側のファイル作成・削除と連動するのかは不明。なんとなく連動はしてない気はするが、これだけ高機能ならもしかしてもしかするのかも？
nvmecontrol format 名前空間名で、namespaceを初期化してみる。Namespace 1 Utilizationが14.4MBに減った。
> sudo nvmecontrol format nvme1ns1 > sudo smartctl -a /dev/nvme1 smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.0-RELEASE-p4 amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: SAMSUNG MZQLB960HAJR-00007 Serial Number: S437NY0KC00XYZ Firmware Version: EDA5202Q PCI Vendor/Subsystem ID: 0x144d IEEE OUI Identifier: 0x002538 Total NVM Capacity: 960,197,124,096 [960 GB] Unallocated NVM Capacity: 0 Controller ID: 4 Number of Namespaces: 1 Namespace 1 Size/Capacity: 960,197,124,096 [960 GB] Namespace 1 Utilization: 14,426,112 [14.4 MB] Namespace 1 Formatted LBA Size: 512 Local Time is: Thu Sep 19 00:41:04 2019 JST Firmware Updates (0x17): 3 Slots, Slot 1 R/O, no Reset required Optional Admin Commands (0x000f): Security Format Frmw_DL NS_Mngmt Optional NVM Commands (0x001f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Maximum Data Transfer Size: 512 Pages Warning Comp. Temp. Threshold: 87 Celsius Critical Comp. Temp. Threshold: 88 Celsius Namespace 1 Features (0x02): NA_Fields Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 10.60W - - 0 0 0 0 0 0 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 0 1 - 4096 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 42 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 36,935,426 [18.9 TB] Data Units Written: 4,227,826 [2.16 TB] Host Read Commands: 2,288,568,067 Host Write Commands: 96,108,724 Controller Busy Time: 241 Power Cycles: 417 Power On Hours: 2,260 Unsafe Shutdowns: 395 Media and Data Integrity Errors: 0 Error Information Log Entries: 11 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 42 Celsius Temperature Sensor 2: 48 Celsius Temperature Sensor 3: 56 Celsius Error Information (NVMe Log 0x01, max 64 entries) No Errors Logged
nvmecontrol format NVMeデバイスファイルでNVMeデバイス全体をフォーマットしてみる。名前空間が無くなっていることがわかる。
> sudo nvmecontrol format nvme1 > sudo smartctl -a /dev/nvme1 smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.0-RELEASE-p4 amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: SAMSUNG MZQLB960HAJR-00007 Serial Number: S437NY0KC00XYZ Firmware Version: EDA5202Q PCI Vendor/Subsystem ID: 0x144d IEEE OUI Identifier: 0x002538 Total NVM Capacity: 960,197,124,096 [960 GB] Unallocated NVM Capacity: 0 Controller ID: 4 Number of Namespaces: 1 Namespace 1 Size/Capacity: 960,197,124,096 [960 GB] Namespace 1 Utilization: 0 Namespace 1 Formatted LBA Size: 512 Local Time is: Thu Sep 19 00:42:30 2019 JST Firmware Updates (0x17): 3 Slots, Slot 1 R/O, no Reset required Optional Admin Commands (0x000f): Security Format Frmw_DL NS_Mngmt Optional NVM Commands (0x001f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Maximum Data Transfer Size: 512 Pages Warning Comp. Temp. Threshold: 87 Celsius Critical Comp. Temp. Threshold: 88 Celsius Namespace 1 Features (0x02): NA_Fields Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 10.60W - - 0 0 0 0 0 0 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 0 1 - 4096 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 42 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 36,938,628 [18.9 TB] Data Units Written: 4,227,826 [2.16 TB] Host Read Commands: 2,288,580,574 Host Write Commands: 96,108,724 Controller Busy Time: 241 Power Cycles: 417 Power On Hours: 2,260 Unsafe Shutdowns: 395 Media and Data Integrity Errors: 0 Error Information Log Entries: 11 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 42 Celsius Temperature Sensor 2: 48 Celsius Temperature Sensor 3: 57 Celsius Error Information (NVMe Log 0x01, max 64 entries) No Errors Logged
nvmecontrol format -E NVMeデバイスファイルでNVMeデバイスをUser Data Eraseしてみる。いわゆるSecure Eraseに相当するらしい。
> sudo nvmecontrol format -E nvme1 > sudo smartctl -a /dev/nvme1 smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.0-RELEASE-p4 amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: SAMSUNG MZQLB960HAJR-00007 Serial Number: S437NY0KC00XYZ Firmware Version: EDA5202Q PCI Vendor/Subsystem ID: 0x144d IEEE OUI Identifier: 0x002538 Total NVM Capacity: 960,197,124,096 [960 GB] Unallocated NVM Capacity: 0 Controller ID: 4 Number of Namespaces: 1 Namespace 1 Size/Capacity: 960,197,124,096 [960 GB] Namespace 1 Utilization: 0 Namespace 1 Formatted LBA Size: 512 Local Time is: Thu Sep 19 00:43:44 2019 JST Firmware Updates (0x17): 3 Slots, Slot 1 R/O, no Reset required Optional Admin Commands (0x000f): Security Format Frmw_DL NS_Mngmt Optional NVM Commands (0x001f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Maximum Data Transfer Size: 512 Pages Warning Comp. Temp. Threshold: 87 Celsius Critical Comp. Temp. Threshold: 88 Celsius Namespace 1 Features (0x02): NA_Fields Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 10.60W - - 0 0 0 0 0 0 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 0 1 - 4096 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 44 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 36,945,030 [18.9 TB] Data Units Written: 4,227,826 [2.16 TB] Host Read Commands: 2,288,605,580 Host Write Commands: 96,108,724 Controller Busy Time: 241 Power Cycles: 417 Power On Hours: 2,260 Unsafe Shutdowns: 395 Media and Data Integrity Errors: 0 Error Information Log Entries: 11 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 44 Celsius Temperature Sensor 2: 50 Celsius Temperature Sensor 3: 58 Celsius Error Information (NVMe Log 0x01, max 64 entries) No Errors Logged
nvmecontrol format -f LBAサイズID 名前空間でLBAサイズを指定して名前空間をフォーマットしてみる。LBAサイズIDはsmartctlのSupported LBA Sizesで表示されるIDを指定する模様。
使ったSSDでは512バイト、4096バイトの一般的なサイズのサポートとなっているが、チェックサム用領域8バイトを含んだ520バイトや4104バイトセクタなどに対応してる製品もあるらしい。
Namespace 1 Formatted LBA Sizeが4096に変わったことが分かる。
なお、LBAサイズの変更はOS側でハンドリングされてないようで、しばらくするとカーネルパニックするので直ぐに再起動すべし（FreeBSD 12.0Rp4現在）
> sudo nvmecontrol format -f 1 nvme0ns1 Decomo@freyja /u/h/Decomo> sudo smartctl -a /dev/nvme0 smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.0-RELEASE-p4 amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: SAMSUNG MZQLB960HAJR-00007 Serial Number: S437NY0KC00XYZ Firmware Version: EDA5202Q PCI Vendor/Subsystem ID: 0x144d IEEE OUI Identifier: 0x002538 Total NVM Capacity: 960,197,124,096 [960 GB] Unallocated NVM Capacity: 0 Controller ID: 4 Number of Namespaces: 1 Namespace 1 Size/Capacity: 960,197,124,096 [960 GB] Namespace 1 Utilization: 24,576 [24.5 KB] Namespace 1 Formatted LBA Size: 4096 Local Time is: Thu Sep 19 10:38:21 2019 JST Firmware Updates (0x17): 3 Slots, Slot 1 R/O, no Reset required Optional Admin Commands (0x000f): Security Format Frmw_DL NS_Mngmt Optional NVM Commands (0x001f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Maximum Data Transfer Size: 512 Pages Warning Comp. Temp. Threshold: 87 Celsius Critical Comp. Temp. Threshold: 88 Celsius Namespace 1 Features (0x02): NA_Fields Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 10.60W - - 0 0 0 0 0 0 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 - 512 0 0 1 + 4096 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 39 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 37,246,040 [19.0 TB] Data Units Written: 4,335,090 [2.21 TB] Host Read Commands: 2,298,018,184 Host Write Commands: 102,873,515 Controller Busy Time: 244 Power Cycles: 425 Power On Hours: 2,269 Unsafe Shutdowns: 399 Media and Data Integrity Errors: 0 Error Information Log Entries: 14 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 39 Celsius Temperature Sensor 2: 45 Celsius Temperature Sensor 3: 52 Celsius Error Information (NVMe Log 0x01, max 64 entries) No Errors Logged