ファイルサーバから表題のような、いやーな音が聞こえてきた。
恐る恐るzpool statusをしてみると…
$ zpool status pool: zdata3 state: DEGRADED status: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: scrub repaired 0 in 8h58m with 0 errors on Sun Jun 30 21:56:04 2013 config: NAME STATE READ WRITE CKSUM zdata3 DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 ada2p1 ONLINE 0 0 0 ada1p1 ONLINE 0 0 0 7910286219826960147 REMOVED 0 0 0 was /dev/ada0p1 raidz1-1 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada5 ONLINE 0 0 0 ada12 ONLINE 0 0 0 logs da1 ONLINE 0 0 0 cache da2 ONLINE 0 0 0 spares ada14p1 AVAIL errors: No known data errors
いやあああああああああああああああ!!
更にsyslogを見てみると…
Aug 19 08:11:04 Freyja kernel: ahcich0: Timeout on slot 9 port 0 Aug 19 08:11:04 Freyja kernel: ahcich0: is 00000000 cs 00000200 ss 00000000 rs 00000200 tfd c0 serr 00000000 cmd 0000c917 Aug 19 09:41:10 Freyja kernel: ahcich0: Timeout on slot 12 port 0 Aug 19 09:41:10 Freyja kernel: ahcich0: is 00000000 cs 00000000 ss 0000f000 rs 0000f000 tfd 40 serr 00000000 cmd 0000cf17 Aug 19 09:41:41 Freyja kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Aug 19 09:42:11 Freyja kernel: ahcich0: Timeout on slot 15 port 0 Aug 19 09:42:11 Freyja kernel: ahcich0: is 00000000 cs 00008000 ss 00000000 rs 00008000 tfd 80 serr 00000000 cmd 0000cf17 Aug 19 09:42:42 Freyja kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Aug 19 09:43:12 Freyja kernel: ahcich0: Timeout on slot 15 port 0 Aug 19 09:43:12 Freyja kernel: ahcich0: is 00000000 cs 00008000 ss 00000000 rs 00008000 tfd 80 serr 00000000 cmd 0000cf17 Aug 19 09:43:12 Freyja kernel: (ada0:ahcich0:0:0:0): lost device Aug 19 09:43:43 Freyja kernel: ahcich0: AHCI reset: device not ready after 31000ms (tfd = 00000080) Aug 19 09:44:13 Freyja kernel: ahcich0: Timeout on slot 15 port 0 Aug 19 09:44:13 Freyja kernel: ahcich0: is 00000000 cs 00078000 ss 00078000 rs 00078000 tfd 80 serr 00000000 cmd 0000cf17 Aug 19 09:44:14 Freyja kernel: (ada0:ahcich0:0:0:0): removing device entry Aug 19 09:44:17 Freyja kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 Aug 19 09:44:17 Freyja kernel: ada0: <ST3000DM001-9YN166 CC4B> ATA-8 SATA 3.x device Aug 19 09:44:17 Freyja kernel: ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) Aug 19 09:44:17 Freyja kernel: ada0: Command Queueing enabled Aug 19 09:44:17 Freyja kernel: ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C) Aug 19 09:44:17 Freyja kernel: ada0: Previously was known as ad4 Aug 19 09:44:30 Freyja kernel: ahcich0: Timeout on slot 18 port 0 Aug 19 09:44:30 Freyja kernel: ahcich0: is 00000000 cs 00040000 ss 00000000 rs 00040000 tfd c0 serr 00000000 cmd 0000d217 Aug 19 09:44:30 Freyja kernel: ahcich0: Error while READ LOG EXT
凄く・・・ご臨終です・・・・。
8/19ってことは、実家帰省のさなか延々カッコンカッコンしてたのか…。RAID-Z1なのでガクブルもの。てか、ホットスペアを自動で使ってくれないのね。何でだろう？
取りあえず、手動でお亡くなりになったHDDとスペアを置き換える。
まず、死んだHDDをプールから切り離す。
$ sudo zpool offline zdata3 ada0p1 $ zpool status zdata3 pool: zdata3 state: DEGRADED status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: scrub repaired 0 in 8h58m with 0 errors on Sun Jun 30 21:56:04 2013 config: NAME STATE READ WRITE CKSUM zdata3 DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 ada2p1 ONLINE 0 0 0 ada1p1 ONLINE 0 0 0 7910286219826960147 OFFLINE 0 0 0 was /dev/ada0p1 raidz1-1 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada5 ONLINE 0 0 0 ada12 ONLINE 0 0 0 logs da1 ONLINE 0 0 0 cache da2 ONLINE 0 0 0 spares ada14p1 AVAIL errors: No known data errors
それから、スペアと入れ替える。
$ sudo zpool replace zdata3 ada0p1 ada14p1
$ zpool status zdata3
pool: zdata3
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Aug 24 10:42:18 2013
53.5M scanned out of 10.1T at 4.86M/s, 602h18m to go
8.17M resilvered, 0.00% done
config:
NAME STATE READ WRITE CKSUM
zdata3 DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
ada2p1 ONLINE 0 0 0
ada1p1 ONLINE 0 0 0
spare-2 OFFLINE 0 0 0
7910286219826960147 OFFLINE 0 0 0 was /dev/ada0p1
ada14p1 ONLINE 0 0 0 (resilvering)
raidz1-1 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada5 ONLINE 0 0 0
ada12 ONLINE 0 0 0
logs
da1 ONLINE 0 0 0
cache
da2 ONLINE 0 0 0
spares
8748654380753058766 INUSE was /dev/ada14p1
errors: No known data errors
replaceコマンドの応答がなかなか(十数秒)返ってこず、少し焦ったｗ 3TBのHDD買ってこないとだな…。
それにしても、自分はSeagateとの相性がすこぶるよろしくない。HDDは去年の6/20に買ったものなので、1年ちょいでお亡くなりという事に。昔使ってた鯖でも、Seagate製が1年足らずで死んだんだよなぁ。RMAで動作品に交換出来ると思うが、さっさと売って新HDD費用の足しにしよう……。
(2014/8/25 1:18追記)
10TBのプールを10時間ほどでリビルド完了。
死んだHDDを切り離す。
$ sudo zpool detach zdata3 ada0p1 $ zpool status zdata3 pool: zdata3 state: ONLINE scan: resilvered 2.37T in 10h6m with 0 errors on Sat Aug 24 20:49:04 2013 config: NAME STATE READ WRITE CKSUM zdata3 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ada2p1 ONLINE 0 0 0 ada1p1 ONLINE 0 0 0 ada14p1 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada5 ONLINE 0 0 0 ada12 ONLINE 0 0 0 logs da1 ONLINE 0 0 0 cache da2 ONLINE 0 0 0 errors: No known data errors
ada0がケース内のどのHDDなのか特定するには、HDDの型番とシリアル番号が分かればいいので、通常はcamcontrol identify ada0すればOK。 ただし今回はada0の情報が全く表示されなかったため、生きている同型のHDDを1つずつidentifyし、死んだ玉を消去法で探し当てた。