Michael Rendell | 18 Dec 02:45 2008
Picon
Picon

mkfs.ext3 taking long time on raid1 with nda disk

Hi Peter,

  Have noticed that creating a file system on a raid1 device
that uses a enbd device sometimes runs into problems: looks
like it is geting errors writing to the server and taking a
while to recover.  Wondering if something is wrong in the
configuration?

The set up is as follows:

/etc/enbd.conf:
    module 0 show_errs=1
    client mDisklessVars /dev/nda nmhd-bs2 1090 -e -n4 -q -t 0

(pwprog is configured but disabled)

cat /proc/nbdinfo
    Device a:       Open
    [a] State:      verify, rw, enabled, validated, show_errs, last error 0, 
lives 0, bp 0
    [a] Queued:     +0R/0W curr (check 0R/0W) +0R/2W max
    [a] Buffersize: 266240  (sectors=520, blocks=65)
    [a] Blocksize:  4096    (log=12)
    [a] Size:       3387392KB
    [a] Blocks:     846848
    [a] Sockets:    4       (+)     (+)     (*)     (+)
    [a] Requested:  0       (0)     (0)     (0)     (0)     0R/0W   max 0
    [a] Despatched: 0       (0)     (0)     (0)     (0)     0R/0W   md5 0W (0 
eq, 0 ne, 0 dn)
    [a] Errored:    0       (0)     (0)     (0)     (0)     0+0
    [a] Pending:    0       (0)     (0)     (0)     (0)     0R/0W+0R/0W
    [a] B/s now:    0       (0R+0W)
    [a] B/s ave:    0       (0R+0W)
    [a] B/s max:    0       (0R+0W)
    [a] Spectrum:   100%0
    [a] OSpectrum:  100%0
    [a] Kthreads:   0       (0 waiting/0 running/0 max)
    [a] Cthreads:   4       (+)     (+)     (+)     (+)
    [a] Cpids:      4       (5816)  (5817)  (5818)  (5824)
    Device b-p:     Closed

/bin/echo merge_requests=4 > /proc/nbdinfo

cat /proc/nbdinfo
    Device a:       Open
    [a] State:      verify, rw, merge_requests 4, enabled, validated, 
show_errs, last error 0, lives 0, bp 0
    [a] Queued:     +0R/0W curr (check 0R/0W) +0R/2W max
    [a] Buffersize: 266240  (sectors=520, blocks=65)
    [a] Blocksize:  4096    (log=12)
    [a] Size:       3387392KB
    [a] Blocks:     846848
    [a] Sockets:    4       (+)     (+)     (*)     (+)
    [a] Requested:  0       (0)     (0)     (0)     (0)     0R/0W   max 0
    [a] Despatched: 0       (0)     (0)     (0)     (0)     0R/0W   md5 0W (0 
eq, 0 ne, 0 dn)
    [a] Errored:    0       (0)     (0)     (0)     (0)     0+0
    [a] Pending:    0       (0)     (0)     (0)     (0)     0R/0W+0R/0W
    [a] B/s now:    0       (0R+0W)
    [a] B/s ave:    0       (0R+0W)
    [a] B/s max:    0       (0R+0W)
    [a] Spectrum:   100%0
    [a] OSpectrum:  100%0
    [a] Kthreads:   0       (0 waiting/0 running/0 max)
    [a] Cthreads:   4       (+)     (+)     (+)     (+)
    [a] Cpids:      4       (5816)  (5817)  (5818)  (5824)
    Device b-p:     Closed

All is OK.  Then the raid1 disk is created and a file system put on it:

mdadm --create --assume-clean -n 2 -l raid1 --homehost=nmhd-bs1 
--bitmap=internal --write-behind /dev/md12 /dev/vg-system/lv-diskless-vars 
--write-mostly /dev/nda
    mdadm: array /dev/md12 started.

cat /proc/mdstat
    Personalities : [raid1] [raid6] [raid5] [raid4]
    md12 : active raid1 nda[1](W) dm-5[0]
	  3387328 blocks [2/2] [UU]
	  bitmap: 0/207 pages [0KB], 8KB chunk

time mkfs.ext3 -v -L /diskless-vars /dev/md12 &
[1] 6253
    mke2fs 1.39 (29-May-2006)
    Filesystem label=/diskless-vars
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    423488 inodes, 846832 blocks
    42341 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=868220928
    26 block groups
    32768 blocks per group, 32768 fragments per group
    16288 inodes per group
    Superblock backups stored on blocks:
	    32768, 98304, 163840, 229376, 294912, 819200

    Writing inode tables: done
    Creating journal (16384 blocks): done
    Writing superblocks and filesystem accounting information:

(wait a while)

cat /proc/mdstat
    Personalities : [raid1] [raid6] [raid5] [raid4]
    md12 : active raid1 nda[1](W) dm-5[0]
	  3387328 blocks [2/2] [UU]
	  bitmap: 29/207 pages [116KB], 8KB chunk

Then errors on console: end_request: I/O error, dev nda, sector 26888 ...
(see syslog below for details)

cat /proc/mdstat
    Personalities : [raid1] [raid6] [raid5] [raid4]
    md12 : active raid1 nda[1](W)(F) dm-5[0]
	  3387328 blocks [2/1] [U_]
	  bitmap: 29/207 pages [116KB], 8KB chunk

mdadm --remove /dev/md12 /dev/nda
    mdadm: hot remove failed for /dev/nda: Device or resource busy

cat /proc/nbdinfo
    Device a:       Open
    [a] State:      verify, rw, merge_requests 4, enabled, validated, 
show_errs, md5sum, last error 0, lives 0, bp 0
    [a] Queued:     +0R/127W curr (check 0R/127W) +4R/156W max
    [a] Buffersize: 266240  (sectors=520, blocks=65)
    [a] Blocksize:  4096    (log=12)
    [a] Size:       3387392KB
    [a] Blocks:     846848
    [a] Sockets:    4       (+)     (+)     (+)     (*)
    [a] Requested:  4.9537K (1.25K) (1.24K) (1.22K) (1.23K) 36R/4.927KW     
max 1
    [a] Despatched: 4.8410K (1.22K) (1.21K) (1.19K) (1.20K) 36R/4.805KW     
md5 3.82KW (3.82K eq, 0 ne, 0 dn)
    [a] Errored:    120     (30)    (30)    (30)    (30)    120+0
    [a] Pending:    4       (1)     (1)     (1)     (1)     0R/4W+0R/0W
    [a] B/s now:    20.0K   (0R+20.0KW)
    [a] B/s ave:    12.0K   (0R+12.0KW)
    [a] B/s max:    232K    (0R+228KW)
    [a] Spectrum:   99%1
    [a] OSpectrum:  99%1
    [a] Kthreads:   0       (0 waiting/0 running/1 max)
    [a] Cthreads:   0       (-)     (-)     (-)     (-)
    [a] Cpids:      0       (5816)  (5817)  (5818)  (5824)
    Device b-p:     Closed

Then, 48 minutes later (from time of mkfs) - mkfs finishes:
    This filesystem will be automatically checked every 39 mounts or
    180 days, whichever comes first.  Use tune2fs -c or -i to override.

cat /proc/mdstat
    Personalities : [raid1] [raid6] [raid5] [raid4]
    md12 : active raid1 nda[2](W)(F) dm-5[0]
	  3387328 blocks [2/1] [U_]
	  bitmap: 15/207 pages [60KB], 8KB chunk

cat /proc/nbdinfo
    Device a:       Open
    [a] State:      verify, rw, merge_requests 4, enabled, validated, 
show_errs, md5sum, last error 0, lives 0, bp 0
    [a] Queued:     +0R/0W curr (check 0R/0W) +4R/156W max
    [a] Buffersize: 266240  (sectors=520, blocks=65)
    [a] Blocksize:  4096    (log=12)
    [a] Size:       3387392KB
    [a] Blocks:     846848
    [a] Sockets:    4       (+)     (+)     (+)     (*)
    [a] Requested:  23.218K (5.85K) (5.84K) (5.82K) (5.83K) 36R/23.32KW     
max 1
    [a] Despatched: 23.246K (5.82K) (5.81K) (5.79K) (5.80K) 36R/23.21KW     
md5 22.2KW (22.2K eq, 2 ne, 0 dn)
    [a] Errored:    120     (30)    (30)    (30)    (30)    120+0
    [a] Pending:    0       (0)     (0)     (0)     (0)     0R/0W+0R/0W
    [a] B/s now:    0       (0R+0W)
    [a] B/s ave:    8.00K   (0R+8.00KW)
    [a] B/s max:    232K    (0R+228KW)
    [a] Spectrum:   99%1
    [a] OSpectrum:  99%1
    [a] Kthreads:   0       (0 waiting/0 running/1 max)
    [a] Cthreads:   4       (+)     (+)     (+)     (+)
    [a] Cpids:      4       (5816)  (5817)  (5818)  (5824)
    Device b-p:     Closed

mdadm --remove /dev/md12 /dev/nda
    mdadm: hot removed /dev/nda

mdadm --re-add /dev/md12 /dev/nda
    mdadm: re-added /dev/nda

cat /proc/mdstat
    Personalities : [raid1] [raid6] [raid5] [raid4]
    md12 : active raid1 nda[1](W) dm-5[0]
	  3387328 blocks [2/1] [U_]
	  [>....................]  recovery =  1.3% (46588/3387328) finish=364.7min 
speed=150K/sec
	  bitmap: 15/207 pages [60KB], 8KB chunk

(3 or 4 minutes later, not 364...)

cat /proc/mdstat
    Personalities : [raid1] [raid6] [raid5] [raid4]
    md12 : active raid1 nda[1](W) dm-5[0]
	  3387328 blocks [2/2] [UU]
	  bitmap: 0/207 pages [0KB], 8KB chunk

Contents of syslog (from creation on):

    Dec 17 17:15:26 nmhd-bs1 kernel: ENBD #4246[0]: enbd_set_max_sectors 
cannot set max_sectors on ndn because no queue is yet attached
    Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #1273[7]: enbd_generic_open 
incremented refcnt to 7 on nda
    Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #1307[2]: enbd_generic_open DISK 
CHECK wanted in open on device nda
    Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #5997[2]: enbd_media_changed (1 msgs 
suppressed) MEDIA CHANGED called
    Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #6057[2]: enbd_media_changed (1 msgs 
suppressed) REMOTE CHECK still has REMOTE VALID on nda
    Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #5806[1]: enbd_release entered for 
inode d3238e30 file 00000000.
    Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #5845[1]: enbd_release decremented 
refcnt on nda to 6
    Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #5877[1]: enbd_release exited for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #1273[8]: enbd_generic_open 
incremented refcnt to 7 on nda
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[2]: enbd_release entered for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[2]: enbd_release decremented 
refcnt on nda to 6
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[2]: enbd_release exited for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: md: bind<dm-5>
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[3]: enbd_release entered for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[3]: enbd_release decremented 
refcnt on nda to 6
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[3]: enbd_release exited for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[4]: enbd_release entered for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[4]: enbd_release decremented 
refcnt on nda to 6
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[4]: enbd_release exited for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[5]: enbd_release entered for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[5]: enbd_release decremented 
refcnt on nda to 6
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[5]: enbd_release exited for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[6]: enbd_release entered for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[6]: enbd_release decremented 
refcnt on nda to 6
    Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[6]: enbd_release exited for 
inode d3238e30 file 00000000.
    Dec 17 17:15:42 nmhd-bs1 kernel: md: bind<nda>
    Dec 17 17:15:42 nmhd-bs1 kernel: raid1: raid set md12 active with 2 out of 
2 mirrors
    Dec 17 17:15:42 nmhd-bs1 kernel: md12: bitmap file is out of date (0 < 1) 
-- forcing full recovery
    Dec 17 17:15:42 nmhd-bs1 kernel: md12: bitmap file is out of date, doing 
full recovery
    Dec 17 17:15:42 nmhd-bs1 kernel: md12: bitmap initialized from disk: read 
13/13 pages, set 423416 bits, status: 0
    Dec 17 17:15:42 nmhd-bs1 kernel: created bitmap (207 pages) for device 
md12
    Dec 17 17:15:43 nmhd-bs1 enbd-pwprog: args are <NewArray /dev/md12>

 # raid now created; mkfs running...

    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1629[0]: enbd_rollback (0): 
erroring too old (delay 18s >= timeout 90s) req cb2f9078 id 0x4745ca1f!
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1550[0]: enbd_error error out req 
cb2f9078 id 0x4745ca1f from slot 0!
    Dec 17 17:18:01 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13792
    Dec 17 17:18:01 nmhd-bs1 kernel: raid1: Disk failure on nda, disabling 
device. 
    Dec 17 17:18:01 nmhd-bs1 kernel: 	Operation continuing on 1 devices
    Dec 17 17:18:01 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13800
    Dec 17 17:18:01 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13808
    Dec 17 17:18:01 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13816
    Dec 17 17:18:01 nmhd-bs1 enbd-pwprog: /dev/md12(/dev/nda) has been faulted 
and we are waiting for it... 
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1273[14]: enbd_generic_open (5 msgs 
suppressed) incremented refcnt to 8 on nda
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1307[3]: enbd_generic_open DISK 
CHECK wanted in open on device nda
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #5997[3]: enbd_media_changed MEDIA 
CHANGED called
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #4744[0]: enbd_get_blk_request took 
too long (0 jiffies, 0 tries, timeout 90000 jiffies) to get spare req
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #4946[0]: do_enbd_remote_ioctl could 
not make a remote ioctl (0x4004ab2a) request
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #5806[7]: enbd_release entered for 
inode d3238e30 file 00000000.
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #5845[7]: enbd_release decremented 
refcnt on nda to 7
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #5877[7]: enbd_release exited for 
inode d3238e30 file 00000000.
    Dec 17 17:18:01 nmhd-bs1 kernel: md: cannot remove active disk nda from 
md12 ... 
    Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1958[0]: enbd_ack (0): ignoring ack 
of req id 4745ca1f which slot lacks
    Dec 17 17:18:02 nmhd-bs1 kernel: ENBD #1629[4]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f9d3c id 
0x4745c9df!
    Dec 17 17:18:02 nmhd-bs1 kernel: ENBD #1550[4]: enbd_error (3 msgs 
suppressed) error out req cb2f9d3c id 0x4745c9df from slot 0!
    Dec 17 17:18:02 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13856
    Dec 17 17:18:02 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13864
    Dec 17 17:18:02 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13872
    Dec 17 17:18:02 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13880
    Dec 17 17:18:02 nmhd-bs1 kernel: ENBD #1958[4]: enbd_ack (3 msgs 
suppressed) (0): ignoring ack of req id 4745c9df which slot lacks
    Dec 17 17:18:03 nmhd-bs1 kernel: ENBD #1629[8]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2f9480 id 
0xee54c99f!
    Dec 17 17:18:03 nmhd-bs1 kernel: ENBD #1550[8]: enbd_error (3 msgs 
suppressed) error out req cb2f9480 id 0xee54c99f from slot 0!
    Dec 17 17:18:03 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13920
    Dec 17 17:18:03 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13928
    Dec 17 17:18:03 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13936
    Dec 17 17:18:03 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13944
    Dec 17 17:18:03 nmhd-bs1 kernel: ENBD #1958[8]: enbd_ack (3 msgs 
suppressed) (0): ignoring ack of req id ee54c99f which slot lacks
    Dec 17 17:18:04 nmhd-bs1 kernel: ENBD #1629[12]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req dcd31124 id 
0xee54c95f!
    Dec 17 17:18:04 nmhd-bs1 kernel: ENBD #1550[12]: enbd_error (3 msgs 
suppressed) error out req dcd31124 id 0xee54c95f from slot 0!
    Dec 17 17:18:04 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13984
    Dec 17 17:18:04 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
13992
    Dec 17 17:18:04 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14000
    Dec 17 17:18:04 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14008
    Dec 17 17:18:04 nmhd-bs1 kernel: ENBD #1958[12]: enbd_ack (3 msgs 
suppressed) (0): ignoring ack of req id ee54c95f which slot lacks
    Dec 17 17:18:05 nmhd-bs1 kernel: ENBD #1629[16]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2e7a8c id 
0xee54c91f!
    Dec 17 17:18:05 nmhd-bs1 kernel: ENBD #1550[16]: enbd_error (3 msgs 
suppressed) error out req cb2e7a8c id 0xee54c91f from slot 0!
    Dec 17 17:18:05 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14048
    Dec 17 17:18:05 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14056
    Dec 17 17:18:05 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14064
    Dec 17 17:18:05 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14072
    Dec 17 17:18:05 nmhd-bs1 kernel: ENBD #1958[16]: enbd_ack (3 msgs 
suppressed) (0): ignoring ack of req id ee54c91f which slot lacks
    Dec 17 17:18:06 nmhd-bs1 kernel: ENBD #1629[20]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req dcd311d0 id 
0xee54c8df!
    Dec 17 17:18:06 nmhd-bs1 kernel: ENBD #1550[20]: enbd_error (3 msgs 
suppressed) error out req dcd311d0 id 0xee54c8df from slot 0!
    Dec 17 17:18:06 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14112
    Dec 17 17:18:06 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14120
    Dec 17 17:18:06 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14128
    Dec 17 17:18:06 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14136
    Dec 17 17:18:07 nmhd-bs1 kernel: ENBD #1958[20]: enbd_ack (3 msgs 
suppressed) (0): ignoring ack of req id ee54c8df which slot lacks
    Dec 17 17:18:07 nmhd-bs1 kernel: ENBD #1629[24]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2f2684 id 
0xa664c89f!
    Dec 17 17:18:07 nmhd-bs1 kernel: ENBD #1550[24]: enbd_error (3 msgs 
suppressed) error out req cb2f2684 id 0xa664c89f from slot 0!
    Dec 17 17:18:07 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14176
    Dec 17 17:18:07 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14184
    Dec 17 17:18:07 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14192
    Dec 17 17:18:07 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14200
    Dec 17 17:18:08 nmhd-bs1 kernel: ENBD #1958[24]: enbd_ack (3 msgs 
suppressed) (0): ignoring ack of req id a664c89f which slot lacks
    Dec 17 17:18:08 nmhd-bs1 kernel: ENBD #1629[28]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2f5f40 id 
0xa664c847!
    Dec 17 17:18:08 nmhd-bs1 kernel: ENBD #1550[28]: enbd_error (3 msgs 
suppressed) error out req cb2f5f40 id 0xa664c847 from slot 0!
    Dec 17 17:18:08 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14264
    Dec 17 17:18:08 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14240
    Dec 17 17:18:08 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14248
    Dec 17 17:18:08 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14256
    Dec 17 17:18:09 nmhd-bs1 kernel: ENBD #1958[28]: enbd_ack (3 msgs 
suppressed) (1): ignoring ack of req id a664c85f which slot lacks
    Dec 17 17:18:09 nmhd-bs1 kernel: ENBD #1629[32]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2f5a8c id 
0xa664c80f!
    Dec 17 17:18:09 nmhd-bs1 kernel: ENBD #1550[32]: enbd_error (3 msgs 
suppressed) error out req cb2f5a8c id 0xa664c80f from slot 0!
    Dec 17 17:18:09 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14320
    Dec 17 17:18:09 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14304
    Dec 17 17:18:09 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14328
    Dec 17 17:18:09 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14312
    Dec 17 17:18:10 nmhd-bs1 kernel: ENBD #1958[32]: enbd_ack (3 msgs 
suppressed) (1): ignoring ack of req id a664c81f which slot lacks
    Dec 17 17:18:10 nmhd-bs1 kernel: ENBD #1629[36]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f552c id 
0xa664c7d7!
    Dec 17 17:18:10 nmhd-bs1 kernel: ENBD #1550[36]: enbd_error (3 msgs 
suppressed) error out req cb2f552c id 0xa664c7d7 from slot 0!
    Dec 17 17:18:10 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14376
    Dec 17 17:18:10 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14392
    Dec 17 17:18:10 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14384
    Dec 17 17:18:10 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14368
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1958[36]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id a664c7df which slot lacks
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1629[40]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2ed730 id 
0x9074c787!
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1550[40]: enbd_error (3 msgs 
suppressed) error out req cb2ed730 id 0x9074c787 from slot 0!
    Dec 17 17:18:11 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14456
    Dec 17 17:18:11 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14448
    Dec 17 17:18:11 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14440
    Dec 17 17:18:11 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14432
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1273[15]: enbd_generic_open 
incremented refcnt to 8 on nda
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1307[4]: enbd_generic_open DISK 
CHECK wanted in open on device nda
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #5997[4]: enbd_media_changed MEDIA 
CHANGED called
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #4744[2]: enbd_get_blk_request (1 
msgs suppressed) took too long (0 jiffies, 0 tries, timeout 90000 jiffies) to 
get spare req
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #4946[2]: do_enbd_remote_ioctl (1 
msgs suppressed) could not make a remote ioctl (0x4004ab2a) request
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #5806[8]: enbd_release entered for 
inode d3238e30 file 00000000.
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #5845[8]: enbd_release decremented 
refcnt on nda to 7
    Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #5877[8]: enbd_release exited for 
inode d3238e30 file 00000000.
    Dec 17 17:18:11 nmhd-bs1 kernel: md: cannot remove active disk nda from 
md12 ... 
    Dec 17 17:18:12 nmhd-bs1 kernel: ENBD #1958[40]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id 9074c79f which slot lacks
    Dec 17 17:18:12 nmhd-bs1 kernel: ENBD #1629[44]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2edc90 id 
0x9074c747!
    Dec 17 17:18:12 nmhd-bs1 kernel: ENBD #1550[44]: enbd_error (3 msgs 
suppressed) error out req cb2edc90 id 0x9074c747 from slot 0!
    Dec 17 17:18:12 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14520
    Dec 17 17:18:12 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14512
    Dec 17 17:18:12 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14504
    Dec 17 17:18:12 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14496
    Dec 17 17:18:13 nmhd-bs1 kernel: ENBD #1958[44]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id 9074c75f which slot lacks
    Dec 17 17:18:13 nmhd-bs1 kernel: ENBD #1629[48]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2ed5d8 id 
0x9074c707!
    Dec 17 17:18:13 nmhd-bs1 kernel: ENBD #1550[48]: enbd_error (3 msgs 
suppressed) error out req cb2ed5d8 id 0x9074c707 from slot 0!
    Dec 17 17:18:13 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14584
    Dec 17 17:18:13 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14568
    Dec 17 17:18:13 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14560
    Dec 17 17:18:13 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14576
    Dec 17 17:18:14 nmhd-bs1 kernel: ENBD #1958[48]: enbd_ack (3 msgs 
suppressed) (2): ignoring ack of req id 9074c71f which slot lacks
    Dec 17 17:18:14 nmhd-bs1 kernel: ENBD #1629[52]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2edde8 id 
0x9074c6df!
    Dec 17 17:18:14 nmhd-bs1 kernel: ENBD #1550[52]: enbd_error (3 msgs 
suppressed) error out req cb2edde8 id 0x9074c6df from slot 0!
    Dec 17 17:18:14 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14624
    Dec 17 17:18:14 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14640
    Dec 17 17:18:14 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14632
    Dec 17 17:18:14 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14648
    Dec 17 17:18:15 nmhd-bs1 kernel: ENBD #1958[52]: enbd_ack (3 msgs 
suppressed) (0): ignoring ack of req id 9074c6df which slot lacks
    Dec 17 17:18:15 nmhd-bs1 kernel: ENBD #1629[56]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2f2f40 id 
0x7a84c68f!
    Dec 17 17:18:15 nmhd-bs1 kernel: ENBD #1550[56]: enbd_error (3 msgs 
suppressed) error out req cb2f2f40 id 0x7a84c68f from slot 0!
    Dec 17 17:18:15 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14704
    Dec 17 17:18:15 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14688
    Dec 17 17:18:15 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14712
    Dec 17 17:18:15 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14696
    Dec 17 17:18:16 nmhd-bs1 kernel: ENBD #1958[56]: enbd_ack (3 msgs 
suppressed) (1): ignoring ack of req id 7a84c69f which slot lacks
    Dec 17 17:18:16 nmhd-bs1 kernel: ENBD #1629[60]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2e7934 id 
0x7a84c647!
    Dec 17 17:18:16 nmhd-bs1 kernel: ENBD #1550[60]: enbd_error (3 msgs 
suppressed) error out req cb2e7934 id 0x7a84c647 from slot 0!
    Dec 17 17:18:16 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14776
    Dec 17 17:18:16 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14760
    Dec 17 17:18:16 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14752
    Dec 17 17:18:16 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14768
    Dec 17 17:18:17 nmhd-bs1 kernel: ENBD #1958[60]: enbd_ack (3 msgs 
suppressed) (2): ignoring ack of req id 7a84c65f which slot lacks
    Dec 17 17:18:17 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14832
    Dec 17 17:18:17 nmhd-bs1 kernel: ENBD #1629[65]: enbd_rollback (4 msgs 
suppressed) (1): erroring too old (delay 18s >= timeout 90s) req cb2e7730 id 
0x7a84c61f!
    Dec 17 17:18:17 nmhd-bs1 kernel: ENBD #1550[65]: enbd_error (4 msgs 
suppressed) error out req cb2e7730 id 0x7a84c61f from slot 1!
    Dec 17 17:18:17 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14816
    Dec 17 17:18:17 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14840
    Dec 17 17:18:17 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14824
    Dec 17 17:18:18 nmhd-bs1 kernel: ENBD #1958[64]: enbd_ack (3 msgs 
suppressed) (1): ignoring ack of req id 7a84c61f which slot lacks
    Dec 17 17:18:18 nmhd-bs1 kernel: ENBD #1629[68]: enbd_rollback (2 msgs 
suppressed) (0): erroring too old (delay 15s >= timeout 90s) req cb2f5124 id 
0x3394c5c7!
    Dec 17 17:18:18 nmhd-bs1 kernel: ENBD #1550[68]: enbd_error (2 msgs 
suppressed) error out req cb2f5124 id 0x3394c5c7 from slot 0!
    Dec 17 17:18:18 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14904
    Dec 17 17:18:18 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14888
    Dec 17 17:18:18 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14880
    Dec 17 17:18:18 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14896
    Dec 17 17:18:19 nmhd-bs1 kernel: ENBD #1958[68]: enbd_ack (3 msgs 
suppressed) (2): ignoring ack of req id 7a84c5df which slot lacks
    Dec 17 17:18:19 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14952
    Dec 17 17:18:19 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14968
    Dec 17 17:18:19 nmhd-bs1 kernel: ENBD #1629[74]: enbd_rollback (5 msgs 
suppressed) (2): erroring too old (delay 16s >= timeout 90s) req cb2f9be4 id 
0x3a94c58f!
    Dec 17 17:18:19 nmhd-bs1 kernel: ENBD #1550[74]: enbd_error (5 msgs 
suppressed) error out req cb2f9be4 id 0x3a94c58f from slot 2!
    Dec 17 17:18:19 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14960
    Dec 17 17:18:19 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
14944
    Dec 17 17:18:20 nmhd-bs1 kernel: ENBD #1958[72]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id 3a94c59f which slot lacks
    Dec 17 17:18:20 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15016
    Dec 17 17:18:20 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15032
    Dec 17 17:18:20 nmhd-bs1 kernel: ENBD #1629[78]: enbd_rollback (3 msgs 
suppressed) (2): erroring too old (delay 17s >= timeout 90s) req de711078 id 
0x3a94c54f!
    Dec 17 17:18:20 nmhd-bs1 kernel: ENBD #1550[78]: enbd_error (3 msgs 
suppressed) error out req de711078 id 0x3a94c54f from slot 2!
    Dec 17 17:18:20 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15024
    Dec 17 17:18:20 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15008
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1958[76]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id 3a94c55f which slot lacks
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1629[80]: enbd_rollback (1 msgs 
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2ea684 id 
0x3a94c50f!
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1550[80]: enbd_error (1 msgs 
suppressed) error out req cb2ea684 id 0x3a94c50f from slot 0!
    Dec 17 17:18:21 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15088
    Dec 17 17:18:21 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15080
    Dec 17 17:18:21 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15096
    Dec 17 17:18:21 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15072
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1273[16]: enbd_generic_open 
incremented refcnt to 8 on nda
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1307[5]: enbd_generic_open DISK 
CHECK wanted in open on device nda
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #5997[5]: enbd_media_changed MEDIA 
CHANGED called
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #4744[4]: enbd_get_blk_request (1 
msgs suppressed) took too long (0 jiffies, 0 tries, timeout 90000 jiffies) to 
get spare req
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #4946[4]: do_enbd_remote_ioctl (1 
msgs suppressed) could not make a remote ioctl (0x4004ab2a) request
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #4946[5]: do_enbd_remote_ioctl could 
not make a remote ioctl (0x4004ab2a) request
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #5806[9]: enbd_release entered for 
inode d3238e30 file 00000000.
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #5845[9]: enbd_release decremented 
refcnt on nda to 7
    Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #5877[9]: enbd_release exited for 
inode d3238e30 file 00000000.
    Dec 17 17:18:21 nmhd-bs1 kernel: md: cannot remove active disk nda from 
md12 ... 
    Dec 17 17:18:22 nmhd-bs1 kernel: ENBD #1958[80]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id 3a94c51f which slot lacks
    Dec 17 17:18:22 nmhd-bs1 kernel: ENBD #1629[84]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f5328 id 
0x3a94c4cf!
    Dec 17 17:18:22 nmhd-bs1 kernel: ENBD #1550[84]: enbd_error (3 msgs 
suppressed) error out req cb2f5328 id 0x3a94c4cf from slot 0!
    Dec 17 17:18:22 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15152
    Dec 17 17:18:22 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15144
    Dec 17 17:18:22 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15160
    Dec 17 17:18:22 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15136
    Dec 17 17:18:23 nmhd-bs1 kernel: ENBD #1958[84]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id 3a94c4df which slot lacks
    Dec 17 17:18:23 nmhd-bs1 kernel: ENBD #1629[88]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req dd357e94 id 
0x13a4c487!
    Dec 17 17:18:23 nmhd-bs1 kernel: ENBD #1550[88]: enbd_error (3 msgs 
suppressed) error out req dd357e94 id 0x13a4c487 from slot 0!
    Dec 17 17:18:23 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15224
    Dec 17 17:18:23 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15216
    Dec 17 17:18:23 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15208
    Dec 17 17:18:23 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15200
    Dec 17 17:18:24 nmhd-bs1 kernel: ENBD #1958[88]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id 13a4c49f which slot lacks
    Dec 17 17:18:24 nmhd-bs1 kernel: ENBD #1629[92]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2e7b38 id 
0x13a4c447!
    Dec 17 17:18:24 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15288
    Dec 17 17:18:24 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15280
    Dec 17 17:18:24 nmhd-bs1 kernel: ENBD #1550[94]: enbd_error (5 msgs 
suppressed) error out req cb2e7d3c id 0x13a4c457 from slot 2!
    Dec 17 17:18:24 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15272
    Dec 17 17:18:24 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15264
    Dec 17 17:18:25 nmhd-bs1 kernel: ENBD #1958[92]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id 13a4c45f which slot lacks
    Dec 17 17:18:25 nmhd-bs1 kernel: ENBD #1629[96]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req dcd31328 id 
0x13a4c407!
    Dec 17 17:18:25 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15352
    Dec 17 17:18:25 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15344
    Dec 17 17:18:25 nmhd-bs1 kernel: ENBD #1550[98]: enbd_error (3 msgs 
suppressed) error out req cb2f9e94 id 0x13a4c417 from slot 2!
    Dec 17 17:18:25 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15336
    Dec 17 17:18:25 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15328
    Dec 17 17:18:26 nmhd-bs1 kernel: ENBD #1958[96]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id 13a4c41f which slot lacks
    Dec 17 17:18:26 nmhd-bs1 kernel: ENBD #1629[100]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f927c id 
0x13a4c3e7!
    Dec 17 17:18:26 nmhd-bs1 kernel: ENBD #1550[100]: enbd_error (1 msgs 
suppressed) error out req cb2f927c id 0x13a4c3e7 from slot 0!
    Dec 17 17:18:26 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15384
    Dec 17 17:18:26 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15376
    Dec 17 17:18:26 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15400
    Dec 17 17:18:26 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15392
    Dec 17 17:18:27 nmhd-bs1 kernel: ENBD #1958[102]: enbd_ack (5 msgs 
suppressed) (3): ignoring ack of req id 13a4c3df which slot lacks
    Dec 17 17:18:27 nmhd-bs1 kernel: ENBD #1629[104]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2ea7dc id 
0xcbb3c3a7!
    Dec 17 17:18:27 nmhd-bs1 kernel: ENBD #1550[104]: enbd_error (3 msgs 
suppressed) error out req cb2ea7dc id 0xcbb3c3a7 from slot 0!
    Dec 17 17:18:27 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15448
    Dec 17 17:18:27 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15440
    Dec 17 17:18:27 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15432
    Dec 17 17:18:27 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15456
    Dec 17 17:18:28 nmhd-bs1 kernel: ENBD #1958[107]: enbd_ack (4 msgs 
suppressed) (3): ignoring ack of req id cbb3c39f which slot lacks
    Dec 17 17:18:28 nmhd-bs1 kernel: ENBD #1629[108]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2f57dc id 
0xcbb3c367!
    Dec 17 17:18:28 nmhd-bs1 kernel: ENBD #1550[108]: enbd_error (3 msgs 
suppressed) error out req cb2f57dc id 0xcbb3c367 from slot 0!
    Dec 17 17:18:28 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15512
    Dec 17 17:18:28 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15504
    Dec 17 17:18:28 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15496
    Dec 17 17:18:28 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15488
    Dec 17 17:18:29 nmhd-bs1 kernel: ENBD #1629[112]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2f5888 id 
0xcbb3c327!
    Dec 17 17:18:29 nmhd-bs1 kernel: ENBD #1550[112]: enbd_error (3 msgs 
suppressed) error out req cb2f5888 id 0xcbb3c327 from slot 0!
    Dec 17 17:18:29 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15576
    Dec 17 17:18:29 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15568
    Dec 17 17:18:29 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15560
    Dec 17 17:18:29 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15552
    Dec 17 17:18:29 nmhd-bs1 kernel: ENBD #1958[112]: enbd_ack (4 msgs 
suppressed) (3): ignoring ack of req id cbb3c33f which slot lacks
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1629[116]: enbd_rollback (3 msgs 
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f227c id 
0xcbb3c2e7!
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1550[116]: enbd_error (3 msgs 
suppressed) error out req cb2f227c id 0xcbb3c2e7 from slot 0!
    Dec 17 17:18:30 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15640
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1629[117]: enbd_rollback (1): 
erroring too old (delay 19s >= timeout 90s) req cb2f2124 id 0xcbb3c2ef!
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1550[117]: enbd_error error out req 
cb2f2124 id 0xcbb3c2ef from slot 1!
    Dec 17 17:18:30 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15632
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1629[118]: enbd_rollback (2): 
erroring too old (delay 19s >= timeout 90s) req cb2f5e94 id 0xcbb3c2f7!
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1550[118]: enbd_error error out req 
cb2f5e94 id 0xcbb3c2f7 from slot 2!
    Dec 17 17:18:30 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15624
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1629[119]: enbd_rollback (3): 
erroring too old (delay 19s >= timeout 90s) req cb2f2328 id 0xcbb3c2ff!
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1550[119]: enbd_error error out req 
cb2f2328 id 0xcbb3c2ff from slot 3!
    Dec 17 17:18:30 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector 
15616
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1958[116]: enbd_ack (3 msgs 
suppressed) (3): ignoring ack of req id cbb3c2ff which slot lacks
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1958[117]: enbd_ack (2): ignoring 
ack of req id cbb3c2f7 which slot lacks
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1958[118]: enbd_ack (1): ignoring 
ack of req id cbb3c2ef which slot lacks
    Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1958[119]: enbd_ack (0): ignoring 
ack of req id cbb3c2e7 which slot lacks
    Dec 17 17:18:31 nmhd-bs1 kernel: md: cannot remove active disk nda from 
md12 ... 
    Dec 17 17:18:41 nmhd-bs1 kernel: md: cannot remove active disk nda from 
md12 ... 
    Dec 17 17:19:36 nmhd-bs1 kernel: md: cannot remove active disk nda from 
md12 ... 
    Dec 17 17:20:29 nmhd-bs1 kernel: md: cannot remove active disk nda from 
md12 ... 
    Dec 17 17:26:22 nmhd-bs1 kernel: md: cannot remove active disk nda from 
md12 ... 
    Dec 17 17:44:13 nmhd-bs1 -- MARK --
    Dec 17 18:04:31 nmhd-bs1 kernel: RAID1 conf printout:
    Dec 17 18:04:31 nmhd-bs1 kernel:  --- wd:1 rd:2
    Dec 17 18:04:31 nmhd-bs1 kernel:  disk 0, wo:0, o:1, dev:dm-5
    Dec 17 18:04:31 nmhd-bs1 kernel:  disk 1, wo:1, o:0, dev:nda
    Dec 17 18:04:31 nmhd-bs1 kernel: RAID1 conf printout:
    Dec 17 18:04:31 nmhd-bs1 kernel:  --- wd:1 rd:2
    Dec 17 18:04:31 nmhd-bs1 kernel:  disk 0, wo:0, o:1, dev:dm-5
    Dec 17 18:44:14 nmhd-bs1 -- MARK --
    Dec 17 19:14:14 nmhd-bs1 -- MARK --
    Dec 17 19:44:14 nmhd-bs1 -- MARK --
    Dec 17 20:14:14 nmhd-bs1 -- MARK --
    Dec 17 20:44:14 nmhd-bs1 -- MARK --
    Dec 17 21:14:14 nmhd-bs1 -- MARK --
    Dec 17 21:14:52 nmhd-bs1 kernel: md: unbind<nda>
    Dec 17 21:14:52 nmhd-bs1 kernel: md: export_rdev(nda)
    Dec 17 21:14:58 nmhd-bs1 kernel: md: bind<nda>
    Dec 17 21:14:58 nmhd-bs1 kernel: RAID1 conf printout:
    Dec 17 21:14:58 nmhd-bs1 kernel:  --- wd:1 rd:2
    Dec 17 21:14:58 nmhd-bs1 kernel:  disk 0, wo:0, o:1, dev:dm-5
    Dec 17 21:14:58 nmhd-bs1 kernel:  disk 1, wo:1, o:1, dev:nda
    Dec 17 21:14:58 nmhd-bs1 kernel: md: syncing RAID array md12
    Dec 17 21:14:58 nmhd-bs1 kernel: md: minimum _guaranteed_ reconstruction 
speed: 1000 KB/sec/disc.
    Dec 17 21:14:58 nmhd-bs1 kernel: md: using maximum available idle IO 
bandwidth (but not more than 200000 KB/sec) for reconstruction.
    Dec 17 21:14:58 nmhd-bs1 kernel: md: using 128k window, over a total of 
3387328 blocks.
    Dec 17 21:14:58 nmhd-bs1 enbd-pwprog: args are <RebuildStarted /dev/md12>
    Dec 17 21:17:37 nmhd-bs1 kernel: md: md12: sync done.
    Dec 17 21:17:41 nmhd-bs1 kernel: RAID1 conf printout:
    Dec 17 21:17:41 nmhd-bs1 kernel:  --- wd:2 rd:2
    Dec 17 21:17:41 nmhd-bs1 kernel:  disk 0, wo:0, o:1, dev:dm-5
    Dec 17 21:17:41 nmhd-bs1 kernel:  disk 1, wo:0, o:1, dev:nda
    Dec 17 21:17:43 nmhd-bs1 enbd-pwprog: args are <RebuildFinished /dev/md12>

Best wishes,
Michael

Gmane