18 Dec 2008 02:45
mkfs.ext3 taking long time on raid1 with nda disk
Michael Rendell <michael <at> cs.mun.ca>
2008-12-18 01:45:20 GMT
2008-12-18 01:45:20 GMT
Hi Peter,
Have noticed that creating a file system on a raid1 device
that uses a enbd device sometimes runs into problems: looks
like it is geting errors writing to the server and taking a
while to recover. Wondering if something is wrong in the
configuration?
The set up is as follows:
/etc/enbd.conf:
module 0 show_errs=1
client mDisklessVars /dev/nda nmhd-bs2 1090 -e -n4 -q -t 0
(pwprog is configured but disabled)
cat /proc/nbdinfo
Device a: Open
[a] State: verify, rw, enabled, validated, show_errs, last error 0,
lives 0, bp 0
[a] Queued: +0R/0W curr (check 0R/0W) +0R/2W max
[a] Buffersize: 266240 (sectors=520, blocks=65)
[a] Blocksize: 4096 (log=12)
[a] Size: 3387392KB
[a] Blocks: 846848
[a] Sockets: 4 (+) (+) (*) (+)
[a] Requested: 0 (0) (0) (0) (0) 0R/0W max 0
[a] Despatched: 0 (0) (0) (0) (0) 0R/0W md5 0W (0
eq, 0 ne, 0 dn)
[a] Errored: 0 (0) (0) (0) (0) 0+0
[a] Pending: 0 (0) (0) (0) (0) 0R/0W+0R/0W
[a] B/s now: 0 (0R+0W)
[a] B/s ave: 0 (0R+0W)
[a] B/s max: 0 (0R+0W)
[a] Spectrum: 100%0
[a] OSpectrum: 100%0
[a] Kthreads: 0 (0 waiting/0 running/0 max)
[a] Cthreads: 4 (+) (+) (+) (+)
[a] Cpids: 4 (5816) (5817) (5818) (5824)
Device b-p: Closed
/bin/echo merge_requests=4 > /proc/nbdinfo
cat /proc/nbdinfo
Device a: Open
[a] State: verify, rw, merge_requests 4, enabled, validated,
show_errs, last error 0, lives 0, bp 0
[a] Queued: +0R/0W curr (check 0R/0W) +0R/2W max
[a] Buffersize: 266240 (sectors=520, blocks=65)
[a] Blocksize: 4096 (log=12)
[a] Size: 3387392KB
[a] Blocks: 846848
[a] Sockets: 4 (+) (+) (*) (+)
[a] Requested: 0 (0) (0) (0) (0) 0R/0W max 0
[a] Despatched: 0 (0) (0) (0) (0) 0R/0W md5 0W (0
eq, 0 ne, 0 dn)
[a] Errored: 0 (0) (0) (0) (0) 0+0
[a] Pending: 0 (0) (0) (0) (0) 0R/0W+0R/0W
[a] B/s now: 0 (0R+0W)
[a] B/s ave: 0 (0R+0W)
[a] B/s max: 0 (0R+0W)
[a] Spectrum: 100%0
[a] OSpectrum: 100%0
[a] Kthreads: 0 (0 waiting/0 running/0 max)
[a] Cthreads: 4 (+) (+) (+) (+)
[a] Cpids: 4 (5816) (5817) (5818) (5824)
Device b-p: Closed
All is OK. Then the raid1 disk is created and a file system put on it:
mdadm --create --assume-clean -n 2 -l raid1 --homehost=nmhd-bs1
--bitmap=internal --write-behind /dev/md12 /dev/vg-system/lv-diskless-vars
--write-mostly /dev/nda
mdadm: array /dev/md12 started.
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md12 : active raid1 nda[1](W) dm-5[0]
3387328 blocks [2/2] [UU]
bitmap: 0/207 pages [0KB], 8KB chunk
time mkfs.ext3 -v -L /diskless-vars /dev/md12 &
[1] 6253
mke2fs 1.39 (29-May-2006)
Filesystem label=/diskless-vars
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
423488 inodes, 846832 blocks
42341 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=868220928
26 block groups
32768 blocks per group, 32768 fragments per group
16288 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information:
(wait a while)
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md12 : active raid1 nda[1](W) dm-5[0]
3387328 blocks [2/2] [UU]
bitmap: 29/207 pages [116KB], 8KB chunk
Then errors on console: end_request: I/O error, dev nda, sector 26888 ...
(see syslog below for details)
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md12 : active raid1 nda[1](W)(F) dm-5[0]
3387328 blocks [2/1] [U_]
bitmap: 29/207 pages [116KB], 8KB chunk
mdadm --remove /dev/md12 /dev/nda
mdadm: hot remove failed for /dev/nda: Device or resource busy
cat /proc/nbdinfo
Device a: Open
[a] State: verify, rw, merge_requests 4, enabled, validated,
show_errs, md5sum, last error 0, lives 0, bp 0
[a] Queued: +0R/127W curr (check 0R/127W) +4R/156W max
[a] Buffersize: 266240 (sectors=520, blocks=65)
[a] Blocksize: 4096 (log=12)
[a] Size: 3387392KB
[a] Blocks: 846848
[a] Sockets: 4 (+) (+) (+) (*)
[a] Requested: 4.9537K (1.25K) (1.24K) (1.22K) (1.23K) 36R/4.927KW
max 1
[a] Despatched: 4.8410K (1.22K) (1.21K) (1.19K) (1.20K) 36R/4.805KW
md5 3.82KW (3.82K eq, 0 ne, 0 dn)
[a] Errored: 120 (30) (30) (30) (30) 120+0
[a] Pending: 4 (1) (1) (1) (1) 0R/4W+0R/0W
[a] B/s now: 20.0K (0R+20.0KW)
[a] B/s ave: 12.0K (0R+12.0KW)
[a] B/s max: 232K (0R+228KW)
[a] Spectrum: 99%1
[a] OSpectrum: 99%1
[a] Kthreads: 0 (0 waiting/0 running/1 max)
[a] Cthreads: 0 (-) (-) (-) (-)
[a] Cpids: 0 (5816) (5817) (5818) (5824)
Device b-p: Closed
Then, 48 minutes later (from time of mkfs) - mkfs finishes:
This filesystem will be automatically checked every 39 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md12 : active raid1 nda[2](W)(F) dm-5[0]
3387328 blocks [2/1] [U_]
bitmap: 15/207 pages [60KB], 8KB chunk
cat /proc/nbdinfo
Device a: Open
[a] State: verify, rw, merge_requests 4, enabled, validated,
show_errs, md5sum, last error 0, lives 0, bp 0
[a] Queued: +0R/0W curr (check 0R/0W) +4R/156W max
[a] Buffersize: 266240 (sectors=520, blocks=65)
[a] Blocksize: 4096 (log=12)
[a] Size: 3387392KB
[a] Blocks: 846848
[a] Sockets: 4 (+) (+) (+) (*)
[a] Requested: 23.218K (5.85K) (5.84K) (5.82K) (5.83K) 36R/23.32KW
max 1
[a] Despatched: 23.246K (5.82K) (5.81K) (5.79K) (5.80K) 36R/23.21KW
md5 22.2KW (22.2K eq, 2 ne, 0 dn)
[a] Errored: 120 (30) (30) (30) (30) 120+0
[a] Pending: 0 (0) (0) (0) (0) 0R/0W+0R/0W
[a] B/s now: 0 (0R+0W)
[a] B/s ave: 8.00K (0R+8.00KW)
[a] B/s max: 232K (0R+228KW)
[a] Spectrum: 99%1
[a] OSpectrum: 99%1
[a] Kthreads: 0 (0 waiting/0 running/1 max)
[a] Cthreads: 4 (+) (+) (+) (+)
[a] Cpids: 4 (5816) (5817) (5818) (5824)
Device b-p: Closed
mdadm --remove /dev/md12 /dev/nda
mdadm: hot removed /dev/nda
mdadm --re-add /dev/md12 /dev/nda
mdadm: re-added /dev/nda
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md12 : active raid1 nda[1](W) dm-5[0]
3387328 blocks [2/1] [U_]
[>....................] recovery = 1.3% (46588/3387328) finish=364.7min
speed=150K/sec
bitmap: 15/207 pages [60KB], 8KB chunk
(3 or 4 minutes later, not 364...)
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md12 : active raid1 nda[1](W) dm-5[0]
3387328 blocks [2/2] [UU]
bitmap: 0/207 pages [0KB], 8KB chunk
Contents of syslog (from creation on):
Dec 17 17:15:26 nmhd-bs1 kernel: ENBD #4246[0]: enbd_set_max_sectors
cannot set max_sectors on ndn because no queue is yet attached
Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #1273[7]: enbd_generic_open
incremented refcnt to 7 on nda
Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #1307[2]: enbd_generic_open DISK
CHECK wanted in open on device nda
Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #5997[2]: enbd_media_changed (1 msgs
suppressed) MEDIA CHANGED called
Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #6057[2]: enbd_media_changed (1 msgs
suppressed) REMOTE CHECK still has REMOTE VALID on nda
Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #5806[1]: enbd_release entered for
inode d3238e30 file 00000000.
Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #5845[1]: enbd_release decremented
refcnt on nda to 6
Dec 17 17:15:39 nmhd-bs1 kernel: ENBD #5877[1]: enbd_release exited for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #1273[8]: enbd_generic_open
incremented refcnt to 7 on nda
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[2]: enbd_release entered for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[2]: enbd_release decremented
refcnt on nda to 6
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[2]: enbd_release exited for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: md: bind<dm-5>
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[3]: enbd_release entered for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[3]: enbd_release decremented
refcnt on nda to 6
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[3]: enbd_release exited for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[4]: enbd_release entered for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[4]: enbd_release decremented
refcnt on nda to 6
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[4]: enbd_release exited for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[5]: enbd_release entered for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[5]: enbd_release decremented
refcnt on nda to 6
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[5]: enbd_release exited for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5806[6]: enbd_release entered for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5845[6]: enbd_release decremented
refcnt on nda to 6
Dec 17 17:15:42 nmhd-bs1 kernel: ENBD #5877[6]: enbd_release exited for
inode d3238e30 file 00000000.
Dec 17 17:15:42 nmhd-bs1 kernel: md: bind<nda>
Dec 17 17:15:42 nmhd-bs1 kernel: raid1: raid set md12 active with 2 out of
2 mirrors
Dec 17 17:15:42 nmhd-bs1 kernel: md12: bitmap file is out of date (0 < 1)
-- forcing full recovery
Dec 17 17:15:42 nmhd-bs1 kernel: md12: bitmap file is out of date, doing
full recovery
Dec 17 17:15:42 nmhd-bs1 kernel: md12: bitmap initialized from disk: read
13/13 pages, set 423416 bits, status: 0
Dec 17 17:15:42 nmhd-bs1 kernel: created bitmap (207 pages) for device
md12
Dec 17 17:15:43 nmhd-bs1 enbd-pwprog: args are <NewArray /dev/md12>
# raid now created; mkfs running...
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1629[0]: enbd_rollback (0):
erroring too old (delay 18s >= timeout 90s) req cb2f9078 id 0x4745ca1f!
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1550[0]: enbd_error error out req
cb2f9078 id 0x4745ca1f from slot 0!
Dec 17 17:18:01 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13792
Dec 17 17:18:01 nmhd-bs1 kernel: raid1: Disk failure on nda, disabling
device.
Dec 17 17:18:01 nmhd-bs1 kernel: Operation continuing on 1 devices
Dec 17 17:18:01 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13800
Dec 17 17:18:01 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13808
Dec 17 17:18:01 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13816
Dec 17 17:18:01 nmhd-bs1 enbd-pwprog: /dev/md12(/dev/nda) has been faulted
and we are waiting for it...
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1273[14]: enbd_generic_open (5 msgs
suppressed) incremented refcnt to 8 on nda
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1307[3]: enbd_generic_open DISK
CHECK wanted in open on device nda
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #5997[3]: enbd_media_changed MEDIA
CHANGED called
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #4744[0]: enbd_get_blk_request took
too long (0 jiffies, 0 tries, timeout 90000 jiffies) to get spare req
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #4946[0]: do_enbd_remote_ioctl could
not make a remote ioctl (0x4004ab2a) request
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #5806[7]: enbd_release entered for
inode d3238e30 file 00000000.
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #5845[7]: enbd_release decremented
refcnt on nda to 7
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #5877[7]: enbd_release exited for
inode d3238e30 file 00000000.
Dec 17 17:18:01 nmhd-bs1 kernel: md: cannot remove active disk nda from
md12 ...
Dec 17 17:18:01 nmhd-bs1 kernel: ENBD #1958[0]: enbd_ack (0): ignoring ack
of req id 4745ca1f which slot lacks
Dec 17 17:18:02 nmhd-bs1 kernel: ENBD #1629[4]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f9d3c id
0x4745c9df!
Dec 17 17:18:02 nmhd-bs1 kernel: ENBD #1550[4]: enbd_error (3 msgs
suppressed) error out req cb2f9d3c id 0x4745c9df from slot 0!
Dec 17 17:18:02 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13856
Dec 17 17:18:02 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13864
Dec 17 17:18:02 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13872
Dec 17 17:18:02 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13880
Dec 17 17:18:02 nmhd-bs1 kernel: ENBD #1958[4]: enbd_ack (3 msgs
suppressed) (0): ignoring ack of req id 4745c9df which slot lacks
Dec 17 17:18:03 nmhd-bs1 kernel: ENBD #1629[8]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2f9480 id
0xee54c99f!
Dec 17 17:18:03 nmhd-bs1 kernel: ENBD #1550[8]: enbd_error (3 msgs
suppressed) error out req cb2f9480 id 0xee54c99f from slot 0!
Dec 17 17:18:03 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13920
Dec 17 17:18:03 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13928
Dec 17 17:18:03 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13936
Dec 17 17:18:03 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13944
Dec 17 17:18:03 nmhd-bs1 kernel: ENBD #1958[8]: enbd_ack (3 msgs
suppressed) (0): ignoring ack of req id ee54c99f which slot lacks
Dec 17 17:18:04 nmhd-bs1 kernel: ENBD #1629[12]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req dcd31124 id
0xee54c95f!
Dec 17 17:18:04 nmhd-bs1 kernel: ENBD #1550[12]: enbd_error (3 msgs
suppressed) error out req dcd31124 id 0xee54c95f from slot 0!
Dec 17 17:18:04 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13984
Dec 17 17:18:04 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
13992
Dec 17 17:18:04 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14000
Dec 17 17:18:04 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14008
Dec 17 17:18:04 nmhd-bs1 kernel: ENBD #1958[12]: enbd_ack (3 msgs
suppressed) (0): ignoring ack of req id ee54c95f which slot lacks
Dec 17 17:18:05 nmhd-bs1 kernel: ENBD #1629[16]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2e7a8c id
0xee54c91f!
Dec 17 17:18:05 nmhd-bs1 kernel: ENBD #1550[16]: enbd_error (3 msgs
suppressed) error out req cb2e7a8c id 0xee54c91f from slot 0!
Dec 17 17:18:05 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14048
Dec 17 17:18:05 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14056
Dec 17 17:18:05 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14064
Dec 17 17:18:05 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14072
Dec 17 17:18:05 nmhd-bs1 kernel: ENBD #1958[16]: enbd_ack (3 msgs
suppressed) (0): ignoring ack of req id ee54c91f which slot lacks
Dec 17 17:18:06 nmhd-bs1 kernel: ENBD #1629[20]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req dcd311d0 id
0xee54c8df!
Dec 17 17:18:06 nmhd-bs1 kernel: ENBD #1550[20]: enbd_error (3 msgs
suppressed) error out req dcd311d0 id 0xee54c8df from slot 0!
Dec 17 17:18:06 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14112
Dec 17 17:18:06 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14120
Dec 17 17:18:06 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14128
Dec 17 17:18:06 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14136
Dec 17 17:18:07 nmhd-bs1 kernel: ENBD #1958[20]: enbd_ack (3 msgs
suppressed) (0): ignoring ack of req id ee54c8df which slot lacks
Dec 17 17:18:07 nmhd-bs1 kernel: ENBD #1629[24]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2f2684 id
0xa664c89f!
Dec 17 17:18:07 nmhd-bs1 kernel: ENBD #1550[24]: enbd_error (3 msgs
suppressed) error out req cb2f2684 id 0xa664c89f from slot 0!
Dec 17 17:18:07 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14176
Dec 17 17:18:07 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14184
Dec 17 17:18:07 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14192
Dec 17 17:18:07 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14200
Dec 17 17:18:08 nmhd-bs1 kernel: ENBD #1958[24]: enbd_ack (3 msgs
suppressed) (0): ignoring ack of req id a664c89f which slot lacks
Dec 17 17:18:08 nmhd-bs1 kernel: ENBD #1629[28]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2f5f40 id
0xa664c847!
Dec 17 17:18:08 nmhd-bs1 kernel: ENBD #1550[28]: enbd_error (3 msgs
suppressed) error out req cb2f5f40 id 0xa664c847 from slot 0!
Dec 17 17:18:08 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14264
Dec 17 17:18:08 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14240
Dec 17 17:18:08 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14248
Dec 17 17:18:08 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14256
Dec 17 17:18:09 nmhd-bs1 kernel: ENBD #1958[28]: enbd_ack (3 msgs
suppressed) (1): ignoring ack of req id a664c85f which slot lacks
Dec 17 17:18:09 nmhd-bs1 kernel: ENBD #1629[32]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2f5a8c id
0xa664c80f!
Dec 17 17:18:09 nmhd-bs1 kernel: ENBD #1550[32]: enbd_error (3 msgs
suppressed) error out req cb2f5a8c id 0xa664c80f from slot 0!
Dec 17 17:18:09 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14320
Dec 17 17:18:09 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14304
Dec 17 17:18:09 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14328
Dec 17 17:18:09 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14312
Dec 17 17:18:10 nmhd-bs1 kernel: ENBD #1958[32]: enbd_ack (3 msgs
suppressed) (1): ignoring ack of req id a664c81f which slot lacks
Dec 17 17:18:10 nmhd-bs1 kernel: ENBD #1629[36]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f552c id
0xa664c7d7!
Dec 17 17:18:10 nmhd-bs1 kernel: ENBD #1550[36]: enbd_error (3 msgs
suppressed) error out req cb2f552c id 0xa664c7d7 from slot 0!
Dec 17 17:18:10 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14376
Dec 17 17:18:10 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14392
Dec 17 17:18:10 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14384
Dec 17 17:18:10 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14368
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1958[36]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id a664c7df which slot lacks
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1629[40]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2ed730 id
0x9074c787!
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1550[40]: enbd_error (3 msgs
suppressed) error out req cb2ed730 id 0x9074c787 from slot 0!
Dec 17 17:18:11 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14456
Dec 17 17:18:11 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14448
Dec 17 17:18:11 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14440
Dec 17 17:18:11 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14432
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1273[15]: enbd_generic_open
incremented refcnt to 8 on nda
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #1307[4]: enbd_generic_open DISK
CHECK wanted in open on device nda
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #5997[4]: enbd_media_changed MEDIA
CHANGED called
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #4744[2]: enbd_get_blk_request (1
msgs suppressed) took too long (0 jiffies, 0 tries, timeout 90000 jiffies) to
get spare req
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #4946[2]: do_enbd_remote_ioctl (1
msgs suppressed) could not make a remote ioctl (0x4004ab2a) request
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #5806[8]: enbd_release entered for
inode d3238e30 file 00000000.
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #5845[8]: enbd_release decremented
refcnt on nda to 7
Dec 17 17:18:11 nmhd-bs1 kernel: ENBD #5877[8]: enbd_release exited for
inode d3238e30 file 00000000.
Dec 17 17:18:11 nmhd-bs1 kernel: md: cannot remove active disk nda from
md12 ...
Dec 17 17:18:12 nmhd-bs1 kernel: ENBD #1958[40]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id 9074c79f which slot lacks
Dec 17 17:18:12 nmhd-bs1 kernel: ENBD #1629[44]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2edc90 id
0x9074c747!
Dec 17 17:18:12 nmhd-bs1 kernel: ENBD #1550[44]: enbd_error (3 msgs
suppressed) error out req cb2edc90 id 0x9074c747 from slot 0!
Dec 17 17:18:12 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14520
Dec 17 17:18:12 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14512
Dec 17 17:18:12 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14504
Dec 17 17:18:12 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14496
Dec 17 17:18:13 nmhd-bs1 kernel: ENBD #1958[44]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id 9074c75f which slot lacks
Dec 17 17:18:13 nmhd-bs1 kernel: ENBD #1629[48]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2ed5d8 id
0x9074c707!
Dec 17 17:18:13 nmhd-bs1 kernel: ENBD #1550[48]: enbd_error (3 msgs
suppressed) error out req cb2ed5d8 id 0x9074c707 from slot 0!
Dec 17 17:18:13 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14584
Dec 17 17:18:13 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14568
Dec 17 17:18:13 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14560
Dec 17 17:18:13 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14576
Dec 17 17:18:14 nmhd-bs1 kernel: ENBD #1958[48]: enbd_ack (3 msgs
suppressed) (2): ignoring ack of req id 9074c71f which slot lacks
Dec 17 17:18:14 nmhd-bs1 kernel: ENBD #1629[52]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2edde8 id
0x9074c6df!
Dec 17 17:18:14 nmhd-bs1 kernel: ENBD #1550[52]: enbd_error (3 msgs
suppressed) error out req cb2edde8 id 0x9074c6df from slot 0!
Dec 17 17:18:14 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14624
Dec 17 17:18:14 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14640
Dec 17 17:18:14 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14632
Dec 17 17:18:14 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14648
Dec 17 17:18:15 nmhd-bs1 kernel: ENBD #1958[52]: enbd_ack (3 msgs
suppressed) (0): ignoring ack of req id 9074c6df which slot lacks
Dec 17 17:18:15 nmhd-bs1 kernel: ENBD #1629[56]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2f2f40 id
0x7a84c68f!
Dec 17 17:18:15 nmhd-bs1 kernel: ENBD #1550[56]: enbd_error (3 msgs
suppressed) error out req cb2f2f40 id 0x7a84c68f from slot 0!
Dec 17 17:18:15 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14704
Dec 17 17:18:15 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14688
Dec 17 17:18:15 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14712
Dec 17 17:18:15 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14696
Dec 17 17:18:16 nmhd-bs1 kernel: ENBD #1958[56]: enbd_ack (3 msgs
suppressed) (1): ignoring ack of req id 7a84c69f which slot lacks
Dec 17 17:18:16 nmhd-bs1 kernel: ENBD #1629[60]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2e7934 id
0x7a84c647!
Dec 17 17:18:16 nmhd-bs1 kernel: ENBD #1550[60]: enbd_error (3 msgs
suppressed) error out req cb2e7934 id 0x7a84c647 from slot 0!
Dec 17 17:18:16 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14776
Dec 17 17:18:16 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14760
Dec 17 17:18:16 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14752
Dec 17 17:18:16 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14768
Dec 17 17:18:17 nmhd-bs1 kernel: ENBD #1958[60]: enbd_ack (3 msgs
suppressed) (2): ignoring ack of req id 7a84c65f which slot lacks
Dec 17 17:18:17 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14832
Dec 17 17:18:17 nmhd-bs1 kernel: ENBD #1629[65]: enbd_rollback (4 msgs
suppressed) (1): erroring too old (delay 18s >= timeout 90s) req cb2e7730 id
0x7a84c61f!
Dec 17 17:18:17 nmhd-bs1 kernel: ENBD #1550[65]: enbd_error (4 msgs
suppressed) error out req cb2e7730 id 0x7a84c61f from slot 1!
Dec 17 17:18:17 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14816
Dec 17 17:18:17 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14840
Dec 17 17:18:17 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14824
Dec 17 17:18:18 nmhd-bs1 kernel: ENBD #1958[64]: enbd_ack (3 msgs
suppressed) (1): ignoring ack of req id 7a84c61f which slot lacks
Dec 17 17:18:18 nmhd-bs1 kernel: ENBD #1629[68]: enbd_rollback (2 msgs
suppressed) (0): erroring too old (delay 15s >= timeout 90s) req cb2f5124 id
0x3394c5c7!
Dec 17 17:18:18 nmhd-bs1 kernel: ENBD #1550[68]: enbd_error (2 msgs
suppressed) error out req cb2f5124 id 0x3394c5c7 from slot 0!
Dec 17 17:18:18 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14904
Dec 17 17:18:18 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14888
Dec 17 17:18:18 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14880
Dec 17 17:18:18 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14896
Dec 17 17:18:19 nmhd-bs1 kernel: ENBD #1958[68]: enbd_ack (3 msgs
suppressed) (2): ignoring ack of req id 7a84c5df which slot lacks
Dec 17 17:18:19 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14952
Dec 17 17:18:19 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14968
Dec 17 17:18:19 nmhd-bs1 kernel: ENBD #1629[74]: enbd_rollback (5 msgs
suppressed) (2): erroring too old (delay 16s >= timeout 90s) req cb2f9be4 id
0x3a94c58f!
Dec 17 17:18:19 nmhd-bs1 kernel: ENBD #1550[74]: enbd_error (5 msgs
suppressed) error out req cb2f9be4 id 0x3a94c58f from slot 2!
Dec 17 17:18:19 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14960
Dec 17 17:18:19 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
14944
Dec 17 17:18:20 nmhd-bs1 kernel: ENBD #1958[72]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id 3a94c59f which slot lacks
Dec 17 17:18:20 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15016
Dec 17 17:18:20 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15032
Dec 17 17:18:20 nmhd-bs1 kernel: ENBD #1629[78]: enbd_rollback (3 msgs
suppressed) (2): erroring too old (delay 17s >= timeout 90s) req de711078 id
0x3a94c54f!
Dec 17 17:18:20 nmhd-bs1 kernel: ENBD #1550[78]: enbd_error (3 msgs
suppressed) error out req de711078 id 0x3a94c54f from slot 2!
Dec 17 17:18:20 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15024
Dec 17 17:18:20 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15008
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1958[76]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id 3a94c55f which slot lacks
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1629[80]: enbd_rollback (1 msgs
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2ea684 id
0x3a94c50f!
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1550[80]: enbd_error (1 msgs
suppressed) error out req cb2ea684 id 0x3a94c50f from slot 0!
Dec 17 17:18:21 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15088
Dec 17 17:18:21 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15080
Dec 17 17:18:21 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15096
Dec 17 17:18:21 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15072
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1273[16]: enbd_generic_open
incremented refcnt to 8 on nda
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #1307[5]: enbd_generic_open DISK
CHECK wanted in open on device nda
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #5997[5]: enbd_media_changed MEDIA
CHANGED called
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #4744[4]: enbd_get_blk_request (1
msgs suppressed) took too long (0 jiffies, 0 tries, timeout 90000 jiffies) to
get spare req
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #4946[4]: do_enbd_remote_ioctl (1
msgs suppressed) could not make a remote ioctl (0x4004ab2a) request
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #4946[5]: do_enbd_remote_ioctl could
not make a remote ioctl (0x4004ab2a) request
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #5806[9]: enbd_release entered for
inode d3238e30 file 00000000.
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #5845[9]: enbd_release decremented
refcnt on nda to 7
Dec 17 17:18:21 nmhd-bs1 kernel: ENBD #5877[9]: enbd_release exited for
inode d3238e30 file 00000000.
Dec 17 17:18:21 nmhd-bs1 kernel: md: cannot remove active disk nda from
md12 ...
Dec 17 17:18:22 nmhd-bs1 kernel: ENBD #1958[80]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id 3a94c51f which slot lacks
Dec 17 17:18:22 nmhd-bs1 kernel: ENBD #1629[84]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f5328 id
0x3a94c4cf!
Dec 17 17:18:22 nmhd-bs1 kernel: ENBD #1550[84]: enbd_error (3 msgs
suppressed) error out req cb2f5328 id 0x3a94c4cf from slot 0!
Dec 17 17:18:22 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15152
Dec 17 17:18:22 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15144
Dec 17 17:18:22 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15160
Dec 17 17:18:22 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15136
Dec 17 17:18:23 nmhd-bs1 kernel: ENBD #1958[84]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id 3a94c4df which slot lacks
Dec 17 17:18:23 nmhd-bs1 kernel: ENBD #1629[88]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req dd357e94 id
0x13a4c487!
Dec 17 17:18:23 nmhd-bs1 kernel: ENBD #1550[88]: enbd_error (3 msgs
suppressed) error out req dd357e94 id 0x13a4c487 from slot 0!
Dec 17 17:18:23 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15224
Dec 17 17:18:23 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15216
Dec 17 17:18:23 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15208
Dec 17 17:18:23 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15200
Dec 17 17:18:24 nmhd-bs1 kernel: ENBD #1958[88]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id 13a4c49f which slot lacks
Dec 17 17:18:24 nmhd-bs1 kernel: ENBD #1629[92]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2e7b38 id
0x13a4c447!
Dec 17 17:18:24 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15288
Dec 17 17:18:24 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15280
Dec 17 17:18:24 nmhd-bs1 kernel: ENBD #1550[94]: enbd_error (5 msgs
suppressed) error out req cb2e7d3c id 0x13a4c457 from slot 2!
Dec 17 17:18:24 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15272
Dec 17 17:18:24 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15264
Dec 17 17:18:25 nmhd-bs1 kernel: ENBD #1958[92]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id 13a4c45f which slot lacks
Dec 17 17:18:25 nmhd-bs1 kernel: ENBD #1629[96]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req dcd31328 id
0x13a4c407!
Dec 17 17:18:25 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15352
Dec 17 17:18:25 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15344
Dec 17 17:18:25 nmhd-bs1 kernel: ENBD #1550[98]: enbd_error (3 msgs
suppressed) error out req cb2f9e94 id 0x13a4c417 from slot 2!
Dec 17 17:18:25 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15336
Dec 17 17:18:25 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15328
Dec 17 17:18:26 nmhd-bs1 kernel: ENBD #1958[96]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id 13a4c41f which slot lacks
Dec 17 17:18:26 nmhd-bs1 kernel: ENBD #1629[100]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f927c id
0x13a4c3e7!
Dec 17 17:18:26 nmhd-bs1 kernel: ENBD #1550[100]: enbd_error (1 msgs
suppressed) error out req cb2f927c id 0x13a4c3e7 from slot 0!
Dec 17 17:18:26 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15384
Dec 17 17:18:26 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15376
Dec 17 17:18:26 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15400
Dec 17 17:18:26 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15392
Dec 17 17:18:27 nmhd-bs1 kernel: ENBD #1958[102]: enbd_ack (5 msgs
suppressed) (3): ignoring ack of req id 13a4c3df which slot lacks
Dec 17 17:18:27 nmhd-bs1 kernel: ENBD #1629[104]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 16s >= timeout 90s) req cb2ea7dc id
0xcbb3c3a7!
Dec 17 17:18:27 nmhd-bs1 kernel: ENBD #1550[104]: enbd_error (3 msgs
suppressed) error out req cb2ea7dc id 0xcbb3c3a7 from slot 0!
Dec 17 17:18:27 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15448
Dec 17 17:18:27 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15440
Dec 17 17:18:27 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15432
Dec 17 17:18:27 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15456
Dec 17 17:18:28 nmhd-bs1 kernel: ENBD #1958[107]: enbd_ack (4 msgs
suppressed) (3): ignoring ack of req id cbb3c39f which slot lacks
Dec 17 17:18:28 nmhd-bs1 kernel: ENBD #1629[108]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 17s >= timeout 90s) req cb2f57dc id
0xcbb3c367!
Dec 17 17:18:28 nmhd-bs1 kernel: ENBD #1550[108]: enbd_error (3 msgs
suppressed) error out req cb2f57dc id 0xcbb3c367 from slot 0!
Dec 17 17:18:28 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15512
Dec 17 17:18:28 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15504
Dec 17 17:18:28 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15496
Dec 17 17:18:28 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15488
Dec 17 17:18:29 nmhd-bs1 kernel: ENBD #1629[112]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 18s >= timeout 90s) req cb2f5888 id
0xcbb3c327!
Dec 17 17:18:29 nmhd-bs1 kernel: ENBD #1550[112]: enbd_error (3 msgs
suppressed) error out req cb2f5888 id 0xcbb3c327 from slot 0!
Dec 17 17:18:29 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15576
Dec 17 17:18:29 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15568
Dec 17 17:18:29 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15560
Dec 17 17:18:29 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15552
Dec 17 17:18:29 nmhd-bs1 kernel: ENBD #1958[112]: enbd_ack (4 msgs
suppressed) (3): ignoring ack of req id cbb3c33f which slot lacks
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1629[116]: enbd_rollback (3 msgs
suppressed) (0): erroring too old (delay 19s >= timeout 90s) req cb2f227c id
0xcbb3c2e7!
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1550[116]: enbd_error (3 msgs
suppressed) error out req cb2f227c id 0xcbb3c2e7 from slot 0!
Dec 17 17:18:30 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15640
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1629[117]: enbd_rollback (1):
erroring too old (delay 19s >= timeout 90s) req cb2f2124 id 0xcbb3c2ef!
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1550[117]: enbd_error error out req
cb2f2124 id 0xcbb3c2ef from slot 1!
Dec 17 17:18:30 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15632
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1629[118]: enbd_rollback (2):
erroring too old (delay 19s >= timeout 90s) req cb2f5e94 id 0xcbb3c2f7!
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1550[118]: enbd_error error out req
cb2f5e94 id 0xcbb3c2f7 from slot 2!
Dec 17 17:18:30 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15624
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1629[119]: enbd_rollback (3):
erroring too old (delay 19s >= timeout 90s) req cb2f2328 id 0xcbb3c2ff!
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1550[119]: enbd_error error out req
cb2f2328 id 0xcbb3c2ff from slot 3!
Dec 17 17:18:30 nmhd-bs1 kernel: end_request: I/O error, dev nda, sector
15616
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1958[116]: enbd_ack (3 msgs
suppressed) (3): ignoring ack of req id cbb3c2ff which slot lacks
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1958[117]: enbd_ack (2): ignoring
ack of req id cbb3c2f7 which slot lacks
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1958[118]: enbd_ack (1): ignoring
ack of req id cbb3c2ef which slot lacks
Dec 17 17:18:30 nmhd-bs1 kernel: ENBD #1958[119]: enbd_ack (0): ignoring
ack of req id cbb3c2e7 which slot lacks
Dec 17 17:18:31 nmhd-bs1 kernel: md: cannot remove active disk nda from
md12 ...
Dec 17 17:18:41 nmhd-bs1 kernel: md: cannot remove active disk nda from
md12 ...
Dec 17 17:19:36 nmhd-bs1 kernel: md: cannot remove active disk nda from
md12 ...
Dec 17 17:20:29 nmhd-bs1 kernel: md: cannot remove active disk nda from
md12 ...
Dec 17 17:26:22 nmhd-bs1 kernel: md: cannot remove active disk nda from
md12 ...
Dec 17 17:44:13 nmhd-bs1 -- MARK --
Dec 17 18:04:31 nmhd-bs1 kernel: RAID1 conf printout:
Dec 17 18:04:31 nmhd-bs1 kernel: --- wd:1 rd:2
Dec 17 18:04:31 nmhd-bs1 kernel: disk 0, wo:0, o:1, dev:dm-5
Dec 17 18:04:31 nmhd-bs1 kernel: disk 1, wo:1, o:0, dev:nda
Dec 17 18:04:31 nmhd-bs1 kernel: RAID1 conf printout:
Dec 17 18:04:31 nmhd-bs1 kernel: --- wd:1 rd:2
Dec 17 18:04:31 nmhd-bs1 kernel: disk 0, wo:0, o:1, dev:dm-5
Dec 17 18:44:14 nmhd-bs1 -- MARK --
Dec 17 19:14:14 nmhd-bs1 -- MARK --
Dec 17 19:44:14 nmhd-bs1 -- MARK --
Dec 17 20:14:14 nmhd-bs1 -- MARK --
Dec 17 20:44:14 nmhd-bs1 -- MARK --
Dec 17 21:14:14 nmhd-bs1 -- MARK --
Dec 17 21:14:52 nmhd-bs1 kernel: md: unbind<nda>
Dec 17 21:14:52 nmhd-bs1 kernel: md: export_rdev(nda)
Dec 17 21:14:58 nmhd-bs1 kernel: md: bind<nda>
Dec 17 21:14:58 nmhd-bs1 kernel: RAID1 conf printout:
Dec 17 21:14:58 nmhd-bs1 kernel: --- wd:1 rd:2
Dec 17 21:14:58 nmhd-bs1 kernel: disk 0, wo:0, o:1, dev:dm-5
Dec 17 21:14:58 nmhd-bs1 kernel: disk 1, wo:1, o:1, dev:nda
Dec 17 21:14:58 nmhd-bs1 kernel: md: syncing RAID array md12
Dec 17 21:14:58 nmhd-bs1 kernel: md: minimum _guaranteed_ reconstruction
speed: 1000 KB/sec/disc.
Dec 17 21:14:58 nmhd-bs1 kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for reconstruction.
Dec 17 21:14:58 nmhd-bs1 kernel: md: using 128k window, over a total of
3387328 blocks.
Dec 17 21:14:58 nmhd-bs1 enbd-pwprog: args are <RebuildStarted /dev/md12>
Dec 17 21:17:37 nmhd-bs1 kernel: md: md12: sync done.
Dec 17 21:17:41 nmhd-bs1 kernel: RAID1 conf printout:
Dec 17 21:17:41 nmhd-bs1 kernel: --- wd:2 rd:2
Dec 17 21:17:41 nmhd-bs1 kernel: disk 0, wo:0, o:1, dev:dm-5
Dec 17 21:17:41 nmhd-bs1 kernel: disk 1, wo:0, o:1, dev:nda
Dec 17 21:17:43 nmhd-bs1 enbd-pwprog: args are <RebuildFinished /dev/md12>
Best wishes,
Michael
RSS Feed