Bart Van Assche | 9 Dec 15:33 2011
Picon

3.1.4: circular locking

Hello,

Has anyone observed the call stack below before ?

device eth0 entered promiscuous mode
scsi3 : FCoE Driver
host3: libfc: Link up on port (000000)

=======================================================
[ INFO: possible circular locking dependency detected ]
3.1.4 #2
-------------------------------------------------------
kworker/0:1/11 is trying to acquire lock:
 (rtnl_mutex){+.+.+.}, at: [<c136b874>] rtnl_lock+0x14/0x20

but task is already holding lock:
 (&fip->ctlr_mutex){+.+...}, at: [<f889f2b0>]
fcoe_ctlr_timer_work+0x3b0/0xa00 [libfcoe]

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #1 (&fip->ctlr_mutex){+.+...}:
       [<c1081cc0>] lock_acquire+0x80/0x110
       [<c142d18c>] mutex_lock_nested+0x6c/0x320
       [<f889eb32>] fcoe_ctlr_link_up+0x22/0x170 [libfcoe]
       [<f885fec6>] fcoe_create+0x3f6/0x528 [fcoe]
       [<f88a1aab>] fcoe_transport_create+0x11b/0x240 [libfcoe]
       [<c1065cad>] param_attr_store+0x3d/0x60
       [<c1065980>] module_attr_store+0x30/0x40
       [<c1181f58>] sysfs_write_file+0xa8/0x110
       [<c112d1ea>] vfs_write+0x9a/0x160
       [<c112d4c2>] sys_write+0x42/0x70
       [<c1436b1f>] sysenter_do_call+0x12/0x38

-> #0 (rtnl_mutex){+.+.+.}:
       [<c10811a8>] __lock_acquire+0x1388/0x1920
       [<c1081cc0>] lock_acquire+0x80/0x110
       [<c142d18c>] mutex_lock_nested+0x6c/0x320
       [<c136b874>] rtnl_lock+0x14/0x20
       [<f885f5ca>] fcoe_update_src_mac+0x2a/0xb0 [fcoe]
       [<f889f56d>] fcoe_ctlr_timer_work+0x66d/0xa00 [libfcoe]
       [<c1062318>] process_one_work+0x178/0x470
       [<c1063b51>] worker_thread+0x131/0x2c0
       [<c1068364>] kthread+0x84/0x90
       [<c1437102>] kernel_thread_helper+0x6/0x10

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&fip->ctlr_mutex);
                               lock(rtnl_mutex);
                               lock(&fip->ctlr_mutex);
  lock(rtnl_mutex);

 *** DEADLOCK ***

3 locks held by kworker/0:1/11:
 #0:  (events){.+.+.+}, at: [<c10622a4>] process_one_work+0x104/0x470
 #1:  ((&fip->timer_work)){+.+...}, at: [<c10622a4>]
process_one_work+0x104/0x470
 #2:  (&fip->ctlr_mutex){+.+...}, at: [<f889f2b0>]
fcoe_ctlr_timer_work+0x3b0/0xa00 [libfcoe]

stack backtrace:
Pid: 11, comm: kworker/0:1 Not tainted 3.1.4 #2
Call Trace:
 [<c142ba86>] ? printk+0x1d/0x1f
 [<c107ec7c>] print_circular_bug+0x1bc/0x290
 [<c10811a8>] __lock_acquire+0x1388/0x1920
 [<c1081cc0>] lock_acquire+0x80/0x110
 [<c136b874>] ? rtnl_lock+0x14/0x20
 [<c142d18c>] mutex_lock_nested+0x6c/0x320
 [<c136b874>] ? rtnl_lock+0x14/0x20
 [<c136b874>] ? rtnl_lock+0x14/0x20
 [<f889f2b0>] ? fcoe_ctlr_timer_work+0x3b0/0xa00 [libfcoe]
 [<c136b874>] rtnl_lock+0x14/0x20
 [<f885f5ca>] fcoe_update_src_mac+0x2a/0xb0 [fcoe]
 [<f889f56d>] fcoe_ctlr_timer_work+0x66d/0xa00 [libfcoe]
 [<c107cf6b>] ? trace_hardirqs_off+0xb/0x10
 [<c106f8bd>] ? local_clock+0x6d/0x70
 [<c1062318>] process_one_work+0x178/0x470
 [<c10622a4>] ? process_one_work+0x104/0x470
 [<f889ef00>] ? fcoe_ctlr_vn_send_claim+0x40/0x40 [libfcoe]
 [<c1063b51>] worker_thread+0x131/0x2c0
 [<c1063a20>] ? manage_workers+0x200/0x200
 [<c1068364>] kthread+0x84/0x90
 [<c10682e0>] ? __init_kthread_worker+0x60/0x60
 [<c1437102>] kernel_thread_helper+0x6/0x10
host3: Assigned Port ID 0092b5

Thanks,

Bart.

Gmane