Kernel Errors After Updating

Hi all,

I have updated my CentOS 6.5 KVM host to kernel 2.6.32-431.23.3 this morning … After 2 hours working, the following kernel error appears and all vm guests goes slowly ….

device monif4 entered promiscuous mode monif: port 5(monif4) entering forwarding state INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message. cgroup D 0000000000000009 0 83 2 0×00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[] __mutex_lock_slowpath+0x13e/0×180
[] ? do_rebuild_sched_domains+0×0/0×50
[] mutex_lock+0x2b/0×50
[] cgroup_lock+0×15/0×20
[] do_rebuild_sched_domains+0×18/0×50
[] worker_thread+0×170/0x2a0
[] ? autoremove_wake_function+0×0/0×40
[] ? worker_thread+0×0/0x2a0
[] kthread+0×96/0xa0
[] child_rip+0xa/0×20
[] ? kthread+0×0/0xa0
[] ? child_rip+0×0/0×20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message. cgroup D 0000000000000009 0 83 2 0×00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[] __mutex_lock_slowpath+0x13e/0×180
[] ? do_rebuild_sched_domains+0×0/0×50
[] mutex_lock+0x2b/0×50
[] cgroup_lock+0×15/0×20
[] do_rebuild_sched_domains+0×18/0×50
[] worker_thread+0×170/0x2a0
[] ? autoremove_wake_function+0×0/0×40
[] ? worker_thread+0×0/0x2a0
[] kthread+0×96/0xa0
[] child_rip+0xa/0×20
[] ? kthread+0×0/0xa0
[] ? child_rip+0×0/0×20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message. cgroup D 0000000000000009 0 83 2 0×00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[] __mutex_lock_slowpath+0x13e/0×180
[] ? do_rebuild_sched_domains+0×0/0×50
[] mutex_lock+0x2b/0×50
[] cgroup_lock+0×15/0×20
[] do_rebuild_sched_domains+0×18/0×50
[] worker_thread+0×170/0x2a0
[] ? autoremove_wake_function+0×0/0×40
[] ? worker_thread+0×0/0x2a0
[] kthread+0×96/0xa0
[] child_rip+0xa/0×20
[] ? kthread+0×0/0xa0
[] ? child_rip+0×0/0×20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message. cgroup D 0000000000000009 0 83 2 0×00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[] __mutex_lock_slowpath+0x13e/0×180
[] ? do_rebuild_sched_domains+0×0/0×50
[] mutex_lock+0x2b/0×50
[] cgroup_lock+0×15/0×20
[] do_rebuild_sched_domains+0×18/0×50
[] worker_thread+0×170/0x2a0
[] ? autoremove_wake_function+0×0/0×40
[] ? worker_thread+0×0/0x2a0
[] kthread+0×96/0xa0
[] child_rip+0xa/0×20
[] ? kthread+0×0/0xa0
[] ? child_rip+0×0/0×20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message. cgroup D 0000000000000009 0 83 2 0×00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[] __mutex_lock_slowpath+0x13e/0×180
[] ? do_rebuild_sched_domains+0×0/0×50
[] mutex_lock+0x2b/0×50
[] cgroup_lock+0×15/0×20
[] do_rebuild_sched_domains+0×18/0×50
[] worker_thread+0×170/0x2a0
[] ? autoremove_wake_function+0×0/0×40
[] ? worker_thread+0×0/0x2a0
[] kthread+0×96/0xa0
[] child_rip+0xa/0×20
[] ? kthread+0×0/0xa0
[] ? child_rip+0×0/0×20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message. cgroup D 0000000000000009 0 83 2 0×00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[] __mutex_lock_slowpath+0x13e/0×180
[] ? do_rebuild_sched_domains+0×0/0×50
[] mutex_lock+0x2b/0×50
[] cgroup_lock+0×15/0×20
[] do_rebuild_sched_domains+0×18/0×50
[] worker_thread+0×170/0x2a0
[] ? autoremove_wake_function+0×0/0×40
[] ? worker_thread+0×0/0x2a0
[] kthread+0×96/0xa0
[] child_rip+0xa/0×20
[] ? kthread+0×0/0xa0
[] ? child_rip+0×0/0×20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message. cgroup D 0000000000000009 0 83 2 0×00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[] __mutex_lock_slowpath+0x13e/0×180
[] ? do_rebuild_sched_domains+0×0/0×50
[] mutex_lock+0x2b/0×50
[] cgroup_lock+0×15/0×20
[] do_rebuild_sched_domains+0×18/0×50
[] worker_thread+0×170/0x2a0
[] ? autoremove_wake_function+0×0/0×40
[] ? worker_thread+0×0/0x2a0
[] kthread+0×96/0xa0
[] child_rip+0xa/0×20
[] ? kthread+0×0/0xa0
[] ? child_rip+0×0/0×20
INFO: task cgroup:83 blocked for more than 120 seconds.
Not tainted 2.6.32-431.23.3.el6.x86_64 #1
“echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message. cgroup D 0000000000000009 0 83 2 0×00000000
ffff880411cb9d60 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 ffff880411cb2aa0
ffff880411cb3058 ffff880411cb9fd8 000000000000fbc8 ffff880411cb3058
Call Trace:
[] __mutex_lock_slowpath+0x13e/0×180
[] ? do_rebuild_sched_domains+0×0/0×50
[] mutex_lock+0x2b/0×50
[] cgroup_lock+0×15/0×20
[] do_rebuild_sched_domains+0×18/0×50
[] worker_thread+0×170/0x2a0
[] ? autoremove_wake_function+0×0/0×40
[] ? worker_thread+0×0/0x2a0
[] kthread+0×96/0xa0
[] child_rip+0xa/0×20
[] ? kthread+0×0/0xa0
[] ? child_rip+0×0/0×20

Uhmm , how can I debug this?? Any idea??