Jul 31 04:06:40 prod-cmp17 kernel: qemu-kvm invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0 Jul 31 04:06:40 prod-cmp17 kernel: qemu-kvm cpuset=vcpu6 mems_allowed=0 Jul 31 04:06:40 prod-cmp17 kernel: Pid: 6433, comm: qemu-kvm Not tainted 2.6.32-358.6.2.el6.x86_64 #1 Jul 31 04:06:40 prod-cmp17 kernel: Call Trace: Jul 31 04:06:40 prod-cmp17 kernel: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? dump_header+0x90/0x1b0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? task_in_mem_cgroup+0xe1/0x120 Jul 31 04:06:40 prod-cmp17 kernel: [] ? oom_kill_process+0x82/0x2a0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? select_bad_process+0x9e/0x120 Jul 31 04:06:40 prod-cmp17 kernel: [] ? mem_cgroup_out_of_memory+0x92/0xb0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? mem_cgroup_handle_oom+0x274/0x2a0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? memcg_oom_wake_function+0x0/0xa0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? __mem_cgroup_try_charge+0x5b9/0x5d0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? mem_cgroup_charge_common+0x87/0xd0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? mem_cgroup_newpage_charge+0x48/0x50 Jul 31 04:06:40 prod-cmp17 kernel: [] ? do_wp_page+0x1a4/0x920 Jul 31 04:06:40 prod-cmp17 kernel: [] ? handle_pte_fault+0x2cd/0xb50 Jul 31 04:06:40 prod-cmp17 kernel: [] ? enqueue_task_fair+0x64/0x100 Jul 31 04:06:40 prod-cmp17 kernel: [] ? check_preempt_curr+0x6d/0x90 Jul 31 04:06:40 prod-cmp17 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? _spin_lock+0x12/0x30 Jul 31 04:06:40 prod-cmp17 kernel: [] ? handle_mm_fault+0x23a/0x310 Jul 31 04:06:40 prod-cmp17 kernel: [] ? __get_user_pages+0x12a/0x430 Jul 31 04:06:40 prod-cmp17 kernel: [] ? get_user_pages+0x49/0x50 Jul 31 04:06:40 prod-cmp17 kernel: [] ? get_user_pages_fast+0x157/0x1c0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? hva_to_pfn+0x33/0x1a0 [kvm] Jul 31 04:06:40 prod-cmp17 kernel: [] ? down_read+0x16/0x30 Jul 31 04:06:40 prod-cmp17 kernel: [] ? mapping_level+0x17b/0x1d0 [kvm] Jul 31 04:06:40 prod-cmp17 kernel: [] ? tdp_page_fault+0x74/0x160 [kvm] Jul 31 04:06:40 prod-cmp17 kernel: [] ? kvm_set_msr_common+0x60a/0xb60 [kvm] Jul 31 04:06:40 prod-cmp17 kernel: [] ? kvm_mmu_page_fault+0x28/0xc0 [kvm] Jul 31 04:06:40 prod-cmp17 kernel: [] ? copy_user_generic_unrolled+0x8e/0xb0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? handle_ept_violation+0x6c/0x140 [kvm_intel] Jul 31 04:06:40 prod-cmp17 kernel: [] ? vmx_handle_exit+0xc3/0x280 [kvm_intel] Jul 31 04:06:40 prod-cmp17 kernel: [] ? kvm_arch_vcpu_ioctl_run+0x4ad/0x10f0 [kvm] Jul 31 04:06:40 prod-cmp17 kernel: [] ? kvm_vcpu_ioctl+0x434/0x580 [kvm] Jul 31 04:06:40 prod-cmp17 kernel: [] ? vfs_ioctl+0x22/0xa0 Jul 31 04:06:40 prod-cmp17 kernel: [] ? do_vfs_ioctl+0x3aa/0x580 Jul 31 04:06:40 prod-cmp17 kernel: [] ? sys_ioctl+0x81/0xa0 Jul 31 04:06:41 prod-cmp17 kernel: [] ? __audit_syscall_exit+0x265/0x290 Jul 31 04:06:41 prod-cmp17 kernel: [] ? system_call_fastpath+0x16/0x1b Jul 31 04:06:41 prod-cmp17 kernel: Task in /libvirt/qemu/i-000010c5 killed as a result of limit of /libvirt/qemu/i-000010c5 Jul 31 04:06:41 prod-cmp17 kernel: memory: usage 36492800kB, limit 36492800kB, failcnt 10319290 Jul 31 04:06:41 prod-cmp17 kernel: memory+swap: usage 48727788kB, limit 9007199254740991kB, failcnt 0 Jul 31 04:06:41 prod-cmp17 kernel: Mem-Info: Jul 31 04:06:41 prod-cmp17 kernel: Node 0 DMA per-cpu: Jul 31 04:06:41 prod-cmp17 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Jul 31 04:06:41 prod-cmp17 kernel: CPU 1: hi: 0, btch: 1 usd: 0 Jul 31 04:06:41 prod-cmp17 kernel: CPU 2: hi: 0, btch: 1 usd: 0 Jul 31 04:06:41 prod-cmp17 kernel: CPU 3: hi: 0, btch: 1 usd: 0 Jul 31 04:06:41 prod-cmp17 kernel: CPU 4: hi: 0, btch: 1 usd: 0 Jul 31 04:06:41 prod-cmp17 kernel: CPU 5: hi: 0, btch: 1 usd: 0 Jul 31 04:06:41 prod-cmp17 kernel: CPU 6: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 7: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 8: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 9: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 10: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 11: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 12: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 13: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 14: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 15: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 16: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 17: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 18: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 19: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 20: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 21: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 22: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: CPU 23: hi: 0, btch: 1 usd: 0 Jul 31 04:06:42 prod-cmp17 kernel: Node 0 DMA32 per-cpu: Jul 31 04:06:42 prod-cmp17 kernel: CPU 0: hi: 186, btch: 31 usd: 158 Jul 31 04:06:42 prod-cmp17 kernel: CPU 1: hi: 186, btch: 31 usd: 184 Jul 31 04:06:42 prod-cmp17 kernel: CPU 2: hi: 186, btch: 31 usd: 183 Jul 31 04:06:42 prod-cmp17 kernel: CPU 3: hi: 186, btch: 31 usd: 190 Jul 31 04:06:42 prod-cmp17 kernel: CPU 4: hi: 186, btch: 31 usd: 160 Jul 31 04:06:42 prod-cmp17 kernel: CPU 5: hi: 186, btch: 31 usd: 182 Jul 31 04:06:42 prod-cmp17 kernel: CPU 6: hi: 186, btch: 31 usd: 163 Jul 31 04:06:42 prod-cmp17 kernel: CPU 7: hi: 186, btch: 31 usd: 157 Jul 31 04:06:43 prod-cmp17 kernel: CPU 8: hi: 186, btch: 31 usd: 181 Jul 31 04:06:43 prod-cmp17 kernel: CPU 9: hi: 186, btch: 31 usd: 160 Jul 31 04:06:43 prod-cmp17 kernel: CPU 10: hi: 186, btch: 31 usd: 181 Jul 31 04:06:43 prod-cmp17 kernel: CPU 11: hi: 186, btch: 31 usd: 156 Jul 31 04:06:43 prod-cmp17 kernel: CPU 12: hi: 186, btch: 31 usd: 166 Jul 31 04:06:43 prod-cmp17 kernel: CPU 13: hi: 186, btch: 31 usd: 159 Jul 31 04:06:43 prod-cmp17 kernel: CPU 14: hi: 186, btch: 31 usd: 184 Jul 31 04:06:43 prod-cmp17 kernel: CPU 15: hi: 186, btch: 31 usd: 164 Jul 31 04:06:43 prod-cmp17 kernel: CPU 16: hi: 186, btch: 31 usd: 184 Jul 31 04:06:43 prod-cmp17 kernel: CPU 17: hi: 186, btch: 31 usd: 183 Jul 31 04:06:43 prod-cmp17 kernel: CPU 18: hi: 186, btch: 31 usd: 164 Jul 31 04:06:43 prod-cmp17 kernel: CPU 19: hi: 186, btch: 31 usd: 167 Jul 31 04:06:43 prod-cmp17 kernel: CPU 20: hi: 186, btch: 31 usd: 169 Jul 31 04:06:43 prod-cmp17 kernel: CPU 21: hi: 186, btch: 31 usd: 159 Jul 31 04:06:43 prod-cmp17 kernel: CPU 22: hi: 186, btch: 31 usd: 176 Jul 31 04:06:43 prod-cmp17 kernel: CPU 23: hi: 186, btch: 31 usd: 155 Jul 31 04:06:43 prod-cmp17 kernel: Node 0 Normal per-cpu: Jul 31 04:06:43 prod-cmp17 kernel: CPU 0: hi: 186, btch: 31 usd: 160 Jul 31 04:06:43 prod-cmp17 kernel: CPU 1: hi: 186, btch: 31 usd: 106 Jul 31 04:06:43 prod-cmp17 kernel: CPU 2: hi: 186, btch: 31 usd: 135 Jul 31 04:06:43 prod-cmp17 kernel: CPU 3: hi: 186, btch: 31 usd: 28 Jul 31 04:06:43 prod-cmp17 kernel: CPU 4: hi: 186, btch: 31 usd: 140 Jul 31 04:06:43 prod-cmp17 kernel: CPU 5: hi: 186, btch: 31 usd: 126 Jul 31 04:06:43 prod-cmp17 kernel: CPU 6: hi: 186, btch: 31 usd: 123 Jul 31 04:06:43 prod-cmp17 kernel: CPU 7: hi: 186, btch: 31 usd: 163 Jul 31 04:06:43 prod-cmp17 kernel: CPU 8: hi: 186, btch: 31 usd: 164 Jul 31 04:06:43 prod-cmp17 kernel: CPU 9: hi: 186, btch: 31 usd: 92 Jul 31 04:06:43 prod-cmp17 kernel: CPU 10: hi: 186, btch: 31 usd: 1 Jul 31 04:06:43 prod-cmp17 kernel: CPU 11: hi: 186, btch: 31 usd: 50 Jul 31 04:06:43 prod-cmp17 kernel: CPU 12: hi: 186, btch: 31 usd: 117 Jul 31 04:06:43 prod-cmp17 kernel: CPU 13: hi: 186, btch: 31 usd: 9 Jul 31 04:06:43 prod-cmp17 kernel: CPU 14: hi: 186, btch: 31 usd: 75 Jul 31 04:06:43 prod-cmp17 kernel: CPU 15: hi: 186, btch: 31 usd: 99 Jul 31 04:06:43 prod-cmp17 kernel: CPU 16: hi: 186, btch: 31 usd: 34 Jul 31 04:06:43 prod-cmp17 kernel: CPU 17: hi: 186, btch: 31 usd: 89 Jul 31 04:06:43 prod-cmp17 kernel: CPU 18: hi: 186, btch: 31 usd: 85 Jul 31 04:06:43 prod-cmp17 kernel: CPU 19: hi: 186, btch: 31 usd: 37 Jul 31 04:06:43 prod-cmp17 kernel: CPU 20: hi: 186, btch: 31 usd: 179 Jul 31 04:06:43 prod-cmp17 kernel: CPU 21: hi: 186, btch: 31 usd: 87 Jul 31 04:06:43 prod-cmp17 kernel: CPU 22: hi: 186, btch: 31 usd: 181 Jul 31 04:06:43 prod-cmp17 kernel: CPU 23: hi: 186, btch: 31 usd: 180 Jul 31 04:06:43 prod-cmp17 kernel: active_anon:15442420 inactive_anon:1708947 isolated_anon:0 Jul 31 04:06:43 prod-cmp17 kernel: active_file:9781 inactive_file:6255 isolated_file:0 Jul 31 04:06:43 prod-cmp17 kernel: unevictable:6356 dirty:3 writeback:0 unstable:0 Jul 31 04:06:43 prod-cmp17 kernel: free:6716678 slab_reclaimable:12259 slab_unreclaimable:470254 Jul 31 04:06:43 prod-cmp17 kernel: mapped:6596 shmem:84 pagetables:45604 bounce:0 Jul 31 04:06:43 prod-cmp17 kernel: Node 0 DMA free:15696kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15304kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 31 04:06:43 prod-cmp17 kernel: lowmem_reserve[]: 0 2980 96910 96910 Jul 31 04:06:44 prod-cmp17 kernel: Node 0 DMA32 free:761244kB min:2076kB low:2592kB high:3112kB active_anon:697808kB inactive_anon:15212kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3051888kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:220kB slab_unreclaimable:13408kB kernel_stack:24kB pagetables:1844kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 31 04:06:44 prod-cmp17 kernel: lowmem_reserve[]: 0 0 93930 93930 Jul 31 04:06:44 prod-cmp17 kernel: Node 0 Normal free:26089772kB min:65492kB low:81864kB high:98236kB active_anon:61071872kB inactive_anon:6820576kB active_file:39124kB inactive_file:25020kB unevictable:25424kB isolated(anon):0kB isolated(file):0kB present:96184320kB mlocked:9072kB dirty:12kB writeback:0kB mapped:26384kB shmem:336kB slab_reclaimable:48816kB slab_unreclaimable:1867608kB kernel_stack:7504kB pagetables:180572kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 31 04:06:44 prod-cmp17 kernel: lowmem_reserve[]: 0 0 0 0 Jul 31 04:06:44 prod-cmp17 kernel: Node 0 DMA: 0*4kB 2*8kB 2*16kB 1*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15696kB Jul 31 04:06:44 prod-cmp17 kernel: Node 0 DMA32: 5207*4kB 2670*8kB 1677*16kB 1036*32kB 748*64kB 547*128kB 418*256kB 280*512kB 224*1024kB 24*2048kB 3*4096kB = 761244kB Jul 31 04:06:44 prod-cmp17 kernel: Node 0 Normal: 7640*4kB 504424*8kB 200459*16kB 66357*32kB 46855*64kB 32460*128kB 17805*256kB 6615*512kB 1557*1024kB 0*2048kB 0*4096kB = 26089648kB Jul 31 04:06:44 prod-cmp17 kernel: 1721282 total pagecache pages Jul 31 04:06:44 prod-cmp17 kernel: 1704554 pages in swap cache Jul 31 04:06:44 prod-cmp17 kernel: Swap cache stats: add 43398730, delete 41694176, find 10407108/13258331 Jul 31 04:06:44 prod-cmp17 kernel: Free swap = 0kB Jul 31 04:06:44 prod-cmp17 kernel: Total swap = 14678008kB Jul 31 04:06:45 prod-cmp17 kernel: 25165808 pages RAM Jul 31 04:06:45 prod-cmp17 kernel: 404624 pages reserved Jul 31 04:06:45 prod-cmp17 kernel: 2912097 pages shared Jul 31 04:06:45 prod-cmp17 kernel: 16687393 pages non-shared Jul 31 04:06:45 prod-cmp17 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Jul 31 04:06:45 prod-cmp17 kernel: [ 6422] 107 6422 6830369 4723771 5 0 0 qemu-kvm Jul 31 04:06:45 prod-cmp17 kernel: Memory cgroup out of memory: Kill process 6422 (qemu-kvm) score 666 or sacrifice child Jul 31 04:06:45 prod-cmp17 kernel: Killed process 6422, UID 107, (qemu-kvm) total-vm:27321476kB, anon-rss:18890492kB, file-rss:4592kB Jul 31 04:06:45 prod-cmp17 kernel: Kill process 6424 (vhost-6422) sharing same memory Jul 31 04:06:48 prod-cmp17 tgtd: conn_close(101) connection closed, 0xdc5808 1