forum.excito.org

Bubba community forum
https://forum.excito.org/

Out of Memory errors

https://forum.excito.org/viewtopic.php?t=2283

Page 1 of 1

Out of Memory errors

Posted: 19 Apr 2010, 22:53
by giorgio
Hi,

I seem to regularly get out of memory errors in the syslog.
It becomes apparent when my network fails and I have to reboot the bubba.
The power switch (pushed with a pen) never works and I have to pull the plug out.

The syslog displays stuff like the following:

Code: Select all

Apr 20 06:23:21 bubba ftd: Tracker announce failed: tracker error
Apr 20 06:24:46 bubba ftd: Tracker announce failed: tracker error
Apr 20 06:26:27 bubba ftd: Tracker announce failed: tracker error
Apr 20 06:26:54 bubba /USR/SBIN/CRON[10149]: (root) CMD (test -x /usr/lib/web-admin/notify-dispatcher.pl && /usr/lib/web-admin/notify-dispatcher.pl)
Apr 20 06:26:54 bubba /USR/SBIN/CRON[10150]: (root) CMD (test -x /usr/bin/php && /usr/bin/php /usr/share/horde3/scripts/alarms.php)
Apr 20 06:26:54 bubba /USR/SBIN/CRON[10151]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Apr 20 06:31:05 bubba /USR/SBIN/CRON[10167]: (root) CMD (test -x /usr/lib/web-admin/notify-dispatcher.pl && /usr/lib/web-admin/notify-dispatcher.pl)
Apr 20 06:31:10 bubba /USR/SBIN/CRON[10168]: (root) CMD (test -x /usr/bin/php && /usr/bin/php /usr/share/horde3/scripts/alarms.php)
Apr 20 06:36:01 bubba /USR/SBIN/CRON[10179]: (root) CMD (test -x /usr/bin/php && /usr/bin/php /usr/share/horde3/scripts/alarms.php)
Apr 20 06:36:01 bubba /USR/SBIN/CRON[10180]: (root) CMD (test -x /usr/lib/web-admin/notify-dispatcher.pl && /usr/lib/web-admin/notify-dispatcher.pl)
Apr 20 06:39:11 bubba kernel: __ratelimit: 1 callbacks suppressed
Apr 20 06:39:25 bubba kernel: php invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Apr 20 06:39:31 bubba kernel: Call Trace:
Apr 20 06:39:31 bubba kernel: [c353dc70] [c0008334] 0xc0008334 (unreliable)
Apr 20 06:39:31 bubba kernel: [c353dca0] [c005fa3c] 0xc005fa3c
Apr 20 06:39:31 bubba kernel: [c353dcd0] [c0060090] 0xc0060090
Apr 20 06:39:31 bubba kernel: [c353dd10] [c0060168] 0xc0060168
Apr 20 06:39:31 bubba kernel: [c353dd30] [c006328c] 0xc006328c
Apr 20 06:39:31 bubba kernel: [c353ddc0] [c007e87c] 0xc007e87c
Apr 20 06:39:31 bubba kernel: [c353de00] [c007e974] 0xc007e974
Apr 20 06:39:31 bubba kernel: [c353de40] [c0073690] 0xc0073690
Apr 20 06:39:31 bubba kernel: [c353de90] [c00172a0] 0xc00172a0
Apr 20 06:39:31 bubba kernel: [c353df40] [c00119d4] 0xc00119d4
Apr 20 06:39:31 bubba kernel: --- Exception: 301 at 0x1028a638
Apr 20 06:39:31 bubba kernel:     LR = 0x1028a044
Apr 20 06:39:31 bubba kernel: Mem-Info:
Apr 20 06:39:31 bubba kernel: DMA per-cpu:
Apr 20 06:39:31 bubba kernel: CPU    0: hi:   90, btch:  15 usd:  69
Apr 20 06:39:31 bubba kernel: active_anon:22973 inactive_anon:23123 isolated_anon:544
Apr 20 06:39:31 bubba kernel:  active_file:1897 inactive_file:3039 isolated_file:161
Apr 20 06:39:31 bubba kernel:  unevictable:0 dirty:14 writeback:2901 unstable:0
Apr 20 06:39:31 bubba kernel:  free:643 slab_reclaimable:1359 slab_unreclaimable:3768
Apr 20 06:39:31 bubba kernel:  mapped:2365 shmem:2068 pagetables:4062 bounce:0
Apr 20 06:39:31 bubba kernel: DMA free:2572kB min:2036kB low:2544kB high:3052kB active_anon:91892kB inactive_anon:92492kB active_file:7588kB inactive_file:12156kB unevictable:0kB isolated(anon):2176kB isolated(file):644kB present:260096kB mlocked:0kB dirty:56kB writeback:11604kB mapped:9460kB shmem:8272kB slab_reclaimable:5436kB slab_unreclaimable:15072kB kernel_stack:4656kB pagetables:16248kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:864 all_unreclaimable? no
Apr 20 06:39:31 bubba kernel: lowmem_reserve[]: 0 0 0
Apr 20 06:39:31 bubba kernel: DMA: 377*4kB 37*8kB 36*16kB 6*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2572kB
Apr 20 06:39:31 bubba kernel: 17981 total pagecache pages
Apr 20 06:39:31 bubba kernel: 10873 pages in swap cache
Apr 20 06:39:31 bubba kernel: Swap cache stats: add 6378201, delete 6367328, find 326612/1083029
Apr 20 06:39:31 bubba kernel: Free swap  = 76kB
Apr 20 06:39:31 bubba kernel: Total swap = 1156672kB
Apr 20 06:39:31 bubba kernel: 65536 pages RAM
Apr 20 06:39:31 bubba kernel: 1581 pages reserved
Apr 20 06:39:31 bubba kernel: 53504 pages shared
Apr 20 06:39:31 bubba kernel: 56337 pages non-shared
Apr 20 06:39:31 bubba kernel: Out of memory: kill process 2222 (mysqld_safe) score 11793 or a child
Apr 20 06:39:31 bubba kernel: Killed process 2267 (mysqld)
Apr 20 06:39:31 bubba kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Apr 20 06:39:31 bubba kernel: Call Trace:
Apr 20 06:39:31 bubba kernel: [c76edc70] [c0008334] 0xc0008334 (unreliable)
Apr 20 06:39:31 bubba kernel: [c76edca0] [c005fa3c] 0xc005fa3c
Apr 20 06:39:31 bubba kernel: [c76edcd0] [c0060090] 0xc0060090
Apr 20 06:39:33 bubba kernel: [c76edd10] [c0060168] 0xc0060168
Apr 20 06:39:58 bubba kernel: [c76edd30] [c006328c] 0xc006328c
Apr 20 06:39:58 bubba kernel: [c76eddc0] [c007e87c] 0xc007e87c
Apr 20 06:39:58 bubba kernel: [c76ede00] [c007e974] 0xc007e974
Apr 20 06:39:58 bubba kernel: [c76ede40] [c0073690] 0xc0073690
Apr 20 06:39:58 bubba kernel: [c76ede90] [c00172a0] 0xc00172a0
Apr 20 06:39:58 bubba kernel: [c76edf40] [c00119d4] 0xc00119d4
Apr 20 06:39:58 bubba kernel: --- Exception: 301 at 0xfebbe14
Apr 20 06:39:59 bubba kernel:     LR = 0xfebbd20
Apr 20 06:39:59 bubba kernel: Mem-Info:
Apr 20 06:39:59 bubba kernel: DMA per-cpu:
Apr 20 06:39:59 bubba kernel: CPU    0: hi:   90, btch:  15 usd:  79
Apr 20 06:39:59 bubba kernel: active_anon:23037 inactive_anon:23154 isolated_anon:95
Apr 20 06:39:59 bubba kernel:  active_file:1906 inactive_file:3652 isolated_file:163
Apr 20 06:40:01 bubba kernel:  unevictable:0 dirty:0 writeback:2961 unstable:0
Apr 20 06:40:01 bubba kernel:  free:603 slab_reclaimable:1359 slab_unreclaimable:3701
Apr 20 06:40:01 bubba kernel:  mapped:2400 shmem:2104 pagetables:4062 bounce:0
Apr 20 06:40:01 bubba kernel: DMA free:2412kB min:2036kB low:2544kB high:3052kB active_anon:92148kB inactive_anon:92616kB active_file:7624kB inactive_file:14608kB unevictable:0kB isolated(anon):380kB isolated(file):652kB present:260096kB mlocked:0kB dirty:0kB writeback:11836kB mapped:9600kB shmem:8416kB slab_reclaimable:5436kB slab_unreclaimable:14804kB kernel_stack:4416kB pagetables:16248kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
Apr 20 06:40:01 bubba kernel: lowmem_reserve[]: 0 0 0
Apr 20 06:40:01 bubba kernel: DMA: 217*4kB 93*8kB 38*16kB 6*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2412kB
Apr 20 06:40:01 bubba kernel: 18238 total pagecache pages
Apr 20 06:40:01 bubba kernel: 10475 pages in swap cache
Apr 20 06:40:01 bubba kernel: Swap cache stats: add 6378331, delete 6367856, find 326612/1083042
Apr 20 06:40:01 bubba kernel: Free swap  = 8kB
Apr 20 06:40:01 bubba kernel: Total swap = 1156672kB
Apr 20 06:40:01 bubba kernel: 65536 pages RAM
Apr 20 06:40:01 bubba kernel: 1581 pages reserved
Apr 20 06:40:01 bubba kernel: 53234 pages shared
Apr 20 06:40:01 bubba kernel: 56676 pages non-shared
Apr 20 06:40:01 bubba kernel: Out of memory: kill process 7611 (apache2) score 8336 or a child
Apr 20 06:40:28 bubba kernel: Killed process 7611 (apache2)
Apr 20 06:40:28 bubba /USR/SBIN/CRON[10197]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm)
Apr 20 06:40:33 bubba /USR/SBIN/CRON[10200]: (root) CMD (test -x /usr/lib/web-admin/notify-dispatcher.pl && /usr/lib/web-admin/notify-dispatcher.pl)
Apr 20 06:40:33 bubba /USR/SBIN/CRON[10201]: (root) CMD (test -x /usr/bin/php && /usr/bin/php /usr/share/horde3/scripts/alarms.php)
Apr 20 06:46:31 bubba /USR/SBIN/CRON[10224]: (root) CMD (test -x /usr/lib/web-admin/notify-dispatcher.pl && /usr/lib/web-admin/notify-dispatcher.pl)
Apr 20 06:46:47 bubba /USR/SBIN/CRON[10227]: (root) CMD (test -x /usr/bin/php && /usr/bin/php /usr/share/horde3/scripts/alarms.php)
Apr 20 06:51:42 bubba mysqld_safe[10235]: Number of processes running now: 0
Apr 20 06:51:51 bubba kernel: php invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Apr 20 06:51:57 bubba kernel: Call Trace:
Apr 20 06:51:57 bubba kernel: [cd14dc70] [c0008334] 0xc0008334 (unreliable)
Apr 20 06:51:57 bubba kernel: [cd14dca0] [c005fa3c] 0xc005fa3c
Apr 20 06:51:57 bubba kernel: [cd14dcd0] [c0060090] 0xc0060090
Apr 20 06:51:57 bubba kernel: [cd14dd10] [c0060168] 0xc0060168
Apr 20 06:51:57 bubba kernel: [cd14dd30] [c006328c] 0xc006328c
Apr 20 06:51:57 bubba kernel: [cd14ddc0] [c007e87c] 0xc007e87c
Apr 20 06:51:57 bubba kernel: [cd14de00] [c007e974] 0xc007e974
Apr 20 06:51:57 bubba kernel: [cd14de40] [c0073690] 0xc0073690
Apr 20 06:51:57 bubba kernel: [cd14de90] [c00172a0] 0xc00172a0
Apr 20 06:51:57 bubba kernel: [cd14df40] [c00119d4] 0xc00119d4
Apr 20 06:51:57 bubba kernel: --- Exception: 301 at 0x48008a58
Apr 20 06:51:57 bubba kernel:     LR = 0x48008b9c
Apr 20 06:51:57 bubba kernel: Mem-Info:
Apr 20 06:51:57 bubba kernel: DMA per-cpu:
Apr 20 06:51:57 bubba kernel: CPU    0: hi:   90, btch:  15 usd:  77
Apr 20 06:51:57 bubba kernel: active_anon:24162 inactive_anon:24080 isolated_anon:224
Apr 20 06:51:57 bubba kernel:  active_file:1359 inactive_file:2061 isolated_file:196
Apr 20 06:51:57 bubba kernel:  unevictable:0 dirty:187 writeback:1286 unstable:0
Apr 20 06:52:01 bubba kernel:  free:572 slab_reclaimable:1284 slab_unreclaimable:3687
Apr 20 06:52:01 bubba kernel:  mapped:1685 shmem:3080 pagetables:4083 bounce:0
Apr 20 06:52:01 bubba kernel: DMA free:2288kB min:2036kB low:2544kB high:3052kB active_anon:96648kB inactive_anon:96320kB active_file:5436kB inactive_file:8244kB unevictable:0kB isolated(anon):896kB isolated(file):784kB present:260096kB mlocked:0kB dirty:748kB writeback:5144kB mapped:6740kB shmem:12320kB slab_reclaimable:5136kB slab_unreclaimable:14748kB kernel_stack:4464kB pagetables:16332kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:288 all_unreclaimable? no
Apr 20 06:52:07 bubba kernel: lowmem_reserve[]: 0 0 0
Apr 20 06:52:20 bubba kernel: DMA: 230*4kB 65*8kB 39*16kB 7*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2288kB
Apr 20 06:52:22 bubba kernel: 14527 total pagecache pages
Apr 20 06:52:22 bubba kernel: 7944 pages in swap cache
Apr 20 06:52:22 bubba kernel: Swap cache stats: add 6698496, delete 6690552, find 333349/1126475
Apr 20 06:52:22 bubba kernel: Free swap  = 60kB
Apr 20 06:52:22 bubba kernel: Total swap = 1156672kB
Apr 20 06:52:22 bubba kernel: 65536 pages RAM
Apr 20 06:52:28 bubba kernel: 1581 pages reserved
Apr 20 06:52:28 bubba kernel: 41421 pages shared
Apr 20 06:52:28 bubba kernel: 58691 pages non-shared
Apr 20 06:52:28 bubba kernel: Out of memory: kill process 7616 (apache2) score 8336 or a child
Apr 20 06:52:28 bubba kernel: Killed process 7616 (apache2)
Apr 20 06:52:28 bubba kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Apr 20 06:52:28 bubba kernel: Call Trace:
I dont really know from the log what is going on.
Any help appreciated.

Full syslog http://dl.dropbox.com/u/1141178/syslog

Cheers
George

Re: Out of Memory errors

Posted: 20 Apr 2010, 16:44
by tor
Hi George,

This sounds strange. You should not get these errors since there should be plenty of swap space if everything is ok.

Could you please give me the output of the commands run on the command line via ssh:

Code: Select all

cat /proc/swaps
and

Code: Select all

cat /etc/fstab
/Tor

Re: Out of Memory errors

Posted: 20 Apr 2010, 16:55
by giorgio
Hi Tor,

Here are the outputs below. I had to pull the plug again this morning....

bubba ~: cat /proc/swaps
Filename Type Size Used Priority
/dev/sda3 partition 1156672 1732 -1

bubba ~: cat /etc/fstab
/dev/sda1 / ext3 noatime,defaults 0 1
/dev/mapper/bubba-storage /home ext3 defaults 0 2
/dev/sda3 none swap sw 0 0
usbfs /proc/bus/usb usbfs defaults 0 0
/proc /proc proc defaults 0 0

Cheers
George

Re: Out of Memory errors

Posted: 26 Jun 2010, 20:55
by booskunk
I ran into the same problem. My system has plenty of swap:

Code: Select all

% free
             total       used       free     shared    buffers     cached
Mem:        255812     222152      33660          0      15280      70468
-/+ buffers/cache:     136404     119408
Swap:      1156672     111808    1044864
The first message was 4 days before:

Jun 21 02:00:56 bubba kernel: squeezecenter-s invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Jun 21 02:01:15 bubba kernel: squeezecenter-s invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0

Perhaps squeezecenter has a memory leak. If squeezecenter is the problem, then how about
changing the launcher script to use "ulimit" to restrict its memory usage. Having just
squeezecenter die is much better than having the whole system die.
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited