This one took a while to suss out.

Symptoms are:

• Database connections idle for long periods of time.
• Processes start and then stop intermittently.
• Processes report ‘Killed’ when you run them.
• Running ‘dmesg’ from the command line shows the OMM process was called.

If this is happening to you you need to check your logs to see if the OMM (out of memory manager) is getting called. To check this run ‘dmesg’ from a Unix command line and look for the term ‘OMM’. The OM process kicks in when there is not enough memory to satisfy memory requests (malloc) that the system has already OKed. his is a particular problem on Linux as it will optimistically over allocate memory without regard to whether there is any actually available. The OMM process has this horrible habit of killing random processes until there is enough memory available. The processes killed are always user processes but are totally unrelated to whatever caused the memory issue in the first place.

There is a solution though. The solution is embedded in later versions of Linux and its a method to control the overallocation of memory allowed so that the OMM process is not called, instead the process that requested the memory is killed with a segmentation fault instead. It still means some process dies however its better than a random act of killing.

The solution was in the post killing the OOM killer – setting the two vm. variables in sysctl.conf stopped the OMM process kicking in and killing random processes. Looking at the ‘dmseg’ instead of OMM calls I now see seg faults. There is still a problem with overallocation of memory but now the process that requested the memory is killed rather than a random process.

My setting for the sysctl.conf file

vm.overcommit_ratio = 80

vm.overcommit_memory = 2

Which is working fine on an Amazon EC2 instance.