PHP and Memcached Returning Random Data

This took a long time to track down and only applies in specific circumstances:

• Your process is forking
• You are using persistent memcache connections
• The data you request does not match what you receive, for instance you expect an array and an integer is returned.

The problem is caused by the persistent connection being shared after the process has forked. This is a known issue on database connections which is why you should always stop and restart your database connections after a process forks however this is not a documented ‘feature’ of the memcache connections. Trying to close and reopen the memcache connections doesn’t help either. What is happening is two processes are using the same pipe to talk to memcache, they both request data at roughly the same time however the responses go back to the wrong processes, so process A gets the data meant for B, and process B gets process A’s data.

The best approach I’ve found to this issue so far is:

• Mark the data stored with the key it was stored under.
• When the data is returned check the key returned to ensure its the same as the key requested.
• If there is a mismatch treat it as a cache miss – return false.

This increases your cache misses however it ensures your application works as you coded. Its a simple change, instead of storing the just the data, you add an additional parameter to indicate the key the data was stored under:

Instead of:

function CacheAdd( $Key, $Data , $TTL ){
    // add code to create your memcache connection
    $memcache->add( $Key, $Data , false, $TTL );
}

You use:

function CacheAdd( $Key, $Data , $TTL ){
    // add code to create your memcache connection
    $DataToStore = array( 'Key'=>$Key, 'Data'=>$Data ) ;
    $memcache->add( $Key, $DataToStore , false, $TTL );
}

and then when you retrieve:

function CacheGet($Key){
    // add code to create your memcache connection
    $DataFromStore = $memcahe->get( $Key );
    if ( !isset($DataFromStore['Key']) || $DataFromStore['Key'] != $Key ) {
        return false;
    }
    return $DataFromStore['Data'];
}

Posted in Web Design and Development | 6 Comments »

Jquery Save to Excel II

Slightly more advanced version of my previous script. This script handles if the table doesn’t have an ID which was a problem previously. It also paves the way for multiple export options including Excel, HTML and PDF – whatever your server can handle.

This code is slightly different to previous code as it will add the export buttons to all tables on the page that do not have a class of ‘noExcel’. Run this script from your jQuery ready() function. A simpler version of this code is available at http://www.topsemtips.com/2008/11/save-html-table-to-excel-using-jquery/:

var RandomIDToUse = 1000;
function AddActionsToTable(  )
{
$(" table:not(.noExcel)").each( function(i){
var TableID = this.id ;

if ( $(this).data('ToolBarAdded') === undefined) {
$(this).data('ToolBarAdded',1);
var TableName = this.id ;
if ( this.id == '' || this.id === undefined ){
TableName = "MyRandomID" + RandomIDToUse ;
$(this).attr('id',TableName);
RandomIDToUse++ ;
}
var FormID = 'form' + TableName ;
var SavetoID = 'saveto' + TableName ;
var DataToDisplayName = "datatodisplay" + TableName ;
var SaveToDisk =
"<div class='TableToolBar'>" +
"<form action='/reports/SaveData/SaveToExcel.php' method='post' target='_blank' id='" + FormID + "'" +
"onsubmit='$(\".DataToDisplay\", this ).val( $(\"<div>\").append( $(\"#" + TableName + "\").eq(0).clone() ).html() )'>" +
"<input type='hidden' id='" + DataToDisplayName + "' name='DataToDisplay' class='DataToDisplay' />" +
"<input type='hidden' id='" + SavetoID + "' name='SaveTo' val='' />" +
"</form>" +
"<input  type='image' src='/images/icons/page_excel.png' width='16' height='16' alt='Save to Excel' title='Save to Excel'" +
" onclick='$(\"input:checked\").attr(\"checked\",true); $(\"#" + SavetoID + "\").val(\"Excel\"); $(\"#" + FormID + "\").submit();' />" +
"  " +
"<input  type='image' src='/images/icons/doc_table.png' width='16' height='16' alt='Save to HTML' title='Save to HTML'" +
" onclick='$(\"input:checked\").attr(\"checked\",true); $(\"#" + SavetoID + "\").val(\"HTML\"); $(\"#" + FormID + "\").submit();' />" +
"  " +
"<input  type='image' src='/images/icons/doc_pdf.png' width='16' height='16' alt='Save to PDF' title='Save to PDF'" +
" onclick='$(\"input:checked\").attr(\"checked\",true); $(\"#" + SavetoID + "\").val(\"PDF\"); $(\"#" + FormID + "\").submit();' />" +
"</div>" ;
$(this).after( SaveToDisk ) ;
}
} ) ;

}

Posted in Uncategorized | 9 Comments »

PHP Processes Getting Killed

This one took a while to suss out.

Symptoms are:

• Database connections idle for long periods of time.
• Processes start and then stop intermittently.
• Processes report ‘Killed’ when you run them.
• Running ‘dmesg’ from the command line shows the OMM process was called.

If this is happening to you you need to check your logs to see if the OMM (out of memory manager) is getting called. To check this run ‘dmesg’ from a Unix command line and look for the term ‘OMM’. The OM process kicks in when there is not enough memory to satisfy memory requests (malloc) that the system has already OKed. his is a particular problem on Linux as it will optimistically over allocate memory without regard to whether there is any actually available. The OMM process has this horrible habit of killing random processes until there is enough memory available. The processes killed are always user processes but are totally unrelated to whatever caused the memory issue in the first place.

There is a solution though. The solution is embedded in later versions of Linux and its a method to control the overallocation of memory allowed so that the OMM process is not called, instead the process that requested the memory is killed with a segmentation fault instead. It still means some process dies however its better than a random act of killing.

The solution was in the post killing the OOM killer – setting the two vm. variables in sysctl.conf stopped the OMM process kicking in and killing random processes. Looking at the ‘dmseg’ instead of OMM calls I now see seg faults. There is still a problem with overallocation of memory but now the process that requested the memory is killed rather than a random process.

My setting for the sysctl.conf file

vm.overcommit_ratio = 80

vm.overcommit_memory = 2

Which is working fine on an Amazon EC2 instance.

Posted in Web Design and Development | No Comments »