[Systems] Memcached

[David Farning]

Sorry I meant to save instead of send:(

On Mon, Oct 19, 2009 at 10:06 PM, David Farning <dfarning at sugarlabs.org> wrote:
> A couple of time memcache has come up over the last couple day.
> Because we have two services which use it, wiki.sl.o and
> activities.sl.o, I thought that it was worth digging into a bit
> deeper.
>
The big reason for memcache is scalability.  As Bernie and Ivan were
discussing, APC makes more sense for the wiki at our current scale.
The memcache gains starts to happen we we scale to multiple database
servers for the wiki.

A similar situation is also true for a.sl.o.  Memcached does not
currently provide a significant gain because sunjammer has enough free
cache that a high percent of mysql calls are store in the mysql cache.
 We have one cache sitting on top of another.

The gains for memcache happen when the free cache on the database
server starts to get tight and churn.  This is what we saw happening
when memory started getting tight before sunjammer moved to a larger
machine.  One of the tuning variable for memcache is how much memory
to reserve for memcache.

Closely related is the ability move the memcach instance to a separate
machine from the database server.  This allow us to use free memory
where ever it is available.  Rather then using one huge machine for
their dbserver, mozilla is running three mememcache machines with 64GB
each.

Where we are at--
Below is our current memcache status.
The key numbers to look at are  * Hit %: and * Quota %:

This memcache instance now has a hit rate of 67%.  The sweet spot for
large a.sl.o instances, such as AMO, is between 90 and 95%.  If you
look at the graphs at
http://sunjammer.sugarlabs.org/munin/sugarlabs.org/sunjammer.sugarlabs.org-memcached_rates.html
we hit that at peak usage times.  However, due to expiration times the
hit rate drops significantly at non-peak times.

Quota reflects that percent of non-expired content in the cache.  When
I took the snapshot below, the quota was at 81%.  The target goal for
quota is as high as possible with out causing excessive evictions of
current content.  Over the last week we have had only 10 evictions.

Tuning rules of thumb for a.sl.o:
1. Don't worry too much about it until the mysql cache hit rate starts to drop.
2. Aim for 90% to 95% hit rate.
3. Due to content expiration times, reserved memcache which is larger
than necessary is wasted.

Communication moving forward--
I will monitor the memcache status every couple of days.  When the
evictions start to climb I will file a request for resources to this
list asking for more memcache space.  Hopefully, this will create an
abstraction barrier where bernie doesn't have to worry specifically
about memcache.

All he needs to worry about is that every couple of weeks he is going
to get a request additional memcache space:(

david

david

----------------------------------------
Memcache Status
127.0.0.1:11211

    * Gets: 26329246
    * Misses: 12937040
    * Total Gets: 39266286
    * Hit %: 67.0530592071
    * Quota %: 81.7168986425
    * pid: 2225
    * uptime: 575516
    * time: 1255991072
    * version: 1.2.2
    * pointer_size: 64
    * rusage_user: 633.151569
    * rusage_system: 1786.179629
    * curr_items: 174787
    * total_items: 13180829
    * bytes: 438714259
    * curr_connections: 31
    * total_connections: 90174
    * connection_structures: 201
    * cmd_get: 39266286
    * cmd_set: 13180819
    * get_hits: 26329246
    * get_misses: 12937040
    * evictions: 10
    * bytes_read: 36882909102
    * bytes_written: 93295117994
    * limit_maxbytes: 536870912
    * threads: 1
---------------------------