You are not logged in.

#1 2007-05-20 12:28:28

jcroft
Member
Registered: 2006-12-29
Posts: 8

On memcached memory usage

I've recently switched my sites to use memcached (which has been a HUGE performance increase!), but I have one question I'm hoping someone can answer...

Memcahced seems to use a lot more memory (resident set size on the local machine) than is allocated for its cache. What's worse, it seems to grow and grow and never stop (from what I can tell). For example, I'm currently just giving memcached 4MB of alloted memory (which doesn't sound like much, but if you think about it, that can store a lot of cache items when they're small). However, a quick look at my memory usage shows that memcached is using close to 20MB of RSS memory.

Is this normal?

For the curious, this is how I'm getting these numbers:

Code:

ps -u jcroft -o rss,command | grep 'memcached -d'  | awk '{print $0}{sum+=$1} END {print "\nMemory usage for memcached:", sum/1024, "MB\n"}'
21016 memcached -d -m 4 -l 127.0.0.1 -p 11211
 616 grep memcached -d

Memory usage for memcached: 21.125 MB

Thanks in advance!

Offline

 

#2 2007-05-20 13:25:06

IAIHMB
Member
From: Hudson, Florida.
Registered: 2006-09-19
Posts: 1362

Re: On memcached memory usage

I've been watching your memcached process for a little while now and it's almost like it's leaking memory because it's closing in on 30 MB. I'm not sure what's causing your problem but given the following description upgrading all of our servers to memcached 1.2.2 couldn't hurt:

2007-05-02: Version 1.2.2 released. Memcached can be configured to support multithreading. Big code cleanup with performance improvements. Memcached now collects eviction and per-object-type statistics.

A memory leak was fixed in version 1.2.1 but Danga didn't mention it in their mailing list or website announcements, hopefully the same has happened for version 1.2.2.


-David Sissitka

Offline

 

#3 2007-05-20 15:13:28

jcroft
Member
Registered: 2006-12-29
Posts: 8

Re: On memcached memory usage

Yeah, it's definitely leaking. When I first start memcached it's about 6MB (which seems reasonable given the 4MB allotment), but it just grows and grows as time goes on.

Let me know if you guys upgrade it. In the meantime, I'll just restart memcached manually periodically so it never gets TOO big.

Thanks!

Offline

 

#4 2007-05-21 08:21:52

IAIHMB
Member
From: Hudson, Florida.
Registered: 2006-09-19
Posts: 1362

Re: On memcached memory usage

Remi has upgraded memcached on all servers to version 1.2.2. smile


-David Sissitka

Offline

 

#5 2007-05-21 08:57:45

jcroft
Member
Registered: 2006-12-29
Posts: 8

Re: On memcached memory usage

Thanks, guys. I'm afraid it didn't seem to help with the memory leak, though. sad

Offline

 

#6 2008-02-13 10:49:35

Fuzztrek
Member
Registered: 2008-01-15
Posts: 1

Re: On memcached memory usage

I realize this thread has been dead over over a year now, but I must say that I've also run into this problem.  I've set memcached to run with a limit of 10 mb, yet it creeps up over that amount within a day or two and eventually must push me over my limit for shared hosting 1 because it mysteriously disappears at some point wink

I'm only running a blog, powered by Django, so it's no huge deal (although I'm using memcached as a session backend, so while the blog is still viewable when memcached is down, no one can sign in.)  However, it would be great to find out how to keep memcached within its limits, after all, if -m is ignored then why even bother with that option in the first place?  Perhaps memcached isn't meant for such small caches.  What little I was able to find on the memcached mailing lists seemed to suggest that memcached will go over the limit for non-cache related memory (e.g. connections.) In that same response I also found:

There's one other place where memcached can exceed its memory limit: if a "set" request comes in whose object size requires a fixed chunk size that hasn't been seen yet, memcached will always allocate a 1MB slab for objects of that size, even if the cache has already reached its size limit. That ensures that you don't get bogus "out of memory" errors after filling up your cache with, say, 1000-byte objects, then sending in a 1500-byte object.

From memcached overrides memory limit

I'm not exactly sure what "fixed chunk size" refers to... surely memcached doesn't allocate 1mb to every value of a size that it hasn't encountered yet?

Anyway, I'll try fiddling with cache timeouts and running memcached with the verbose setting to see what's happening before I resort to a periodic CRON job to kill and restart memcached.

Offline

 

#7 2008-02-14 13:25:57

IAIHMB
Member
From: Hudson, Florida.
Registered: 2006-09-19
Posts: 1362

Re: On memcached memory usage

Assuming I understand what Steven's said your memory usage should eventually stabilize so I suggest setting configuring memcached to use say 10 MB of memory and letting it sit for a couple of days. When you fire up memcached I suggest appending " > $HOME/logs/user/memcached.log 2>&1 &" to the command as apposed to using the -d flag so that you can determine how memcached goes down if it goes down.

Once you've let memcached run for a significant period of time you should be able to determine your memory efficiency and once you've determined your memory efficiency you should be able to determine how much memory you need to allocate for memcached for it to use the desired amount of memory.

Hope it helps. smile


-David Sissitka

Offline

 

#8 2008-06-13 18:25:41

qmanic
Member
Registered: 2008-06-13
Posts: 3

Re: On memcached memory usage

I've had this issue as well, so I thought I'd post what I've found.

I'm not exactly sure what "fixed chunk size" refers to... surely memcached doesn't allocate 1mb to every value of a size that it hasn't encountered yet?

This is actually what happens - not an issue when you've got a 2GB cache, bit more of a problem when you're running in Shared1 and trying to stay < 10MB.

This script will show you details of what slabs have been allocated and what's in them. Mine used to look like this:

Code:

 #  Item_Size   Max_age  1MB_pages Count   Full?
  2      60 B    51771 s       1     633      no
  3      76 B    51771 s       1    3739      no
  4      96 B    51771 s       1     309      no
  5     120 B      101 s       1       1      no
  8     240 B    40606 s       1       2      no
  9     300 B     5043 s       1       2      no
 10     376 B    38892 s       1       1      no
 11     472 B       62 s       1       1      no
 12     592 B      101 s       1       1      no
 13     740 B      456 s       1       2      no
 14     928 B      452 s       1       2      no
 15     1.1 kB     101 s       1       1      no
 16     1.4 kB     101 s       1       1      no
 20     3.5 kB   50734 s       1       1      no
 21     4.3 kB   38883 s       1       4      no
 24     8.5 kB   49086 s       1       1      no
 25    10.6 kB   49090 s       1       2      no
 36   123.4 kB     556 s       1       1      no

I had set m=10 but here you can see memcached has allocated 18MB of RAM, but most of the 1MB chunks only have 1 key in them (there's only about 500KB data in there in total). So, I bumped up the growth factor for the item sizes from the default of 1.25 to 8. This means that, for the larger items you end up wasting more memory per item, but because I've got so few items, this is nothing compared to the memory savings from getting rid of all those near-empty slabs.

Code:

[bios@web38 bin]$ memcached -l 127.0.0.1 -m 10 -p 2553 -n 64 -f 8 -d
[bios@web38 bin]$ ./memcached-tool 127.0.0.1:2553 display
  #  Item_Size   Max_age  1MB_pages Count   Full?
  1      96 B      139 s       1     162      no
  2     768 B      109 s       1       6      no
  3     6.0 kB     139 s       1       5      no
  4    48.0 kB      27 s       1       2      no

Which is waaaay better. You'll want to tweak f and n depending on the distribution of the sizes of the items in your cache.

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2005 Rickard Andersson