Memcache 调研
文档:http://code.google.com/p/memcached/wiki/NewStart?tm=6
这位仁兄收集了些文章:
Memcached深度分析 [好文,必读]
memcached server LRU 深入分析[这个针对实际问题阐述memcached内存存储的调整,回头再看Memcached深度分析 ]
Memcached 原理和使用详解(PPT/PDF) [这个写的很好,必读]
Memcached使用点滴 [这个需要对memcache有一定了解之后,打算自己实现client可做借鉴]
memcached全面剖析–PDF总结篇 [very good,日本mixi网站工程师写的,国人翻译的]
api client 列表 http://code.google.com/p/memcached/wiki/Clients
参见http://blog.ureshika.com/archives/753.html
memcached 安装见http://www.ccvita.com/257.html
以下是我的安装命令[需要root权限]:
------------------------------------------------------------------------------------------------------------
#安装libevent
cd /tmp
wget http://memcached.googlecode.com/files/memcached-1.4.13.tar.gz
wget --no-check-certificate https://github.com/downloads/libevent/libevent/libevent-2.0.17-stable.tar.gz
tar zxvf libevent-2.0.17-stable.tar.gz
cd libevent-2.0.17-stable
./configure --prefix=/usr
make
make install
#检查是否安装
ls -al /usr/lib | grep libevent
#安装memcached
tar zxvf memcached-1.4.13.tar.gz
cd memcached-1.4.13
./configure --with-libevent=/usr
make
make install
#测试是否成功安装memcached:
ls -al /usr/local/bin/mem*
#查看12000端口是否被占用
netstat -tl | grep 12000
#启动并将进程号存入memcached.pid
/usr/local/bin/memcached -d -m 10 -u root -l 192.168.0.200 -p 12000 -c 256 -P memcached.pid
#查看配置状态
echo "stats settings" | nc localhost 12000
#查看内存分块状态
echo "stats slabs" | nc localhost 12000
#查看健康状况
echo "stats" | nc localhost 12000
详细参数列表:
-p <num> TCP port number to listen on (default: 11211) -U <num> UDP port number to listen on (default: 11211, 0 is off) -s <file> UNIX socket path to listen on (disables network support) -a <mask> access mask for UNIX socket, in octal (default: 0700) -l <addr> interface to listen on (default: INADDR_ANY, all addresses) <addr> may be specified as host:port. If you don't specify a port number, the value you specified with -p or -U is used. You may specify multiple addresses separated by comma or by using -l multiple times -d run as a daemon -r maximize core file limit -u <username> assume identity of <username> (only when run as root) -m <num> max memory to use for items in megabytes (default: 64 MB) -M return error on memory exhausted (rather than removing items) -c <num> max simultaneous connections (default: 1024) -k lock down all paged memory. Note that there is a limit on how much memory you may lock. Trying to allocate more than that would fail, so be sure you set the limit correctly for the user you started the daemon with (not for -u <username> user; under sh this is done with 'ulimit -S -l NUM_KB'). -v verbose (print errors/warnings while in event loop) -vv very verbose (also print client commands/reponses) -vvv extremely verbose (also print internal state transitions) -h print this help and exit -i print memcached and libevent license -P <file> save PID in <file>, only used with -d option -f <factor> chunk size growth factor (default: 1.25) -n <bytes> minimum space allocated for key+value+flags (default: 48) -L Try to use large memory pages (if available). Increasing the memory page size could reduce the number of TLB misses and improve the performance. In order to get large pages from the OS, memcached will allocate the total item-cache in one large chunk. -D <char> Use <char> as the delimiter between key prefixes and IDs. This is used for per-prefix stats reporting. The default is ":" (colon). If this option is specified, stats collection is turned on automatically; if not, then it may be turned on by sending the "stats detail on" command to the server. -t <num> number of threads to use (default: 4) -R Maximum number of requests per event, limits the number of requests process for a given connection to prevent starvation (default: 20) -C Disable use of CAS -b Set the backlog queue limit (default: 1024) -B Binding protocol - one of ascii, binary, or auto (default) -I Override the size of each slab page. Adjusts max item size (default: 1mb, min: 1k, max: 128m) -o Comma separated list of extended or experimental options - (EXPERIMENTAL) maxconns_fast: immediately close new connections if over maxconns limit - hashpower: An integer multiplier for how large the hash table should be. Can be grown at runtime if not big enough. Set this based on "STAT hash_power_level" before a restart.
注意-f 增量 -n chunck块初始大小[默认1024b] -I slab page大小[不能超过slab大小,1k—128M,默认1M]
------------------------------------------------------------------------------------------------------------
以下摘自: Memcached 原理和使用详解(PPT/PDF)
数据存储方式:Slab Allocation
Slab Alloction 构造图:
Slab Allocator的基本原理是按照预先规定的大小,将分配的内存分割成特定长度的块,以完全解决内存碎片问题。
Slab Allocation的原理相当简单。 将分配的内存分割成各种尺寸的块(chunk),并把尺寸相同的块分成组(chunk的集合)
Slab Classes 分配图
Page:分配给Slab的内存空间,默认是1MB。分配给Slab之后根据slab的大小切分成chunk。
Chunk:用于缓存记录的内存空间。
注意:摘自http://y.jmeye.com/?aid=185
Slab是一个内存块,它是memcached一次申请内存的最小单位。 在启动memcached的时候一般会使用参数-m指定其可用内存,但是并不是在启动的那一刻所有的内存就全部分配出去了,只有在需要的时候才会去申请, 而且每次申请一定是一个slab。Slab的大小固定为1M(1048576 Byte),一个slab由若干个大小相等的chunk组成。每个chunk中都保存了一个item结构体、一对key和value。
注意:
chunk中不仅保持了缓存对象的value,而且保存了缓存对象的key,expire time, flag等详细信息
参见: http://tank.blogs.tkiicpp.com/category/programming/memcache/
Slab Class:特定大小的chunk的组。
memcached根据收到的数据的大小,选择最适合数据大小的slab。
memcached中保存着slab内空闲chunk的列表,根据该列表选择chunk,然后将数据缓存于其中。
Slab Alloction 缺点
这个问题就是,由于分配的是特定长度的内存,因此无法有效利用分配的内存。例如,将100字节的数据缓存到128字节的chunk中,剩余的28字节就浪费了。
以下摘自: http://blog.ureshika.com/archives/750.html
以下摘自:
http://blog.csdn.net/jarfield/article/details/4322953
http://blog.csdn.net/jarfield/article/details/4336035
http://blog.csdn.net/jarfield/article/details/4341819
所有的被发送到memcached的单个命令是完全原子的。如果您针对同一份数据同时发送了一个set命令和一个get命令,它们不会影响对方。它们将被串行化、先后执行。
通常,基于memcached中item的值来修改item,是一件棘手的事情。除非您很清楚自己在做什么,否则请不要做这样的事情。
STAT 1:chunk_size 80
STAT 1:chunks_per_page 13107
STAT 1:total_pages 1
STAT 1:total_chunks 13107
STAT 1:used_chunks 13107
STAT 1:free_chunks 0
STAT 1:free_chunks_end 13107
•在 Memcached 中可以保存的item数据量是没有限制的,只有内存足够
• 最大键长为250字节,大于该长度无法存储,常量代码中KEY_MAX_LENGTH 250 控制
• 单个item最大数据默认是1MB,超过1MB数据不予存储,常量代码POWER_BLOCK 1048576 进行控制,参数-I控制的是slab page大小,因此它不能超过slab大小,slab大小只能通过编译源码来改。
32位系统下Memcached单进程最大使用内存为2G,要使用更多内存,可以分多个端口开启多个Memcached进程或改为64位系统。
-----------------------------------------------------------------------------
它的接口文档见http://code.google.com/p/memcached/wiki/NewCommands
其中有些还没被Memcached-Java-Client实现,比如cas,可能是新加的接口。
memcached提供的接口目标就是简单,其用起来可以说简单,但是不简单的地方是什么呢,当然他的内部机理比较复杂,最主要的是实际应用场景的相关问题。
你做数据缓存,那么哪些数据应该缓存?怎么样缓存?过期策略?
做session集群共享,要不要保证可靠性?
很多时候不该memcached做的事情也让它来做了,这时可考虑下nosql地盘的东东了。
可参考转载的这篇文章:http://blog.ureshika.com/archives/757.html