redis集群

redis集群

evobot 1,226 2018-08-22
  1. redis集群介绍
  2. redis集群搭建配置
  3. redis集群操作

redis集群介绍

  • redis集群官方称为redis-cluster,是在redis3.0版本之后开始支持的;
  • redis集群可以实现多个redis节点网络互联、数据共享;
  • 所有的节点都是一主一从或一主多从,其中从不提供服务,仅作为备用;
  • redis集群不支持同时处理多个键(mset/mget),因为redis需要把键均为分布在各个节点上,并发量高的情况下,同时创建键值会降低性能并导致不可预测的行为;
  • 支持在线增加、删除节点;
  • 客户端可以连接任何一个主节点进行读写。

redis集群配置

集群架构

  • redis集群需要ruby2.2以上版本,并且只需要一台机器运行;

  • 架构设置:

    1. 两台机器,分别开启三个redis服务;
    2. A机器上三个端口7000,7002,7004,全部为主
    3. B机器上三个端口7001,7003,7005,全部为从
    4. 两台机器上都要编译安装redis,然后编辑并复制3个不通的redis.conf配置文件,分别设置不同的端口号、dir等参数,还需要增加cluster相关的参数,然后启动6个redis服务,cluster参数如下:
    cluster-enabled yes	#开启集群
    cluster-config-file nodes-6379.conf  #不同的节点文件名也要不同,该文件会在dir配置的目录中自动生成
    cluster-node-timeout 10100
    
    1. 6个redis服务的配置如下:
    port 7000
    bind 192.168.133.130
    daemonize yes
    pidfile /var/run/redis_7000.pid
    dir /data/redis_data/7000
    cluster-enabled yes
    cluster-config-file nodes_7000.conf
    cluster-node-timeout 10100
    appendonly yes
    
    1. 启动redis服务后,进程如下:
    $ ps aux |grep redis
    root       7909  0.1  1.2 147356  2784 ?        Ssl  01:50   0:00 redis-server 192.168.78.131:7001 [cluster]
    root       7914  0.1  1.2 147356  2780 ?        Ssl  01:51   0:00 redis-server 192.168.78.131:7003 [cluster]
    root       7919  0.1  1.2 147356  2780 ?        Ssl  01:51   0:00 redis-server 192.168.78.131:7005 [cluster]
    
    

配置集群

  1. 安装ruby:
  • 方法1:
$ yum groupinstall -y "Development Tools"
$ yum install -y gdbm-devel libdb4-devel libffi-devel libyaml libyaml-devel ncurses-devel openssl-devel readline-devel tcl-devel
$ cd /root/
$ mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
# 下载ruby源码包
$ wget https://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.3.tar.gz -p rpmbuild/SOURCES/
$ wget https://raw.githubusercontent.com/tjinjin/automate-ruby-rpm/master/ruby22x.spec -P rpmbuild/SPECS
$ rpmbuild -bb rpmbuild/SPECS/ruby22x.spec
# 使用yum安装本地rpm软件包
$ yum -y localinstall rpmbuild/RPMS/x86_64/ruby-2.2.3-1.el7.centos.x86_64.rpm
$ gem install redis
  • 方法2:

访问rvm.io,官方提供的安装ruby的方法:

$ gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB

$ curl -sSL https://get.rvm.io | bash -s stable
$ source /etc/profile.d/rvm.sh
$ rvm list known # 查看可以安装的ruby版本
$ rvm install 2.2.10 # 安装所需的版本
$ gem install redis
  1. redis-trib.rb命令:

    $ cp /usr/local/src/redis-4.0.11/src/redis-trib.rb /usr/bin/
    $ redis-trib.rb create --replicas 1 192.168.78.130:7000 192.168.78.130:7002 192.168.78.130:7004 192.168.78.131:7001 192.168.78.131:7003 192.168.78.131:7005
    >>> Creating cluster
    >>> Performing hash slots allocation on 6 nodes...
    Using 3 masters:	# 自动分配master
    192.168.78.130:7000
    192.168.78.131:7001
    192.168.78.130:7002
    # 自动分配slave
    Adding replica 192.168.78.131:7005 to 192.168.78.130:7000
    Adding replica 192.168.78.130:7004 to 192.168.78.131:7001
    Adding replica 192.168.78.131:7003 to 192.168.78.130:7002
    M: aac6fd26e2ec4d8d261a6def8e634d13008fe82c 192.168.78.130:7000
       slots:0-5460 (5461 slots) master
    M: 4b9454aa3c7ffbc88e9e917213a13814ee5488fd 192.168.78.130:7002
       slots:10923-16383 (5461 slots) master
    S: 8bd6d130b50854224bcc016193053c14b364c18c 192.168.78.130:7004
       replicates 901ffea2f83e552e6991711dc3e39d81116f3f61
    M: 901ffea2f83e552e6991711dc3e39d81116f3f61 192.168.78.131:7001
       slots:5461-10922 (5462 slots) master
    S: 14f4947795e4c7b2774ae11bd4692a837eb98f09 192.168.78.131:7003
       replicates 4b9454aa3c7ffbc88e9e917213a13814ee5488fd
    S: 93a1dc5d65820876edfa5549872f35b964ca3bbd 192.168.78.131:7005
       replicates aac6fd26e2ec4d8d261a6def8e634d13008fe82c
    Can I set the above configuration? (type 'yes' to accept):yes
    >>> Nodes configuration updated
    >>> Assign a different config epoch to each node
    >>> Sending CLUSTER MEET messages to join the cluster
    Waiting for the cluster to join.....
    >>> Performing Cluster Check (using node 192.168.78.130:7000)
    M: aac6fd26e2ec4d8d261a6def8e634d13008fe82c 192.168.78.130:7000
       slots:0-5460 (5461 slots) master
       1 additional replica(s)
    S: 93a1dc5d65820876edfa5549872f35b964ca3bbd 192.168.78.131:7005
       slots: (0 slots) slave
       replicates aac6fd26e2ec4d8d261a6def8e634d13008fe82c
    M: 901ffea2f83e552e6991711dc3e39d81116f3f61 192.168.78.131:7001
       slots:5461-10922 (5462 slots) master
       1 additional replica(s)
    S: 8bd6d130b50854224bcc016193053c14b364c18c 192.168.78.130:7004
       slots: (0 slots) slave
       replicates 901ffea2f83e552e6991711dc3e39d81116f3f61
    S: 14f4947795e4c7b2774ae11bd4692a837eb98f09 192.168.78.131:7003
       slots: (0 slots) slave
       replicates 4b9454aa3c7ffbc88e9e917213a13814ee5488fd
    M: 4b9454aa3c7ffbc88e9e917213a13814ee5488fd 192.168.78.130:7002
       slots:10923-16383 (5461 slots) master
       1 additional replica(s)
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
    
    

集群操作

  • redis集群登陆命令行可以使用节点中的任意一个端口,使用-c选项以集群方式登陆,-h指定host:

    $ redis-cli -c -h 192.168.78.130 -p 7000
    
  • 任意一个节点都可以创建或查看key,并且redis会自动分配存储key的节点,并且会打印相应的消息:

    192.168.78.130:7000> set key1 aaa
    -> Redirected to slot [9189] located at 192.168.78.131:7001
    OK
    192.168.78.131:7001> set key2 bbb
    -> Redirected to slot [4998] located at 192.168.78.130:7000
    OK
    # 没有打印保存到哪个节点就表示存在本机
    192.168.78.130:7000> set key3 ccc
    OK
    
    
  • 同样的查看key也会显示相关信息:

    192.168.78.130:7002> get key1
    -> Redirected to slot [9189] located at 192.168.78.131:7001
    "aaa"
    192.168.78.131:7001> get key2
    -> Redirected to slot [4998] located at 192.168.78.130:7000
    "bbb"
    192.168.78.130:7000> get key3
    "eee"
    
    
  • 检测集群状态,使用redis-trib.rb check [ip:port]命令,其中ip和port可以使用集群中任意一个节点:

    $ redis-trib.rb check 192.168.78.130:7000
    >>> Performing Cluster Check (using node 192.168.78.130:7000)
    M: aac6fd26e2ec4d8d261a6def8e634d13008fe82c 192.168.78.130:7000
       slots:0-5460 (5461 slots) master
       1 additional replica(s)
    S: 93a1dc5d65820876edfa5549872f35b964ca3bbd 192.168.78.131:7005
       slots: (0 slots) slave
       replicates aac6fd26e2ec4d8d261a6def8e634d13008fe82c
    M: 901ffea2f83e552e6991711dc3e39d81116f3f61 192.168.78.131:7001
       slots:5461-10922 (5462 slots) master
       1 additional replica(s)
    S: 8bd6d130b50854224bcc016193053c14b364c18c 192.168.78.130:7004
       slots: (0 slots) slave
       replicates 901ffea2f83e552e6991711dc3e39d81116f3f61
    S: 14f4947795e4c7b2774ae11bd4692a837eb98f09 192.168.78.131:7003
       slots: (0 slots) slave
       replicates 4b9454aa3c7ffbc88e9e917213a13814ee5488fd
    M: 4b9454aa3c7ffbc88e9e917213a13814ee5488fd 192.168.78.130:7002
       slots:10923-16383 (5461 slots) master
       1 additional replica(s)
    [OK] All nodes agree about slots configuration.
    >>> Check for open slots...
    >>> Check slots coverage...
    [OK] All 16384 slots covered.
    
    

redis集群管理命令

  • CLUSTER NODES:查看集群节点:

    192.168.78.130:7000> CLUSTER NODES
    93a1dc5d65820876edfa5549872f35b964ca3bbd 192.168.78.131:7005@17005 slave aac6fd26e2ec4d8d261a6def8e634d13008fe82c 0 1535048218000 6 connected
    aac6fd26e2ec4d8d261a6def8e634d13008fe82c 192.168.78.130:7000@17000 myself,master - 0 1535048220000 1 connected 0-5460
    901ffea2f83e552e6991711dc3e39d81116f3f61 192.168.78.131:7001@17001 master - 0 1535048219723 4 connected 5461-10922
    8bd6d130b50854224bcc016193053c14b364c18c 192.168.78.130:7004@17004 slave 901ffea2f83e552e6991711dc3e39d81116f3f61 0 1535048219000 4 connected
    14f4947795e4c7b2774ae11bd4692a837eb98f09 192.168.78.131:7003@17003 slave 4b9454aa3c7ffbc88e9e917213a13814ee5488fd 0 1535048219000 5 connected
    4b9454aa3c7ffbc88e9e917213a13814ee5488fd 192.168.78.130:7002@17002 master - 0 1535048220730 2 connected 10923-16383
    
    

    该命令会标注节点是master还是slave,并且会用一串字符串表示slave对应的master,myself表示本机。

  • CLUSTER INFO:查看集群信息:

    192.168.78.130:7000> CLUSTER INFO
    cluster_state:ok
    cluster_slots_assigned:16384
    cluster_slots_ok:16384
    cluster_slots_pfail:0
    cluster_slots_fail:0
    cluster_known_nodes:6
    cluster_size:3
    cluster_current_epoch:6
    cluster_my_epoch:1
    cluster_stats_messages_ping_sent:960
    cluster_stats_messages_pong_sent:1015
    cluster_stats_messages_sent:1975
    cluster_stats_messages_ping_received:1010
    cluster_stats_messages_pong_received:960
    cluster_stats_messages_meet_received:5
    cluster_stats_messages_received:1975
    
    
  • CLUSTER MEET [ip] [port]:添加新节点,默认添加的新节点都是master节点:

    $ redis-cli -c -h 192.168.78.130 -p 7000
    192.168.78.130:7000> CLUSTER MEET 192.168.78.130 7006
    OK
    192.168.78.130:7000> cluster nodes
    93a1dc5d65820876edfa5549872f35b964ca3bbd 192.168.78.131:7005@17005 slave aac6fd26e2ec4d8d261a6def8e634d13008fe82c 0 1535049063785 6 connected
    # 默认添加为master节点
    7759f139d2b7ca0b4907bc75f101aa8be02d4e4d 192.168.78.130:7006@17006 master - 0 1535049064793 0 connected	
    aac6fd26e2ec4d8d261a6def8e634d13008fe82c 192.168.78.130:7000@17000 myself,master - 0 1535049063000 1 connected 0-5460
    901ffea2f83e552e6991711dc3e39d81116f3f61 192.168.78.131:7001@17001 master - 0 1535049062779 4 connected 5461-10922
    8bd6d130b50854224bcc016193053c14b364c18c 192.168.78.130:7004@17004 slave 901ffea2f83e552e6991711dc3e39d81116f3f61 0 1535049063000 4 connected
    14f4947795e4c7b2774ae11bd4692a837eb98f09 192.168.78.131:7003@17003 slave 4b9454aa3c7ffbc88e9e917213a13814ee5488fd 0 1535049062000 5 connected
    4b9454aa3c7ffbc88e9e917213a13814ee5488fd 192.168.78.130:7002@17002 master - 0 1535049063000 2 connected 10923-16383
    
    192.168.78.130:7000> CLUSTER MEET 192.168.78.131 7007
    OK
    192.168.78.130:7000> cluster nodes
    93a1dc5d65820876edfa5549872f35b964ca3bbd 192.168.78.131:7005@17005 slave aac6fd26e2ec4d8d261a6def8e634d13008fe82c 0 1535049121000 6 connected
    7759f139d2b7ca0b4907bc75f101aa8be02d4e4d 192.168.78.130:7006@17006 master - 0 1535049121635 0 connected
    aac6fd26e2ec4d8d261a6def8e634d13008fe82c 192.168.78.130:7000@17000 myself,master - 0 1535049119000 1 connected 0-5460
    901ffea2f83e552e6991711dc3e39d81116f3f61 192.168.78.131:7001@17001 master - 0 1535049121000 4 connected 5461-10922
    8bd6d130b50854224bcc016193053c14b364c18c 192.168.78.130:7004@17004 slave 901ffea2f83e552e6991711dc3e39d81116f3f61 0 1535049120121 4 connected
    14f4947795e4c7b2774ae11bd4692a837eb98f09 192.168.78.131:7003@17003 slave 4b9454aa3c7ffbc88e9e917213a13814ee5488fd 0 1535049122000 5 connected
    4b9454aa3c7ffbc88e9e917213a13814ee5488fd 192.168.78.130:7002@17002 master - 0 1535049122642 2 connected 10923-16383
    96f1283eb326bd6c3c7e67e0d8e715a56a5e749a 192.168.78.131:7007@17007 master - 0 1535049121334 0 connected
    
    
  • CLUSTER REPLICATE [node_id]将当前节点设置为指定节点的从,需要先登陆到需要设置的节点上,例如前面增加的两个新节点,将7007设置为7006的从节点,首先登陆到7007:

    $ redis-cli -c -h 192.168.78.131 -p 7007
    
    # 复制7006的节点号
    192.168.78.131:7007> CLUSTER REPLICATE 7759f139d2b7ca0b4907bc75f101aa8be02d4e4d
    OK
    192.168.78.131:7007> CLUSTER NODES
    93a1dc5d65820876edfa5549872f35b964ca3bbd 192.168.78.131:7005@17005 slave aac6fd26e2ec4d8d261a6def8e634d13008fe82c 0 1535049357000 1 connected
    7759f139d2b7ca0b4907bc75f101aa8be02d4e4d 192.168.78.130:7006@17006 master - 0 1535049359774 7 connected
    14f4947795e4c7b2774ae11bd4692a837eb98f09 192.168.78.131:7003@17003 slave 4b9454aa3c7ffbc88e9e917213a13814ee5488fd 0 1535049357000 2 connected
    4b9454aa3c7ffbc88e9e917213a13814ee5488fd 192.168.78.130:7002@17002 master - 0 1535049357759 2 connected 10923-16383
    96f1283eb326bd6c3c7e67e0d8e715a56a5e749a 192.168.78.131:7007@17007 myself,slave 7759f139d2b7ca0b4907bc75f101aa8be02d4e4d 0 1535049356000 0 connected
    8bd6d130b50854224bcc016193053c14b364c18c 192.168.78.130:7004@17004 slave 901ffea2f83e552e6991711dc3e39d81116f3f61 0 1535049358767 4 connected
    aac6fd26e2ec4d8d261a6def8e634d13008fe82c 192.168.78.130:7000@17000 master - 0 1535049358000 1 connected 0-5460
    901ffea2f83e552e6991711dc3e39d81116f3f61 192.168.78.131:7001@17001 master - 0 1535049358000 4 connected 5461-10922
    
    
  • CLUSTER FORGET [node_id]:移除一个从节点,注意要移除的节点不能为master,如果想要移除master,可以将其设置为另一个节点的从节点之后再进行删除作业:

    192.168.78.131:7007> CLUSTER FORGET 7759f139d2b7ca0b4907bc75f101aa8be02d4e4d
    (error) ERR Can't forget my master!
    
    # 也不能移除当前节点
    192.168.78.131:7007> CLUSTER FORGET 96f1283eb326bd6c3c7e67e0d8e715a56a5e749a
    (error) ERR I tried hard but I can't forget myself...
    
    $ redis-cli -c -h 192.168.78.130 -p 7000
    192.168.78.130:7000> CLUSTER FORGET 96f1283eb326bd6c3c7e67e0d8e715a56a5e749a
    OK
    
    
  • CLUSTER SAVECONFIG:保存集群配置文件,会保存到cluster-config-file配置项指定的文件中:

    $ ls -l /data/redis7000/
    总用量 12
    -rw-r--r--. 1 root root 151 8月  24 02:11 appendonly.aof
    -rw-r--r--. 1 root root 202 8月  24 02:20 dump.rdb
    -rw-r--r--. 1 root root 915 8月  24 02:44 nodes-7000.conf
    $ redis-cli -c -h 192.168.78.130 -p 7000
    192.168.78.130:7000> CLUSTER SAVECONFIG
    OK
    192.168.78.130:7000>
    $ ls -l /data/redis7000/
    总用量 12
    -rw-r--r--. 1 root root 151 8月  24 02:11 appendonly.aof
    -rw-r--r--. 1 root root 202 8月  24 02:20 dump.rdb
    -rw-r--r--. 1 root root 915 8月  24 02:45 nodes-7000.conf
    
    


# redis