5분 안에 구축하는 Redis-Sentinel

이전 글에서 [5분 안에 구축하는 Redis(레디스)] 레디스에 대해 알아 보았습니다

이번 시간에는 Redis Sentinel 에 대해 알아 봅시다!

Redis Sentinel 은 장애 발생시 운영 서비스에 영향 없도록(=고가용성) 레디스 마스터 모니터링 및 자동 장애 극복 조치 (=Fail-over) 해주는 서비스 입니다.

아래와 같은 아키텍쳐로 구성 합니다.

정상 운영시

위와 같은 아키텍쳐로 구성시 한대 서버 장애가 나도 정상적인 운영이 가능합니다.

장애 감지시 sentinel 끼리 투표를 하는데 일정 투표수(=쿼럼::quorum) 이상이 장애 서버라고 동의하면 fail over 작업을(마스터 승격) 진행합니다 (쿼럼이 3일시, fail over 불가)

한대 장애시

만약, 죽은 서버를 복구하여 서비스 투입시 아래와 같습니다.(기존 SLAVE 가 MASTER 로 변경 되지 않습니다)

장애 서버 복구시

참 쉽죠잉

레디스를 설치 해봅시다 ! (OS MAC 기준)

// 만약 mac 에 wget 이 설치 되어 있지 않다면 Homebrew를 통해 설치
$brew install wget
$ mkdir redis
$ cd redis
$ wget http://download.redis.io/releases/redis-5.0.6.tar.gz
$ tar xzf redis-5.0.6.tar.gz
$ cd redis-5.0.6
$ make // 소스 컴파일(=소스 파일을 실행 가능한 형태로 만들어 준다)
$ sudo make install // /usr/local/bin에 redis-server, redis-cli 등 실행 파일이 복사
// redis 서버 실행 (오류 안뜨고 잘되면 성공!)
$ redis-5.0.6/src/redis-server redis.conf
or
$ redis-server redis_6382.conf

위에 아키텍쳐대로 구성을 하기에는 가상머신을 띄워서 해야하므로 번거로우니 아래 처럼 로컬 PC에 모두 띄워보자 ! (테스트 하기에는 충분하다)

# 레디스 서버를 3대(master, slave, slave) 띄워야 하므로 conf 파일을 3개, 센티널도 3개 이므로 conf 파일 3개를 각각 만든다. 

$ cd redis-5.0.6
// 레디스 관련 로그는 logs 파일 안에 저장 예정.
$ mkdir logs
$ cp redis.conf redis_6382.conf
$ cp redis.conf redis_6383.conf
$ cp redis.conf redis_6384.conf
$ cp sentinel.conf sentinel_5000.conf
$ cp sentinel.conf sentinel_5001.conf
$ cp sentinel.conf sentinel_5002.conf

# 설정

- 공통 수정
<redis_*.conf>
# 포트 설정
port 각자포트
# 백그라운드에서 시작하도록 설정
daemonize yes
# log 파일 남도록 설정
logfile logs/redis_[각자포트].log

- 슬레이브 서버 설정
<redis_6383.conf, redis_6384.conf>
# 데이터 복제 관련 설정
slaveof 127.0.0.1 6382
replicaof 127.0.0.1 6382

- 공통 센티널 수정
<sentinel_*.conf>
# 포트 설정
port 각자포트
# 백그라운드에서 시작하도록 설정
daemonize yes
# log 파일 남도록 설정
dir "/Users/hongyoolee/redis/redis-5.0.6/"
logfile "logs/redis_[각자포트].log"
# 감시할 마스터 정보 및 쿼럼(quorum) 설정
sentinel monitor mymaster 127.0.0.1 6382 2
# 다운 되었다고 인지하는 시간 (3초)
- 마스터 서버에 정기적으로 PING을 보내는데, 이 시간 동안 응답이 없으면 다운된 것으로 판단하고 장애조치(failover) 작업을 시작합니다
sentinel down-after-milliseconds mymaster 3000

# 실행

# 실행 
1. redis 먼저 3개 실행
src/redis-server redis_[포트].conf
2. sentinel 3개 실행
src/redis-sentinel sentinel_[포트].conf

# 클라이언트 포트 별로 접속 (센티널도 같다)
$ src/redis-cli -p 6382
$ src/redis-cli -p 5000

# 명령어 (마지막 줄에 master는 127.0.0.1:6382 이고, slave 는 2대, sentinel 은 3대로 구성 되어있다는 정보가 나옴)

# 센티널에서 정보보기
127.0.0.1:5000> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=127.0.0.1:6382,slaves=2,sentinels=3

 

그럼 서비스 구성을 해볼까요 ?

:: github.com/works-code/redis-sentinel

works-code/redis-sentinel

# 라이브러리 정의 (이전 포스트를 보고 오셨다면 이 부분은 이전 포스트와 동일 합니다.)

    // lettuce 사용시
    //compile 'org.springframework.boot:spring-boot-starter-data-redis'
    
    // jedis 사용시
    compile ('org.springframework.boot:spring-boot-starter-data-redis') {
        exclude(group: 'io.lettuce', module: 'lettuce-core')
    }
    compile group: 'redis.clients', name: 'jedis'

# 설정 (이전 포스트를 보고 오셨다면 redisTemplate 정의 안하고 기존에 만든 RedisService 이용하셔도 되며, redisConnectionFactory는 아래처럼 변경 하셔야 합니다)

- 단일 레디스 서버 구성시에는 센티널 서버 ip, port를 정의 해줬다면 센티널 구성시에는 센티널 ip, port, 마스터명 만 정의해주면 서비스가 센티널에 붙어서 마스터 정보를 가져온다(장애시 마스터가 다른 서버로 승격 되면 승격된 마스터 정보를 알아서 가져온다.)

package com.code.configuration;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.connection.RedisSentinelConfiguration;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.StringRedisSerializer;

@Configuration
public class RedisConfiguration {
    // lettuce 사용시
    /*@Bean
    public RedisConnectionFactory redisConnectionFactory(){
        RedisSentinelConfiguration redisSentinelConfiguration = new RedisSentinelConfiguration()
                .master("mymaster")
                .sentinel("localhost",5000)
                .sentinel("localhost",5001)
                .sentinel("localhost",5002);
        LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(redisSentinelConfiguration);
        return lettuceConnectionFactory;
    }*/

    // jedis 사용시
    @Bean
    public RedisConnectionFactory redisConnectionFactory(){
        RedisSentinelConfiguration redisSentinelConfiguration = new RedisSentinelConfiguration()
                .master("mymaster")
                .sentinel("localhost",5000)
                .sentinel("localhost",5001)
                .sentinel("localhost",5002);
        JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory(redisSentinelConfiguration);
        return jedisConnectionFactory;
    }

    @Bean
    public RedisTemplate<String, String> redisTemplate(RedisConnectionFactory redisConnectionFactory){
        RedisTemplate<String, String> redisTemplate = new RedisTemplate<>();
        redisTemplate.setConnectionFactory(redisConnectionFactory);
        redisTemplate.setKeySerializer(new StringRedisSerializer());
        redisTemplate.setValueSerializer(new StringRedisSerializer());
        return redisTemplate;
    }
}

# VO (레디스 캐시 키, 값을 받고 응답하기 위한 클래스)

package com.code.vo;

import lombok.Data;

@Data
public class RedisInfo {

    public String key = "";
    public String value = "";

}

# 레디스 서비스 (캐시 등록을 쉽게 하기위해 만든 서비스로 이전 포스트 보고 오셨다면 이전 포스트에서 사용한 서비스 사용해도 됩니다)

package com.code.service;

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Component;

@Slf4j
@Component
public class RedisService {

    @Autowired
    private RedisTemplate<String, String> redisTemplate;

    // set이 아니기에 키가 존재하면 값을 overwrite 함.
    public void addKey(String key, String value){
        try{
            redisTemplate.opsForValue().set(key, value);
        }catch (Exception e){
            log.error("### Redis Set Key Error !!! ::: {}", e.getMessage());
        }
    }

    public String getValue(String key){
        String value = "";
        try{
            if(redisTemplate.hasKey(key)){
                value = redisTemplate.opsForValue().get(key);
            }
        }catch (Exception e){
            log.error("### Redis Set Key Error !!! ::: {}", e.getMessage());
        }
        return value;
    }
}

# 컨트롤러 (캐시 등록, 삭제 API 정의)

package com.code.controller;

import com.code.service.RedisService;
import com.code.vo.RedisInfo;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class RedisController {

    @Autowired
    private RedisService redisService;

	// 캐시 등록
    @RequestMapping(value = "/register", method = RequestMethod.POST, produces = "application/json; charset=utf8")
    public Object register(@RequestBody RedisInfo redisInfo){
        redisService.addKey(redisInfo.getKey(), redisInfo.getValue());
        return redisInfo;
    }

	// 캐시에 저장된 key의 값 가져오기
    @RequestMapping(value = "/get", method = RequestMethod.POST, produces = "application/json; charset=utf8")
    public Object get(@RequestBody RedisInfo redisInfo){
        String value = redisService.getValue(redisInfo.getKey());
        redisInfo.setValue(value);
        return redisInfo;
    }
}

# Junit (테스트 코드 :: 1초마다 계속적으로 캐시 가져오는 API 호출하도록 하여 실제 운영되고 있는 서비스 처럼 구현 한다)

package com.code;

import com.code.vo.RedisInfo;
import kong.unirest.HttpResponse;
import kong.unirest.JsonNode;
import kong.unirest.Unirest;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;

@Slf4j
@SpringBootTest
class RedisSentinelApplicationTests {

    @Test
    void 테스트() {
        // 계속 반복
        while (true){
            try {
                Thread.sleep(1000); // 1초 마다 호출

                RedisInfo redisInfo = new RedisInfo();
                redisInfo.setKey("key");
                redisInfo.setValue("hello_3");
                HttpResponse<JsonNode> response = Unirest.post("http://localhost:8080/get")
                        .header("Content-Type", "application/json")
                        .body(redisInfo)
                        .asJson();

                log.info("### 테스트 결과 => status : {} | response : {}", response.getStatus() , response.getBody().getObject());

            }catch (Exception e){
                log.error("### 테스트 에러 발생 => {}", e.getMessage());
            }
        }
    }

}

 

그럼 테스트 해보겠습니다!

테스트는 아래와 같이 진행합니다 ㅎㅎ

- 사전 준비

1. redis 서버 3대, redis sentinel 3개 모두 기동

2. 서비스 기동 (lettuce library 사용)

3. 테스트 코드 실행 (1초마다 캐시 값 얻어오는 API 호출)

- 테스트 케이스

1. 마스터 다운 : 약 5초간 서비스 안됨 (다운 감지 시간 + fail-over시간)

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트)

// 20초에 연결 실패 한뒤, 25초에 다시 연결됨
2021-04-05 21:19:20.764  INFO 18928 --- [ecutorLoop-1-11] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was /127.0.0.1:6382
2021-04-05 21:19:20.768  WARN 18928 --- [ioEventLoop-6-4] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [127.0.0.1/<unresolved>:6382]: Connection refused: /127.0.0.1:6382

2021-04-05 21:19:25.666  INFO 18928 --- [ioEventLoop-6-2] i.l.core.protocol.ReconnectionHandler    : Reconnected to 127.0.0.1/<unresolved>:6383

- 클라이언트 (Junit 으로 API 계속 호출하는 곳)

// 20초에 호출한 뒤 연결이 끊어지지 않고 있다가 결과 리턴후, 25초(마스터 승격 완료)에 다시 붙음.
2021-04-05 21:19:20.019  INFO 21025 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}

2021-04-05 21:19:25.667  INFO 21025 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:19:26.671  INFO 21025 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:19:27.678  INFO 21025 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:19:28.684  INFO 21025 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:19:29.691  INFO 21025 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:19:30.698  INFO 21025 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}

- 마스터로 승격된 Redis-server 서비스 (6383)

// 설정한 값 3초 간 계속적으로 마스터로 연결시도 하다가 fail over 진행
// 21031:S > 21031:M 변경된 것을 알 수 있다. (PID:역할)
// 총 연결 서비스 안되는 시간 = 다운되었다고인지하는시간 + failover시간 = 3초 + 2초 = 총 5초 소요
21031:S 05 Apr 2021 21:19:20.884 * Connecting to MASTER 127.0.0.1:6382
21031:S 05 Apr 2021 21:19:20.885 * MASTER <-> REPLICA sync started
21031:S 05 Apr 2021 21:19:20.885 # Error condition on socket for SYNC: Connection refused
21031:S 05 Apr 2021 21:19:21.894 * Connecting to MASTER 127.0.0.1:6382
21031:S 05 Apr 2021 21:19:21.895 * MASTER <-> REPLICA sync started
21031:S 05 Apr 2021 21:19:21.895 # Error condition on socket for SYNC: Connection refused
21031:S 05 Apr 2021 21:19:22.904 * Connecting to MASTER 127.0.0.1:6382
21031:S 05 Apr 2021 21:19:22.905 * MASTER <-> REPLICA sync started
21031:S 05 Apr 2021 21:19:22.905 # Error condition on socket for SYNC: Connection refused
21031:S 05 Apr 2021 21:19:23.915 * Connecting to MASTER 127.0.0.1:6382
21031:S 05 Apr 2021 21:19:23.915 * MASTER <-> REPLICA sync started
21031:S 05 Apr 2021 21:19:23.915 # Error condition on socket for SYNC: Connection refused

- 승격(failover)
21031:M 05 Apr 2021 21:19:24.032 # Setting secondary replication ID to 4f6feb8a8e99713b21e6aa4bd1ea6ab7a3dfd59c, valid up to offset: 1286313. New replication ID is 857a41df37b16f56fdf05c9346b28baf045b93cf
21031:M 05 Apr 2021 21:19:24.033 * Discarding previously cached master state.
21031:M 05 Apr 2021 21:19:24.033 * MASTER MODE enabled (user request from 'id=3 addr=127.0.0.1:49911 fd=7 name=sentinel-08380ba6-cmd age=33 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=154 qbuf-free=32614 obl=36 oll=0 omem=0 events=r cmd=exec')
21031:M 05 Apr 2021 21:19:24.036 # CONFIG REWRITE executed with success.
21031:M 05 Apr 2021 21:19:25.478 * Replica 127.0.0.1:6384 asks for synchronization
21031:M 05 Apr 2021 21:19:25.478 * Partial resynchronization request from 127.0.0.1:6384 accepted. Sending 419 bytes of backlog starting from offset 1286313.

 

2. 마스터 기동 : 서비스 이상 없음, 구 마스터(6383)가 MASTER > SLAVE 로 변경 되고, 새 마스터(6383)가 구 마스터(6382)에게 데이터 복제 진행

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트) : 로그 변화 없음

- 클라이언트 (Junit 으로 API 계속 호출하는 곳) : 로그 변화 없음

- 새 마스터 Redis-server 서비스 (6383)

21031:M 05 Apr 2021 21:24:05.034 * Replica 127.0.0.1:6382 asks for synchronization
21031:M 05 Apr 2021 21:24:05.035 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'b167e913fb260122a13a1e87bc2d776e550628f6', my replication IDs are '857a41df37b16f56fdf05c9346b28baf045b93cf' and '4f6feb8a8e99713b21e6aa4bd1ea6ab7a3dfd59c')
21031:M 05 Apr 2021 21:24:05.035 * Starting BGSAVE for SYNC with target: disk
21031:M 05 Apr 2021 21:24:05.036 * Background saving started by pid 21107
21107:C 05 Apr 2021 21:24:05.037 * DB saved on disk
21031:M 05 Apr 2021 21:24:05.076 * Background saving terminated with success
// 구 마스터 서버인 6382 에게 데이터 복제함
21031:M 05 Apr 2021 21:24:05.076 * Synchronization with replica 127.0.0.1:6382 succeeded

- 구 마스터 Redis-server 서비스 (6382)

21103:M 05 Apr 2021 21:23:53.924 * Increased maximum number of open files to 10032 (it was originally set to 256).
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 5.0.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6382
 |    `-._   `._    /     _.-'    |     PID: 21103
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

21103:M 05 Apr 2021 21:23:53.926 # Server initialized
21103:M 05 Apr 2021 21:23:53.926 * DB loaded from disk: 0.000 seconds
21103:M 05 Apr 2021 21:23:53.926 * Ready to accept connections
// 처음에 마스터로 떴다가 슬레이브로 변경됨.(21103:M > 21103:S)
// 새 마스터 서버(6383)로 부터 데이터를 복제 받음
21103:S 05 Apr 2021 21:24:04.040 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
21103:S 05 Apr 2021 21:24:04.040 * REPLICAOF 127.0.0.1:6383 enabled (user request from 'id=3 addr=127.0.0.1:51597 fd=7 name=sentinel-96b8dbd3-cmd age=11 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=148 qbuf-free=32620 obl=36 oll=0 omem=0 events=r cmd=exec')
21103:S 05 Apr 2021 21:24:04.043 # CONFIG REWRITE executed with success.
21103:S 05 Apr 2021 21:24:05.032 * Connecting to MASTER 127.0.0.1:6383
21103:S 05 Apr 2021 21:24:05.033 * MASTER <-> REPLICA sync started
21103:S 05 Apr 2021 21:24:05.033 * Non blocking connect for SYNC fired the event.
21103:S 05 Apr 2021 21:24:05.033 * Master replied to PING, replication can continue...
21103:S 05 Apr 2021 21:24:05.034 * Trying a partial resynchronization (request b167e913fb260122a13a1e87bc2d776e550628f6:1).
21103:S 05 Apr 2021 21:24:05.036 * Full resync from master: 857a41df37b16f56fdf05c9346b28baf045b93cf:1341375
21103:S 05 Apr 2021 21:24:05.037 * Discarding previously cached master state.
21103:S 05 Apr 2021 21:24:05.076 * MASTER <-> REPLICA sync: receiving 196 bytes from master
21103:S 05 Apr 2021 21:24:05.076 * MASTER <-> REPLICA sync: Flushing old data
21103:S 05 Apr 2021 21:24:05.077 * MASTER <-> REPLICA sync: Loading DB in memory
21103:S 05 Apr 2021 21:24:05.077 * MASTER <-> REPLICA sync: Finished with success

3. 센티널 1대 다운 : 서비스 이상 없음, 다른 레디스 서버도 변화 없음

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트) : 로그 변화 없음

- 클라이언트 (Junit 으로 API 계속 호출하는 곳) : 로그 변화 없음

- 센티널 (5001, 5002) 

23286:X 05 Apr 2021 21:30:41.459 # +sdown sentinel 22dc08c4977c01fa46c9e256df111de813e15d7c 127.0.0.1 5000 @ mymaster 127.0.0.1 6383

 

 

4. 마스터 다운 : 5초간 서비스 오류 발생함, 마스터가 죽고 나서야 서버에서 센티널의 죽음을 인지함

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트)

// 마스터와 센티널의 죽음을 인지하고 계속적으로 연결 시도함
2021-04-05 21:34:41.459  INFO 18928 --- [ecutorLoop-1-10] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was /127.0.0.1:6383
2021-04-05 21:34:41.461  WARN 18928 --- [ioEventLoop-6-3] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:41.465  WARN 18928 --- [ioEventLoop-6-5] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [127.0.0.1/<unresolved>:6383]: Connection refused: /127.0.0.1:6383
2021-04-05 21:34:41.562  WARN 18928 --- [ioEventLoop-6-6] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:41.662  WARN 18928 --- [ioEventLoop-6-9] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:41.761  WARN 18928 --- [oEventLoop-6-12] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:41.862  WARN 18928 --- [ioEventLoop-6-3] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:41.958  WARN 18928 --- [ioEventLoop-6-6] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:42.062  WARN 18928 --- [ioEventLoop-6-9] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:42.262  WARN 18928 --- [oEventLoop-6-12] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:42.562  WARN 18928 --- [ioEventLoop-6-3] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:43.162  WARN 18928 --- [ioEventLoop-6-6] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:44.262  WARN 18928 --- [ioEventLoop-6-9] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:34:46.362  WARN 18928 --- [oEventLoop-6-12] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
// 새 마스터(6384)에 연결됨
2021-04-05 21:34:46.365  INFO 18928 --- [ioEventLoop-6-2] i.l.core.protocol.ReconnectionHandler    : Reconnected to 127.0.0.1/<unresolved>:6384

- 클라이언트 (Junit 으로 API 계속 호출하는 곳)

2021-04-05 21:34:40.566  INFO 21180 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
// 5초간 연결된 상태로 대기하다가 연결됨
2021-04-05 21:34:46.366  INFO 21180 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:34:47.375  INFO 21180 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:34:48.383  INFO 21180 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:34:49.387  INFO 21180 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}

- 마스터로 승격된 Redis-server 서비스 (6384) : 3초간 연결 시도 한뒤 fail-over 진행

5. 마스터 기동 : 서비스 이상 없음, 구 마스터(6383)가 MASTER > SLAVE 로 변경 되고, 새 마스터(6384)가 구 마스터(6383)에게 데이터 복제 진행

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트) : 로그 변화 없음

- 클라이언트 (Junit 으로 API 계속 호출하는 곳) : 로그 변화 없음

- 캐시 서버 (redis) : 새 마스터(6384)가 구 마스터(6383)에게 데이터 복제 진행

6. 센티널 1대 다운 : 서비스 이상 없음, 다른 레디스 서버도 변화 없음

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트) : 로그 변화 없음

- 클라이언트 (Junit 으로 API 계속 호출하는 곳) : 로그 변화 없음

- 캐시 서버 (redis) : 로그 변화 없음

7. 마스터 다운 : 서비스 계속적으로 안됨, 쿼럼을 2로 설정 하였는데 살아있는 서버가 1대라 의사결정이 불가능하여 fail-over 할 수 없음.

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트)

// 센티널, 마스터 연결 시도함
2021-04-05 21:41:31.886  INFO 18928 --- [xecutorLoop-1-9] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was /127.0.0.1:6384
2021-04-05 21:41:31.888  WARN 18928 --- [ioEventLoop-6-3] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:31.892  WARN 18928 --- [ioEventLoop-6-5] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [127.0.0.1/<unresolved>:6384]: Connection refused: /127.0.0.1:6384
2021-04-05 21:41:31.987  WARN 18928 --- [ioEventLoop-6-6] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:32.089  WARN 18928 --- [ioEventLoop-6-9] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:32.191  WARN 18928 --- [oEventLoop-6-12] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:32.287  WARN 18928 --- [ioEventLoop-6-3] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:32.387  WARN 18928 --- [ioEventLoop-6-6] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:32.489  WARN 18928 --- [ioEventLoop-6-9] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:32.691  WARN 18928 --- [oEventLoop-6-12] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:32.989  WARN 18928 --- [ioEventLoop-6-3] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:33.591  WARN 18928 --- [ioEventLoop-6-6] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:34.688  WARN 18928 --- [ioEventLoop-6-9] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:36.788  WARN 18928 --- [oEventLoop-6-12] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:40.988  INFO 18928 --- [xecutorLoop-1-9] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 127.0.0.1/<unresolved>:6384
2021-04-05 21:41:40.990  WARN 18928 --- [ioEventLoop-6-3] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:40.992  WARN 18928 --- [ioEventLoop-6-5] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [127.0.0.1/<unresolved>:6384]: Connection refused: /127.0.0.1:6384
2021-04-05 21:41:49.287  INFO 18928 --- [ecutorLoop-1-10] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 127.0.0.1/<unresolved>:6384
2021-04-05 21:41:49.290  WARN 18928 --- [ioEventLoop-6-6] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:41:49.293  WARN 18928 --- [ioEventLoop-6-8] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [127.0.0.1/<unresolved>:6384]: Connection refused: /127.0.0.1:6384
2021-04-05 21:42:05.688  INFO 18928 --- [ecutorLoop-1-11] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 127.0.0.1/<unresolved>:6384
2021-04-05 21:42:05.690  WARN 18928 --- [ioEventLoop-6-9] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:42:05.692  WARN 18928 --- [oEventLoop-6-11] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [127.0.0.1/<unresolved>:6384]: Connection refused: /127.0.0.1:6384
// lettuce 연결 timeout 을 설정 하지 않은 상태라 default 값 1분 동안 연결이 안되어 에러 발생후 다시 연결 시도
2021-04-05 21:42:32.228 ERROR 18928 --- [nio-8080-exec-4] com.code.service.RedisService            : ### Redis Set Key Error !!! ::: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
2021-04-05 21:42:35.787  INFO 18928 --- [ecutorLoop-1-10] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 127.0.0.1/<unresolved>:6384
2021-04-05 21:42:35.789  WARN 18928 --- [oEventLoop-6-12] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:42:35.791  WARN 18928 --- [ioEventLoop-6-2] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [127.0.0.1/<unresolved>:6384]: Connection refused: /127.0.0.1:6384
2021-04-05 21:43:05.886  INFO 18928 --- [xecutorLoop-1-7] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was 127.0.0.1/<unresolved>:6384
2021-04-05 21:43:05.890  WARN 18928 --- [ioEventLoop-6-4] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [127.0.0.1/<unresolved>:6384]: Connection refused: /127.0.0.1:6384

- 클라이언트 (Junit 으로 API 계속 호출하는 곳)

2021-04-05 21:41:30.213  INFO 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:41:31.219  INFO 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
// 41분 31초에 마스터 죽고 나서 1분이 지난 시점인 42분 32초에 에러 발생 (그 뒤에도 1분 마다 에러 발생 = 1분간 연결을 계속적으로 잡고 있다는 뜻)
2021-04-05 21:42:32.227 ERROR 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 에러 발생 => java.net.SocketTimeoutException: Read timed out
2021-04-05 21:43:33.234 ERROR 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 에러 발생 => java.net.SocketTimeoutException: Read timed out
2021-04-05 21:44:34.238 ERROR 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 에러 발생 => java.net.SocketTimeoutException: Read timed out
2021-04-05 21:45:35.241 ERROR 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 에러 발생 => java.net.SocketTimeoutException: Read timed out

- 슬레이브 노드 2대 (redis)

// 계속 마스터(6383)에 연결 시도 하지만 어느 슬레이브 노드도 마스터로 승격되지 않는다.
21103:S 05 Apr 2021 21:41:40.509 * Connecting to MASTER 127.0.0.1:6384
21103:S 05 Apr 2021 21:41:40.509 * MASTER <-> REPLICA sync started
21103:S 05 Apr 2021 21:41:40.509 # Error condition on socket for SYNC: Connection refused
21103:S 05 Apr 2021 21:41:41.518 * Connecting to MASTER 127.0.0.1:6384
21103:S 05 Apr 2021 21:41:41.518 * MASTER <-> REPLICA sync started
21103:S 05 Apr 2021 21:41:41.518 # Error condition on socket for SYNC: Connection refused
21103:S 05 Apr 2021 21:41:42.527 * Connecting to MASTER 127.0.0.1:6384
21103:S 05 Apr 2021 21:41:42.528 * MASTER <-> REPLICA sync started
21103:S 05 Apr 2021 21:41:43.528 # Error condition on socket for SYNC: Connection refused
21103:S 05 Apr 2021 22:41:43.512 * Connecting to MASTER 127.0.0.1:6383
21103:S 05 Apr 2021 22:41:44.513 * MASTER <-> REPLICA sync started
21103:S 05 Apr 2021 22:41:44.513 # Error condition on socket for SYNC: Connection refused
21103:S 05 Apr 2021 22:41:45.524 * Connecting to MASTER 127.0.0.1:6383
21103:S 05 Apr 2021 22:41:45.525 * MASTER <-> REPLICA sync started
21103:S 05 Apr 2021 22:41:46.525 # Error condition on socket for SYNC: Connection refused
21103:S 05 Apr 2021 22:41:46.533 * Connecting to MASTER 127.0.0.1:6383
21103:S 05 Apr 2021 22:41:47.533 * MASTER <-> REPLICA sync started
21103:S 05 Apr 2021 22:41:47.533 # Error condition on socket for SYNC: Connection refused

- 센티널 (5002)

// 마스터 다운 감지
23288:X 05 Apr 2021 21:41:30.213 # +sdown master mymaster 127.0.0.1 6383

 

8. 센티널 2대 기동 : 서비스 안됨, 주관적다운(sdown) > 객관적다운(odown) 으로 판단하여 마땅한 마스터 노드를 찾지만 찾지 못하면  계속적으로 서비스 안됨 (여러번 테스트 결과 바로 fail-over 한 케이스도 있고, 20분이 걸린뒤 정상 서비스 되기도 함. 그러나 정리한 테스트에서는 기다리지 않고 마스터를 살리는 것으로 함)

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트) : 계속적으로 1분마다 연결 시도 로그

- 클라이언트 (Junit 으로 API 계속 호출하는 곳) : 1분 마다 에러 발생

- 센티널 (5000, 5001)

70671:X 05 Apr 2021 21:41:30.213 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
70671:X 05 Apr 2021 21:41:30.593 # Redis version=5.0.6, bits=64, commit=00000000, modified=0, pid=70671, just started
70671:X 05 Apr 2021 21:41:31.593 # Configuration loaded
70671:X 05 Apr 2021 21:41:31.595 * Increased maximum number of open files to 10032 (it was originally set to 256).
70671:X 05 Apr 2021 21:41:32.598 * Running mode=sentinel, port=5000.
70671:X 05 Apr 2021 21:41:32.598 # Sentinel ID is 22dc08c4977c01fa46c9e256df111de813e15d7c
70671:X 05 Apr 2021 21:41:33.598 # +monitor master mymaster 127.0.0.1 6383 quorum 2
70671:X 05 Apr 2021 21:41:33.656 # +sdown master mymaster 127.0.0.1 6383
70671:X 05 Apr 2021 21:41:34.656 # +sdown sentinel 08380ba6755b5d41583676cc1f86883b0ac4ad62 127.0.0.1 5001 @ mymaster 127.0.0.1 6383
// 주관적 다운(sdown) > 객관적 다운(odown) 이 됐지만 failover 에 실패함
70671:X 05 Apr 2021 21:41:34.741 # +odown master mymaster 127.0.0.1 6383 #quorum 2/2
70671:X 05 Apr 2021 21:41:35.742 # +new-epoch 11
70671:X 05 Apr 2021 21:41:35.743 # +try-failover master mymaster 127.0.0.1 6383
70671:X 05 Apr 2021 21:41:36.744 # +vote-for-leader 22dc08c4977c01fa46c9e256df111de813e15d7c 11
70671:X 05 Apr 2021 21:41:36.748 # 96b8dbd3461f1dae175a9c683dc8925aefa1664c voted for 22dc08c4977c01fa46c9e256df111de813e15d7c 11
70671:X 05 Apr 2021 21:41:37.813 # +elected-leader master mymaster 127.0.0.1 6383
70671:X 05 Apr 2021 21:41:37.813 # +failover-state-select-slave master mymaster 127.0.0.1 6383
70671:X 05 Apr 2021 21:41:38.872 # -failover-abort-no-good-slave master mymaster 127.0.0.1 6383
70671:X 05 Apr 2021 21:41:38.929 # Next failover delay: I will not start a failover before Fri Apr  9 23:30:55 2021

 

- 센티널 (5002)

// 센티널(5000) 살렸을때 fail-over 실패 로그
23288:X 05 Apr 2021 21:41:30.950 # +sdown master mymaster 127.0.0.1 6383
23288:X 05 Apr 2021 21:41:30.752 # -sdown sentinel 22dc08c4977c01fa46c9e256df111de813e15d7c 127.0.0.1 5000 @ mymaster 127.0.0.1 6383
23288:X 05 Apr 2021 21:41:31.747 # +new-epoch 11
23288:X 05 Apr 2021 21:41:31.748 # +vote-for-leader 22dc08c4977c01fa46c9e256df111de813e15d7c 11
23288:X 05 Apr 2021 21:41:32.911 # +odown master mymaster 127.0.0.1 6383 #quorum 2/2
// 재투표 가능한 시간인 해당 시간에 fail over 가 됨 (5002번은 이미 살아있는 상태라)
23288:X 05 Apr 2021 21:41:32.911 # Next failover delay: I will not start a failover before Fri Apr  5 21:49:56 2021

// 센티널(5001) 살렸을때 fail-over 실패 로그
23288:X 05 Apr 2021 21:42:44.791 # -sdown sentinel 08380ba6755b5d41583676cc1f86883b0ac4ad62 127.0.0.1 5001 @ mymaster 127.0.0.1 6383
23288:X 05 Apr 2021 21:42:44.505 # +new-epoch 12
23288:X 05 Apr 2021 21:42:45.506 # +vote-for-leader 08380ba6755b5d41583676cc1f86883b0ac4ad62 12
23288:X 05 Apr 2021 21:42:45.537 # Next failover delay: I will not start a failover before Fri Apr  5 21:48:35:20 2021

09. 마스터 기동 : 서비스 정상화 됨

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트)

// 마스터(6384)살리고(21:45:55) 약 20초 뒤에 다시 연결됨
2021-04-05 21:46:06.489  INFO 18928 --- [ioEventLoop-6-4] i.l.core.protocol.ReconnectionHandler    : Reconnected to 127.0.0.1/<unresolved>:6384

- 클라이언트 (Junit 으로 API 계속 호출하는 곳)

// 마스터 아직 안살렸을때 1분 마다 에러 발생
2021-04-05 21:43:33.234 ERROR 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 에러 발생 => java.net.SocketTimeoutException: Read timed out
2021-04-05 21:44:34.238 ERROR 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 에러 발생 => java.net.SocketTimeoutException: Read timed out
2021-04-05 21:45:35.241 ERROR 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 에러 발생 => java.net.SocketTimeoutException: Read timed out
// 마스터 연결 된 시점으로부터 정상화
2021-04-05 21:46:06.490  INFO 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:46:07.498  INFO 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}
2021-04-05 21:46:08.504  INFO 21340 --- [           main] com.code.RedisSentinelApplicationTests   : ### 테스트 결과 => status : 200 | response : {"key":"key","value":"hello_2"}

 

10. 슬레이브 다운 : 서비스 이상 없음, 다른 로그 변화 없음

- 아키텍쳐

11. 센티널 다운 : 서비스 이상 없음, 다른 로그 변화 없음

- 아키텍쳐

12. 마스터 다운 : 5초간 서비스 안되고 정상화 됨

- 아키텍쳐

- 서비스 서버 (APP:: 띄워둔 API 프로젝트)

// 센티널, 마스터 연결 시도
2021-04-05 21:53:56.784  INFO 18928 --- [ecutorLoop-1-10] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was /127.0.0.1:6384
2021-04-05 21:53:56.787  WARN 18928 --- [ioEventLoop-6-5] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:56.789  WARN 18928 --- [ioEventLoop-6-7] i.l.core.protocol.ConnectionWatchdog     : Cannot reconnect to [127.0.0.1/<unresolved>:6384]: Connection refused: /127.0.0.1:6384
2021-04-05 21:53:56.888  WARN 18928 --- [ioEventLoop-6-8] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:56.988  WARN 18928 --- [oEventLoop-6-11] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:57.086  WARN 18928 --- [ioEventLoop-6-2] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:57.182  WARN 18928 --- [ioEventLoop-6-5] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:57.287  WARN 18928 --- [ioEventLoop-6-8] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:57.386  WARN 18928 --- [oEventLoop-6-11] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:57.586  WARN 18928 --- [ioEventLoop-6-2] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:57.887  WARN 18928 --- [ioEventLoop-6-5] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:58.483  WARN 18928 --- [ioEventLoop-6-8] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:53:59.586  WARN 18928 --- [oEventLoop-6-11] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
2021-04-05 21:54:01.684  WARN 18928 --- [ioEventLoop-6-2] io.lettuce.core.RedisClient              : Cannot connect Redis Sentinel at redis://localhost:5000: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:5000
// 새 마스터(6383) 승격됨
2021-04-05 21:54:01.702  INFO 8928 --- [ioEventLoop-6-4] i.l.core.protocol.ReconnectionHandler    : Reconnected to 127.0.0.1/<unresolved>:6383

 

# 결론

- sentinel 쿼럼수 이상 살아있으면 sentinel 죽는건 상관 없으나 이상이 살아있지 않다면, 마스터가 죽었을시 승격되지 않음

- fail-over 시간동안 서비스가 안되기 때문에 장애 대비하여 redis 이용하는 부분에 try catch 를 하여 오류 발생시 다른 곳(DB)에서 데이터를 가져와서 리턴 할 수있도록 한다.