Fault tolerance

YugabyteDB can automatically handle failures and therefore provides high availability. You will create YSQL tables with a replication factor of 3 that allows a fault tolerance of 1. This means the cluster will remain available for both reads and writes even if one node fails. However, if another node fails bringing the number of failures to two, then writes will become unavailable on the cluster in order to preserve data consistency.

Prerequisite

Install a local YugabyteDB universe on Docker using the steps below.

mkdir ~/yugabyte && cd ~/yugabyte
wget https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/bin/yb-docker-ctl && chmod +x yb-docker-ctl
docker pull yugabytedb/yugabyte

1. Create universe

If you have a previously running local universe, destroy it using the following command.

$ ./yb-docker-ctl destroy

Start a new local universe with replication factor of 5.

$ ./yb-docker-ctl create --rf 5

Connect to ycqlsh on node 1.

$ YB_TSERVER_N1_ADDR=$(docker container inspect -f '{{ $network := index .NetworkSettings.Networks "yb-net" }}{{ $network.IPAddress }}' yb-tserver-n1)
$ docker exec -it yb-tserver-n1 /home/yugabyte/bin/ycqlsh $YB_TSERVER_N1_ADDR

Connected to local cluster at 127.0.0.1:9042.
[ycqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
ycqlsh>

Create a keyspace and a table.

ycqlsh> CREATE KEYSPACE users;

ycqlsh> CREATE TABLE users.profile (id bigint PRIMARY KEY,
                                    email text,
                                    password text,
                                    profile frozen<map<text, text>>);

2. Insert data through a node

Now insert some data by typing the following into ycqlsh shell.

ycqlsh> INSERT INTO users.profile (id, email, password, profile) VALUES
  (1000, 'james.bond@yugabyte.com', 'licensed2Kill',
   {'firstname': 'James', 'lastname': 'Bond', 'nickname': '007'}
  );

ycqlsh> INSERT INTO users.profile (id, email, password, profile) VALUES
  (2000, 'sherlock.holmes@yugabyte.com', 'itsElementary',
   {'firstname': 'Sherlock', 'lastname': 'Holmes'}
  );

Query all the rows.

ycqlsh> SELECT email, profile FROM users.profile;

 email                        | profile
------------------------------+---------------------------------------------------------------
      james.bond@yugabyte.com | {'firstname': 'James', 'lastname': 'Bond', 'nickname': '007'}
 sherlock.holmes@yugabyte.com |               {'firstname': 'Sherlock', 'lastname': 'Holmes'}

(2 rows)

3. Read data through another node

Let us now query the data from node 5.

$ YB_TSERVER_N5_ADDR=$(docker container inspect -f '{{ $network := index .NetworkSettings.Networks "yb-net" }}{{ $network.IPAddress }}' yb-tserver-n5)
$ docker exec -it yb-tserver-n5 /home/yugabyte/bin/ycqlsh $YB_TSERVER_N5_ADDR

ycqlsh> SELECT email, profile FROM users.profile;

 email                        | profile
------------------------------+---------------------------------------------------------------
      james.bond@yugabyte.com | {'firstname': 'James', 'lastname': 'Bond', 'nickname': '007'}
 sherlock.holmes@yugabyte.com |               {'firstname': 'Sherlock', 'lastname': 'Holmes'}

(2 rows)

4. Verify that one node failure has no impact

We have five nodes in this universe. You can verify this by running the following.

$ ./yb-docker-ctl status

Let us simulate node 5 failure by doing the following.

$ ./yb-docker-ctl remove_node 5

Now running the status command should show only four nodes:

$ ./yb-docker-ctl status

Now connect to node 4.

$ YB_TSERVER_N4_ADDR=$(docker container inspect -f '{{ $network := index .NetworkSettings.Networks "yb-net" }}{{ $network.IPAddress }}' yb-tserver-n4)
$ docker exec -it yb-tserver-n4 /home/yugabyte/bin/ycqlsh $YB_TSERVER_N4_ADDR

Let us insert some data.

ycqlsh> INSERT INTO users.profile (id, email, password, profile) VALUES
  (3000, 'austin.powers@yugabyte.com', 'imGroovy',
   {'firstname': 'Austin', 'lastname': 'Powers'});

Now query the data.

ycqlsh> SELECT email, profile FROM users.profile;

 email                        | profile
------------------------------+---------------------------------------------------------------
      james.bond@yugabyte.com | {'firstname': 'James', 'lastname': 'Bond', 'nickname': '007'}
 sherlock.holmes@yugabyte.com |               {'firstname': 'Sherlock', 'lastname': 'Holmes'}
   austin.powers@yugabyte.com |                 {'firstname': 'Austin', 'lastname': 'Powers'}

(3 rows)

5. Verify that second node failure has no impact

This cluster was created with a replication factor of 5 and hence needs only three replicas to make consensus. Therefore, it is resilient to 2 failures without any data loss. Let us simulate another node failure.

$ ./yb-docker-ctl remove_node 1

We can check the status to verify:

$ ./yb-docker-ctl status

Now let us connect to node 2.

$ YB_TSERVER_N2_ADDR=$(docker container inspect -f '{{ $network := index .NetworkSettings.Networks "yb-net" }}{{ $network.IPAddress }}' yb-tserver-n2)
$ docker exec -it yb-tserver-n2 /home/yugabyte/bin/ycqlsh $YB_TSERVER_N2_ADDR

Insert some data.

ycqlsh> INSERT INTO users.profile (id, email, password, profile) VALUES
  (4000, 'superman@yugabyte.com', 'iCanFly',
   {'firstname': 'Clark', 'lastname': 'Kent'});

Run the query.

ycqlsh> SELECT email, profile FROM users.profile;

 email                        | profile
------------------------------+---------------------------------------------------------------
        superman@yugabyte.com |                    {'firstname': 'Clark', 'lastname': 'Kent'}
      james.bond@yugabyte.com | {'firstname': 'James', 'lastname': 'Bond', 'nickname': '007'}
 sherlock.holmes@yugabyte.com |               {'firstname': 'Sherlock', 'lastname': 'Holmes'}
   austin.powers@yugabyte.com |                 {'firstname': 'Austin', 'lastname': 'Powers'}

(4 rows)

Step 6. Clean up (optional)

Optionally, you can shut down the local cluster you created earlier.

$ ./yb-docker-ctl destroy