Configure High Availability – HDFS/YARN via Cloudera

In earlier releases, the NameNode was a single point of failure (SPOF) in a HDFS cluster. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable until the NameNode was either restarted or brought up on a separate machine. The Secondary NameNode did not provide failover capability. The HA architecture solved this problem of NameNode availability by allowing us to have two NameNodes in an active/passive configuration. The NameNode is the centerpiece of an HDFS file system

To enable Namenode HA in cloudera, you must ensure that the two nodes are of same configuration in terms of memory, disk, etc for optimal performance. Here are the steps.

ZooKeeper:

First of install “ZooKeeper to set up HA for NameNode.

Select cluster -> Action -> Add Service and pop will appear.

Add zookeeper from listed services.

Add ZooKeeper to 3 servers.

Add Zookeeper service to Cluster and click next.

Next step will initialize and start the ZooKeeper services. Click Next and ZooKeeper service successfully added to the cluster.

HDFS HA set up:

Now select the HDFS Service from cluster and see the status page.

Select Action -> Enable HA option.

Give Nameservice Name and click next.

Now select the location of Second NameNode service. In my case, I selected RM as a secondary NN location and 3 X JournalNodes (Odd Numbers are Required for OJM. See here for more information.

Now change the directory for Journal Nodes edit.

click next and you can see the progress steps.

NameNode formatting step will fail, that is fine. Wait till all the steps get finished.

Now its completed.

You can see the Active/Passive Node set up as below.

See the ZooKeeper status and one will be the leader and others as followers.

HA Test:

HA test is very simple and quick.  Check the status NameNodes services and select active Name Node -> action for selected -> stop.

Now check the status and after few seconds, the standby node will come up as active node.

Now start the stopped NN service and will come up as standby NN service after a while.

Setup HA for YARN:

Go to the YARN service.

Select Action “enable high availability”.

Select Second node for YARN service. I selected CM node.

Click next and it’ll enable the HA.

Now go to the ResourceManager you’ll see both Active/Passive RM nodes.

Once you have configured the YARN resource manager for HA, if the active resource manager is down or is no longer on the list, one of the standby resource managers becomes active and resumes resource manager responsibilities. This way, jobs continue to run and complete successfully.