by In this blog, I am going to talk about how to configure and manage a High availability HDFS (CDH 5.12.0) cluster. In earlier releases, the NameNode was a single point of failure (SPOF) in a HDFS cluster. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable until the NameNode was either restarted or brought up on a separate machine. The Secondary NameNode did not provide failover capability.
The HA architecture solved this problem of NameNode availability by allowing us to have two NameNodes in an active/passive configuration. So, we have two running NameNodes at the same time in a High Availability cluster:
- Active NameNode
- Standby/Passive NameNode.
We can implement the Active and Standby NameNode configuration in following two ways:
- Using Quorum Journal Nodes
- Shared Storage using NFS
Using the Quorum Journal Manager (QJM) is the preferred method for achieving high availability for HDFS. Read here to know more about QJM and NFS methods. In this blog, I’ll implement the HA configuration for quorum based storage and here are the IP address and corresponding machines Names/roles.
- NameNode machines – NN1/NN2 of equivalent hardware and spec
- JournalNode machines – The JournalNode daemon is relatively lightweight, so these daemons can reasonably be collocated on machines with other Hadoop daemons, for example NameNodes, the JobTracker, or the YARN ResourceManager. There must be at least three JournalNode daemons, since edit log modifications must be written to a majority of JournalNodes.So 3 JN’s runs on NN1/NN2 and MGT Server.
- Note that when running with N JournalNodes, the system can tolerate at most (N – 1) / 2 failures and continue to function normally.
- The ZookeerFailoverController (ZKFC) is a Zookeeper client that also monitors and manages the NameNode status. Each of the NameNode runs a ZKFC also. ZKFC is responsible for monitoring the health of the NameNodes periodically.
- Resource Manager Running on same NameNode NN1/NN2.
- Two Data Nodes – DN1 and DN2
Pre-requirements:
First of all, we have to edit hosts file in /etc/ folder in NameNode(NN1) , specify the IP address of each system followed by their host names. Each machine need a static IP address and all VM’s machines should be ping able from each other.
|
1 2 3 4 5 6 |
vi /etc/hosts 192.168.1.150 NN1 192.168.1.151 NN2 192.168.1.152 DN1 192.168.1.153 DN2 192.168.1.154 MGT |
All VM machines are set up with REHL 7 operating system. And disable the firewall restrictions on all VM’s.
|
1 2 3 4 5 |
[root@NN1 yum.repos.d]# yum install iptables-services [root@NN1 yum.repos.d]# service iptables stop Redirecting to /bin/systemctl stop iptables.service [root@NN1 yum.repos.d]# chkconfig iptables off Note: Forwarding request to 'systemctl disable iptables.service'. |
Setup Java:
Hadoop is written in Java so we need to set up Java first. Install the Oracle Java Development Kit (JDK) as below on all nodes.
|
1 2 3 4 5 6 7 8 9 |
[root@NN1 ~] mkdir /usr/java/ [root@NN1 ~] mv /root/software/jdk-7u75-linux-x64.tar.gz /usr/java/ [root@NN1 ~] tar -xvf jdk-7u75-linux-x64.tar.gz [root@NN1 ~] ln -s jdk1.7.0_75 default [root@NN1 ~] vim /etc/profile.d/java.sh export JAVA_HOME=/usr/java/default export PATH=$PATH:$JAVA_HOME:$JAVA_HOME/bin # Made file executable. [root@NN1 ~] chmod +x /etc/profile.d/java.sh |
Logout and login and you can see java version.
|
1 2 3 4 |
java -version java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) |
Download/install CDH 5.13.X
If you want to create your own YUM repository, download the appropriate repo file, create the repo, distribute the repo file as described under:
http://archive.cloudera.com/cdh5/repo-as-tarball/5.13.0/
You can use Hadoop 2.x distribution as well.
|
1 2 3 4 5 6 7 |
[root@NN1 ~] vi local-repos.repo [cloudera-cdh5] name=cloudera baseurl=file:///root/cdhrepo/cdh/5 gpgcheck=0 enabled=1 [root@NN1 ~] yum repolist |
Now install below components on NameNode1/2 (NN1/NN2)/and disable the run level start of components.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
[root@NN1 conf] yum install hadoop-hdfs-namenode -y [root@NN1 conf] yum install hadoop-yarn-resourcemanager -y [root@NN1 conf] yum install hadoop-mapreduce-historyserver -y [root@NN1 conf] yum install hadoop-hdfs-journalnode hadoop-hdfs-zkfc zookeeper-server zookeeper -y [root@NN2 conf] yum install hadoop-hdfs-journalnode hadoop-hdfs-zkfc zookeeper-server zookeeper -y # Set run level start off [root@NN1 ~]cd /etc/init.d/ [root@NN1 init.d]ls -altr [root@NN1 init.d] chkconfig hadoop-hdfs-namenode off [root@NN1 init.d] chkconfig zookeeper-server off [root@NN1 init.d] chkconfig hadoop-hdfs-zkfc off [root@NN1 init.d] chkconfig hadoop-hdfs-journalnode off [root@NN1 init.d] chkconfig hadoop-yarn-resourcemanager off |
Similarly install the DataNode and NodeManager rpms on DN1/DN2.
|
1 2 |
[root@DN1 conf] yum install hadoop-yarn-nodemanager -y [root@DN1 conf] yum install hadoop-hdfs-datanode -y |
Managment server (MGT) will be installed with journalNode, ZooKeeper and client Service.
|
1 2 |
[root@DN1 conf] yum install hadoop-client -y [root@MGT conf]yum install hadoop-hdfs-journalnode zookeeper-server zookeeper -y |
Distribute Authentication Key-pairs:
Login to NN1 as the root user, and generate an ssh-key. Copy the generated ssh Key to all Node’s authorized keys.
|
1 2 |
[root@NN1] .ssh]$ ssh-keygen [root@NN1] ssh-copy-id -i .ssh/id_rsa.pub 192.168.1.150 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[root@NN1 ~]# vi hosts 192.168.1.150 192.168.1.151 192.168.1.152 192.168.1.153 192.168.1.154 # Display HOSTNAME with password from NN1[root@NN1 ~]# for i in `cat hosts`; do ssh $i "hostname -f"; done NN1 NN2 DN1 DN2 MGT |
Configuration Details:
To configure HA NameNodes, you must add several configuration options to your hdfs-site.xml configuration file. Choose a logical name for this nameservice, for example “ha-cluster”. Open the core-site.xml file from the Active Name node and add the below properties.
cd /etc/hadoop/conf
|
1 2 3 4 5 6 7 |
[root@NN1 conf]# vi core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://ha-cluster</value> </property> </configuration> |
Open the HDFS-site.xml file, add this Datanode directory path in dfs.datanode.data.dir property and other properties.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
[root@NN1 conf]# vi hdfs-site.xml <configuration> <property> <name>dfs.namenode.name.dir</name> <value>/data/dfs/nn</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/data/dfs/dn</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.blocksize</name> <value>134217728</value> </property> <property> <name>dfs.nameservices</name> <value>ha-cluster</value> </property> <property> <name>dfs.ha.namenodes.ha-cluster</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.ha-cluster.nn1</name> <value>192.168.1.150:8020</value> </property> <property> <name>dfs.namenode.rpc-address.ha-cluster.nn2</name> <value>192.168.1.151:8020</value> </property> <property> <name>dfs.namenode.http-address.ha-cluster.nn1</name> <value>192.168.1.150:50070</value> </property> <property> <name>dfs.namenode.http-address.ha-cluster.nn2</name> <value>192.168.1.151:50070</value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://192.168.1.150:8485;192.168.1.151:8485;192.168.1.154:8485/ha-cluster</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/data/dfs/jn</value> </property> <property> <name>dfs.client.failover.proxy.provider.ha-cluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>shell(/bin/true)</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>192.168.1.150:2181,192.168.1.151:2181,192.168.1.154:2181</value> </property> </configuration> |
Edit yarn-site.xml on NN1.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 |
[root@NN1 conf]# vi yarn-site.xml <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <description>List of directories to store localized files in.</description> <name>yarn.nodemanager.local-dirs</name> <value>file:///data/yarn/nm-local-dir</value> </property> <property> <description>Where to store container logs.</description> <name>yarn.nodemanager.log-dirs</name> <value>file:///var/log/hadoop-yarn/containers</value> </property> <property> <description>Where to aggregate logs to.</description> <name>yarn.nodemanager.remote-app-log-dir</name> <value>hdfs://ha-cluster:8020/var/log/hadoop-yarn/apps</value> </property> <property> <description>Classpath for typical applications.</description> <name>yarn.application.classpath</name> <value> $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*, $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/* </value> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>ha-cluster</value> </property> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <property> <name>yarn.client.failover.proxy-provider</name> <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value> </property> <property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value> </property> <property> <name>yarn.resourcemanager.recovery.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>192.168.1.150</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>192.168.1.151</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm1</name> <value>192.168.1.150:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.address.rm2</name> <value>192.168.1.151:8088</value> </property> <property> <name>yarn.resourcemanager.zk-address</name> <value>192.168.1.150:2181,192.168.1.151:2181,192.168.1.154:2181</value> </property> </configuration> |
edit and set yarn as the default framework for MapReduce operations and other HA properties.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
[root@NN1 conf]# vi mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>192.168.1.151:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>192.168.1.151:19888</value> </property> <property> <name>yarn.ap.mapreduce.am.staging-dir</name> <value>/user</value> </property> </configuration> |
update slaves with DataNodes VM’s DN1/DN2.
|
1 2 3 |
[root@NN1 conf]# vi slaves 192.168.1.152 192.168.1.153 |
Once all files updated on NN1 and now copy configs files from NN1 to all other nodes.
|
1 |
[root@NN1 ~]# for i in `cat hosts`; do scp /etc/hadoop/conf/* $i:/etc/hadoop/conf/; done |
Deploying Zookeeper:
In a conf directory you have zoo_sample.cfg file, create the zoo.cfg using zoo_sample.cfg file. Open the zoo.cfg file. Add the custom directory path to the dataDir property if you want and add the below details regarding remaining node, in the zoo.cfg file. I kept the same directory. See screen shot below and copy the Zookeeper conf file to NN2 and MGT server.
|
1 2 3 4 5 6 7 |
[root@NN1 ~]# vi /etc/zookeeper/conf/zoo.cfg # add at the end server.1=192.168.1.150:2888:3888 server.2=192.168.1.151:2888:3888 server.3=192.168.1.154:2888:3888 [root@NN1 conf]# scp zoo.cfg 192.168.1.151:/etc/zookeeper/conf/ [root@NN1 conf]# scp zoo.cfg 192.168.1.154:/etc/zookeeper/conf/ |
Now initialize the zookeeper and start the Zookeeper service.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[root@NN1 conf]# service zookeeper-server init --myid=1 [root@NN2 ~]# service zookeeper-server init --myid=2 [root@MGT ~]# service zookeeper-server init --myid=3 # If directory already exist the use force option # service zookeeper-server init --myid=1 --force # Start the ZK Service. [root@NN1 conf]# service zookeeper-server start JMX enabled by default Using config: /etc/zookeeper/conf/zoo.cfg Starting zookeeper ... STARTED [root@NN2 ~]# service zookeeper-server start [root@MGT ~]# service zookeeper-server start |
Now start journal node now.
|
1 2 3 |
[root@NN1 bin]# service hadoop-hdfs-journalnode start [root@NN2 bin]# service hadoop-hdfs-journalnode start [root@MGT bin]# service hadoop-hdfs-journalnode start |
Format HDFS:
Format the NameNode on NN1 only if new cluster. If converting the existing cluster from NON-HA to HA then re-initilize it as i did.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
[root@NN1 ~]# sudo -u hdfs hdfs namenode -initializeSharedEdits --- 18/05/21 19:16:03 INFO namenode.EditLogInputStream: Fast-forwarding stream '/data/dfs/nn/current/edits_0000000000000000209-0000000000000000209' to transaction ID 169 18/05/21 19:16:03 INFO namenode.FSEditLog: Starting log segment at 209 18/05/21 19:16:03 INFO namenode.FSEditLog: Ending log segment 209, 209 18/05/21 19:16:03 INFO namenode.FSEditLog: logSyncAll toSyncToTxId=209 lastSyncedTxid=209 mostRecentTxid=209 18/05/21 19:16:03 INFO namenode.FSEditLog: Done logSyncAll lastWrittenTxId=209 lastSyncedTxid=209 mostRecentTxid=209 18/05/21 19:16:03 INFO namenode.FSEditLog: Number of transactions: 1 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 7 18/05/21 19:16:03 INFO util.ExitUtil: Exiting with status 0 18/05/21 19:16:03 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at NN1/192.168.1.150 ************************************************************/ ### Format In new cluster sudo -u hdfs hdfs namenode -format |
Start And Monitor HDFS Services:
Now start the NameNode Service on NN1.
|
1 |
[root@NN1 ~]# service hadoop-hdfs-namenode start |
Now NN2 has different meta data.. So how you’ll sync it with NN1, you can either copy the entire meta data directory or use this command. Start the NameNode servie on NN2.
|
1 2 3 4 |
[root@NN2 ~]# sudo -u hdfs hdfs namenode -bootstrapStandby [root@NN2 ~]# service hadoop-hdfs-namenode start starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-NN2.out Started Hadoop namenode: [ OK |
Start the DataNode Services on DN1/DN2.
|
1 2 3 4 5 6 |
[root@DN1 ~]# service hadoop-hdfs-datanode start starting datanode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-datanode-DN1.out Started Hadoop datanode (hadoop-hdfs-datanode): [ OK ] [root@DN2 ~]# service hadoop-hdfs-datanode start starting datanode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-datanode-DN2.out Started Hadoop datanode (hadoop-hdfs-datanode): [ OK ] |
The next step is to initialize required state in ZooKeeper. You can do so by running the following command from one of the NameNode hosts. This will create a znode in ZooKeeper inside of which the automatic failover system stores its data.
|
1 2 3 |
[root@NN1 hadoop-hdfs]# sudo -u hdfs hdfs zkfc -formatZK --- Check this line from logs 18/05/21 19:45:06 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ha-cluster in ZK. |
Check Zookeeper connected or not.
|
1 2 3 4 5 |
[root@NN1 conf]# zookeeper-client -server=192.168.1.150:2181 WatchedEvent state:SyncConnected type:None path:null JLine support is enabled [zk: localhost:2181(CONNECTED) 0] [zk: localhost:2181(CONNECTED) 1] ls /hadoop-ha/ha-cluster |
Now start ZKFC service. Note: wherever you start the ZKFC first, that’ll become the active node.
|
1 2 3 4 5 6 |
[root@NN1 hadoop-hdfs]# service hadoop-hdfs-zkfc start starting zkfc, logging to /var/log/hadoop-hdfs/hadoop-hdfs-zkfc-NN1.out Started Hadoop zkfc: [ OK ] [root@NN2 ~]# service hadoop-hdfs-zkfc start starting zkfc, logging to /var/log/hadoop-hdfs/hadoop-hdfs-zkfc-NN2.out Started Hadoop zkfc: [ OK ] |
Web Interface:
Open the browser and explore further.
http://192.168.1.150:50070/dfshealth.html#tab-overview
http://192.168.1.151:50070/dfshealth.html#tab-overview
Check Services with command line.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
[root@NN1 ~]# for i in `cat hosts`; do ssh -t $i "hostname -f;sudo -u hdfs /usr/java/default/bin/jps;sudo -u yarn /usr/java/default/bin/jps;sudo -u mapred /usr/java/default/bin/jps;sudo -u zookeeper /usr/java/default/bin/jps;echo -e '\n' "; done NN1 3053 DFSZKFailoverController 2039 NameNode 1889 JournalNode 2755 QuorumPeerMain NN2 2869 DFSZKFailoverController 1914 JournalNode 2147 NameNode 2690 QuorumPeerMain DN1 1747 Jps 1605 DataNode DN2 1637 DataNode MGT 1822 JournalNode 2196 QuorumPeerMain |
Test HA set up:
See process Id and kill the active NameNode on NN1.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
[root@NN1 ~]# sudo -u hdfs hdfs haadmin -getServiceState nn1 active [root@NN1 ~]# sudo -u hdfs hdfs haadmin -getServiceState nn2 standby [root@NN1 ~]# kill -9 2039 [root@NN1 ~]# sudo -u hdfs hdfs haadmin -getServiceState nn2 active [root@NN1 ~]# service hadoop-hdfs-namenode start starting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-NN1.out Started Hadoop namenode: [ OK ] [root@NN1 ~]# sudo -u hdfs hdfs haadmin -getServiceState nn1 standby |
Repeat the same test on NN2 and also check the web URL’s.
YARN HA set up:
Now start the ResourceManager and NodeManager on NN1/NN2.
|
1 2 3 4 5 6 |
[root@NN1 ~]# service hadoop-yarn-resourcemanager start [root@NN2 ~]# service hadoop-yarn-resourcemanager start # Start the Node Manager [root@DN1 ~]# service hadoop-yarn-nodemanager start [root@DN2 ~]# service hadoop-yarn-nodemanager start |
Complete the Failover test here.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
[root@NN1 ~]# sudo -u yarn yarn rmadmin -getServiceState rm1 active [root@NN1 ~]# sudo -u yarn yarn rmadmin -getServiceState rm2 standby [root@NN1 ~]# kill -9 3787 [root@NN1 ~]# sudo -u yarn yarn rmadmin -getServiceState rm2 active [root@NN1 ~]# service hadoop-yarn-resourcemanager start starting resourcemanager, logging to /var/log/hadoop-yarn/yarn-yarn-resourcemanager-NN1.out Started Hadoop resourcemanager: [ OK ] [root@NN1 ~]# sudo -u yarn yarn rmadmin -getServiceState rm1 standb |
http://192.168.1.150:8088/cluster/cluster
http://192.168.1.151:8088/cluster/cluster
Once you have configured the YARN resource manager for HA, if the active resource manager is down or is no longer on the list, one of the standby resource managers becomes active and resumes resource manager responsibilities. This way, jobs continue to run and complete successfully.
Please share and like if you this blog is help for you.
Thanks
Mandy
1 Comment