by From my previous blog, we learnt how to set up a Hadoop Single Node Installation. Now, I will show how to set up a Hadoop Multi Node Cluster. A Multi Node Cluster in Hadoop contains two or more DataNodes in a distributed Hadoop environment. This is practically used in organizations to store and analyse their Petabytes and Exabytes of data.
Here in this blog, we are taking three machine to set up multi-node cluster – MN and DN1/DN2.
- Master node (MN) will run the NameNode and ResourcesManager Daemons.
- Data Nodes (DN1 and DN2) will be our data nodes that stores the actual data and provide processing power to run the jobs. Both hosts will run the DataNode and NodeManager daemons.
Software Required:
- REHL 7 – Set up MN and DN1/DN2 with REHL 7 operating system – Minimal Install.
- Hadoop-2.7.3
- JAVA 7
- SSH
Configure the System
First of all, we have to edit hosts file in /etc/ folder in MasterNode (MN) , specify the IP address of each system followed by their host names.
|
1 2 3 4 5 |
# vi /etc/hosts enter the following lines in the /etc/hosts file. 192.168.1.77 MN 192.168.1.78 DN1 192.168.1.79 DN2 |
Disable the firewall restrictions.
|
1 2 3 4 5 |
[root@MN yum.repos.d]# yum install iptables-services [root@MN yum.repos.d]# service iptables stop Redirecting to /bin/systemctl stop iptables.service [root@MN yum.repos.d]# chkconfig iptables off Note: Forwarding request to 'systemctl disable iptables.service'. |
Now set up OS group and user for Hadoop software.
|
1 2 3 |
[root@MN yum.repos.d]# groupadd hadoop [root@MN yum.repos.d]# useradd -g hadoop hduser [root@MN yum.repos.d]# passwd hduser |
Add the directories to keep the hdfs files.
|
1 2 3 4 5 |
[root@MN ~]# mkdir -p /data/dfs/nn [root@MN ~]# mkdir -p /data/dfs/dn [root@MN ~]# chown -R hduser:hadoop /data/dfs [root@MN ~]# mkdir -p /data/yarn/nm-local-dir [root@MN ~]# chown -R hduser:hadoop /data/yarn |
Download and Unpack Hadoop/Java Binaries.
Download and extract the Java Tar File on Master node. And Similarly download hadoop 2.7.3 Package on Master Node (MN) and extract the Hadoop tar File.
|
1 2 3 4 5 6 7 8 9 10 |
[root@MN ~]# wget "https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz" [root@MN ~]# wget "http://download.oracle.com/otn-pub/java/jdk/jdk-7u75-linux-x64.tar.gz" [root@MN ~]# mv hadoop-2.7.3.tar.gz /home/hduser [root@MN ~]# mv jdk-7u75-linux-x64.tar.gz /home/hduser [root@MN ~]# chown hduser:hadoop /home/hduser/* [root@MN ~]#su - hduser [hduser@MN ~]$ tar zxvf hadoop-2.7.3.tar.gz [hduser@MN ~]$ tar -xvf jdk-7u75-linux-x64.tar.gz [hduser@MN ~]$ ln -s hadoop-2.7.3 hadoop [hduser@MN ~]$ ln -s jdk1.7.0_75 jdk |
Set Environment Variables:
Add Hadoop/java binaries to your PATH. Edit /home/hadoop/.bash_profile and add the following lines in master node. Then, save the bash file and close it.
|
1 2 3 4 |
vi home/hadoop/.bash_profile export HADOOP_HOME=/home/hduser/hadoop export JAVA_HOME=/home/hduser/jdk PATH=$PATH:$HOME/.local/bin:$HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin |
For applying all these changes to the current Terminal, execute the source command.
|
1 |
[hduser@MN ~]$ source .bash_profile |
To make sure that Java and Hadoop have been properly installed on your system and can be accessed through the Terminal, execute the java -version and hadoop version commands.
|
1 2 3 4 5 6 7 8 9 10 |
hduser@MN ~]$ javac -version javac 1.7.0_75 [hduser@MN ~]$ java -version java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) [hduser@MN ~]$ hadoop version Hadoop 2.7.3 |
Now clone the Master Node MN to Data Node DN1/DN2.
Distribute Authentication Key-pairs for the Hadoop User:
Login to MN as the hadoop user, and generate an ssh-key. Copy the generated ssh Key to Master Node’s authorized keys.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
[hduser@MN .ssh]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/hduser/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hduser/.ssh/id_rsa. Your public key has been saved in /home/hduser/.ssh/id_rsa.pub. The key fingerprint is: SHA256:KN1GTdk/tbG74BYLvKpg9J64L3eYAwZJaT9Bjv8GTwU hduser@MN The key's randomart image is: +---[RSA 2048]----+ | o. E .o | | +o. .o. . ..| | o.o.. ... . .+| | o.+ +. oo | | o++.S . ..| | .+*. o o . | | .o.=o + + .| | ..==.. . + . | | o==+.. . | +----[SHA256]-----+ [hduser@MN ~]$ ssh-copy-id -i .ssh/id_rsa.pub 192.168.1.77 |
Copy the master node’s ssh key to DN1 and DN2 authorized keys.
|
1 2 3 4 5 6 |
[hduser@MN ~]$ for i in `cat hosts`; do scp /home/hduser/.ssh/authorized_keys $i:/home/hduser/.ssh; done authorized_keys 100% 391 715.9KB/s 00:00 hduser@192.168.1.78's password: authorized_keys 100% 391 545.2KB/s 00:00 hduser@192.168.1.79's password: authorized_keys 100% 391 482.8KB/s 00:00 |
Now test the passwordless connectivity login through SSH.
|
1 2 3 4 |
[hduser@MN ~]$ for i in `cat hosts`; do ssh $i "hostname -f"; done MN DN1 DN2 |
Configure Hadoop:
Now edit the configuration files in hadoop/etc/hadoop directory in master node. Set the NameNode location.
|
1 2 3 4 5 6 7 |
[hduser@MN hadoop]$ vi core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://MN:8020</value> </property> </configuration> |
Set path for HDFS:
Edit hdfs-site.conf on master node for NameNode and DataNode file location.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
hduser@MN hadoop]$ vi hdfs-site.xml <configuration> <property> <description>Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently. </description> <name>dfs.namenode.name.dir</name> <value>/data/dfs/nn</value> </property> <property> <description>list of paths on the local filesystem of a DataNode where it should store its blocks. </description> <name>dfs.datanode.data.dir</name> <value>/data/dfs/dn</value> </property> <property> <name>dfs.namenode.checkpoint.dir</name> <value>/data/dfs/snn</value> </property> <property> <description>Secondary Name Node. </description> <name>dfs.namenode.secondary.http-address</name> <value>MN:50090</value> </property> <property> <name>dfs.namenode.checkpoint.dir</name> <value>/data/dfs/snn</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> </configuration> |
The last property, dfs.replication, indicates how many times data is replicated in the cluster. You can set 2 to have all the data duplicated on the two nodes. Don’t enter a value higher than the actual number of data nodes.
Set YARN as Job Schedule:
Copy mapred-site from the template in configuration folder and the edit mapred-site.xml on Master node. Set yarn as the default framework for MapReduce operations.
|
1 2 3 4 5 6 7 8 9 |
[hduser@MN hadoop]$ cp mapred-site.xml.template mapred-site.xml [hduser@MN hadoop]$ vi mapred-site.xml </configuration> <property> <description>Execution framework set to Hadoop YARN. </description> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> |
Configure YARN:
Edit yarn-site.xml on master node.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 |
[hduser@MN hadoop]$ vi yarn-site.xml <configuration> <property> <description>Shuffle service that needs to be set for Map Reduce applications. </description> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <description>aggregation logs</description> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <description>List of directories to store localized files in.</description> <name>yarn.nodemanager.local-dirs</name> <value>file:///data/yarn/nm-local-dir</value> </property> <property> <description>Where to store container logs.</description> <name>yarn.nodemanager.log-dirs</name> <value>file:///var/log/hadoop-yarn/containers</value> </property> <property> <description>Where to aggregate logs to.</description> <name>yarn.nodemanager.remote-app-log-dir</name> <value>hdfs://MN:8020/var/log/hadoop-yarn/apps</value> </property> <property> <description>Classpath for typical applications.</description> <name>yarn.application.classpath</name> <value> $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*, $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/* </value> </property> <property> <description>ResourceManager address for clients to submit jobs. </description> <name>yarn.resourcemanager.address</name> <value>MN:8032</value> </property> <property> <description>ResourceManager Host:port for applicationMaster to talk to Scheduler to obtain resources. </description> <name>yarn.resourcemanager.scheduler.address</name> <value>MN:8030</value> </property> <property> <description>ResourceManager Host:port for NodeManagers </description> <name>yarn.resourcemanager.resource-tracker.address</name> <value>MN:8025</value> </property> </configuration> |
Configure Slaves
The file slaves is used by startup scripts to start required daemons on all nodes. Edit ~/hadoop/etc/hadoop/slaves to be.
|
1 2 3 |
[hduser@MN hadoop]$ vi slaves DN1 DN2 |
Format HDFS:
Format the namenode (Only on master machine).
|
1 2 3 4 5 6 7 8 9 10 11 |
[root@MN ~]# cd /home/hduser/hadoop/bin [root@MN ~]# hdfs namenode -format 18/05/17 21:53:54 INFO common.Storage: Storage directory /data/dfs/nn has been successfully formatted. 18/05/17 21:53:54 INFO namenode.FSImageFormatProtobuf: Saving image file /data/dfs/nn/current/fsimage.ckpt_0000000000000000000 using no compression 18/05/17 21:53:55 INFO namenode.FSImageFormatProtobuf: Image file /data/dfs/nn/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds. 18/05/17 21:53:55 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 18/05/17 21:53:55 INFO util.ExitUtil: Exiting with status 0 18/05/17 21:53:55 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at MN/192.168.1.77 ************************************************************/ |
Run and Monitor HDFS:
Now start Hadoop services by executing the following commands. It’ll start NameNode and SecondaryNameNode on node-master, and DataNode on node1 and node2, according to the configuration in the slaves config file.
|
1 2 3 4 5 6 7 8 |
[hduser@MN sbin]$ /home/hduser/hadoop/sbin [hduser@MN sbin]$ start-dfs.sh Starting namenodes on [MN] MN: starting namenode, logging to /home/hduser/hadoop-2.7.3/logs/hadoop-hduser-namenode-MN.out DN1: starting datanode, logging to /home/hduser/hadoop-2.7.3/logs/hadoop-hduser-datanode-DN1.out DN2: starting datanode, logging to /home/hduser/hadoop-2.7.3/logs/hadoop-hduser-datanode-DN2.out Starting secondary namenodes [MN] MN: starting secondarynamenode, logging to /home/hduser/hadoop-2.7.3/logs/hadoop-hduser-secondarynamenode-MN.out |
In addition to the previous HDFS daemon, you should see a ResourceManager on node-master, and a NodeManager on node1 and node2.
|
1 2 3 4 5 |
[hduser@MN sbin]$ start-yarn.sh starting yarn daemons starting resourcemanager, logging to /home/hduser/hadoop-2.7.3/logs/yarn-hduser-resourcemanager-MN.out DN2: starting nodemanager, logging to /home/hduser/hadoop-2.7.3/logs/yarn-hduser-nodemanager-DN2.out DN1: starting nodemanager, logging to /home/hduser/hadoop-2.7.3/logs/yarn-hduser-nodemanager-DN1.out |
Check all the daemons running on both master and slave machines.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
[hduser@MN ~]$ for i in `cat hosts`; do ssh -t $i "hostname -f; /home/hduser/jdk/bin/jps;echo -e '\n' "; done MN 17185 ResourceManager 17035 SecondaryNameNode 16851 NameNode 17593 Jps Connection to 192.168.1.77 closed. DN1 16353 DataNode 16655 Jps 16446 NodeManager Connection to 192.168.1.78 closed. DN2 17201 NodeManager 17410 Jps 17107 DataNode Connection to 192.168.1.79 closed. |
Web Interface:
At last, open the browser and go to master:50070/dfshealth.html on your master machine, this will give you the NameNode interface.
To view Hadoop clusters and all applications, browse the following address into your browser.
http://192.168.1.77:8088/cluster
You will see the information about NodeManager.
Secondary NameNode information avaiable via following link.
http://192.168.1.77:50090/status.html
Stop the Services:
Services can be stopped in following order.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
[hduser@MN sbin]$ /home/hduser/hadoop/sbin [hduser@MN sbin]$ stop-yarn.sh stopping yarn daemons stopping resourcemanager DN2: stopping nodemanager DN1: stopping nodemanager no proxyserver to stop [hduser@MN sbin]$ stop-dfs.sh Stopping namenodes on [MN] MN: stopping namenode DN2: stopping datanode DN1: stopping datanode Stopping secondary namenodes [MN] MN: stopping secondarynamenode |
I hope you would have successfully installed a Hadoop Multi Node Cluster. If you are facing any problem, you can comment below, we will be replying shortly.
Mandy!!!
Leave a Reply