by Hadoop is a Java-based programming framework that supports the processing and storage of extremely large datasets on a cluster of inexpensive machines. It was the first major open source project in the big data playing field and provides high throughput access to application data .
The main goal of this tutorial is to get a simple Hadoop installation up and running so that you can play around with the software and learn more about it.
Environment: This blog has been tested in the following software version.
- REHL ( Red hat Linux 7.4) on Virtual box 5.2
- Hadoop 2.7.3 version
- update /etc/hosts file with Hostname and IP address.
[root@cdhs ~]# cat /etc/hosts
10.0.2.5 cdhs
Dedicated Hadoop system user:
After VM set up, please add a non sudo user dedicated to Hadoop which will be used to configure Hadoop. Following command will add the user hduser and the group hadoop to VM machine.
|
1 2 3 4 5 6 7 8 |
[root@cdhs ~]# groupadd hadoop [root@cdhs ~]# useradd -g hadoop hduser # Password set for hduser [root@cdhs ~]# passwd hduser Changing password for user hduser. New password: Retype new password: passwd: all authentication tokens updated successfully. |
Apache Software:
download the apache hadoop software from official site.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
[root@cdhs yum.repos.d]# wget <a href="https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz">https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz</a>--2018-05-03 14:57:40-- <a href="https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz">https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz</a>Resolving archive.apache.org (archive.apache.org)... 163.172.17.199 Connecting to archive.apache.org (archive.apache.org)|163.172.17.199|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 214092195 (204M) [application/x-gzip] Saving to: ‘hadoop-2.7.3.tar.gz’ 100%[============================================================================================>]214,092,195 6.33MB/s in 46s [root@cdhs ~]# mv hadoop-2.7.3.tar.gz /home/hduser [root@cdhs hduser]# chown hduser:hadoop /home/hduser/* [root@cdhs ~]# mv hadoop-2.7.3.tar.gz /home/hduser [root@cdhs hduser]# su - hduser hduser@cdhs ~]$ tar zxvf hadoop-2.7.3.tar.gz [hduser@cdhs ~]$ ln -s hadoop-2.7.3 hadoop |
Install Java:
Hadoop is written in Java, hence before installing Apache Hadoop we will need to install Java first. To install Java in your system first we will need to download the file from oracle website from here.
|
1 2 3 4 5 6 |
wget "http://download.oracle.com/otn-pub/java/jdk/jdk-7u75-linux-x64.tar" [root@cdhs ~]# mv jdk-7u75-linux-x64.tar /home/hduser [root@cdhs hduser]# su - hduser Last login: Thu May 3 14:44:38 NZST 2018 on pts/0 [hduser@cdhs ~]$ tar -xvf jdk-7u75-linux-x64.tar.gz [hduser@cdhs ~]$ ln -s jdk1.7.0_75 jdk |
Once the Java is installed in your system, you can check the version of Java using the following command.
|
1 2 3 4 |
[hduser@cdhs ~]$ java -version java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) |
Now edit .bash_profile file using your favorite editor and add Hadoop/java home.
|
1 2 3 4 5 6 7 |
[hduser@cdhs ~]$ cat .bash_profile export HADOOP_HOME=/home/hduser/hadoop export JAVA_HOME=/home/hduser/jdk # PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin:$HOME/bin:$PATH PATH=$PATH:$HOME/.local/bin:$HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin # Initialize the variables using the following command. Source .bash_profile |
After unpack the downloaded Hadoop distribution, edit the file $HADOOP_HOME/etc/hadoop/hadoop-env.sh to define some parameters as follows. It is important that we setup Java path here, otherwise Hadoop will not be able to use Java.
|
1 2 |
# set to the root of your Java installation export JAVA_HOME=/home/hduser/jdk |
Once done, you can now check if the environment variables are now set or not. Run the following command.
|
1 2 |
echo $JAVA_HOME echo $HADOOP_HOME |
Configuring Hadoop:
Hadoop has many configuration files, which are located at $HADOOP_HOME/etc/hadoop directory. You can view the list of configuration files using the following command. As we are installing Hadoop on a single node in pseudo distributed mode. We will need to edit some configuration files in order for Hadoop to work.
The first file is “core-site.xml” file which contains configuration of the port number used by HDFS.
|
1 2 3 4 5 6 7 8 |
[hduser@cdhs hadoop]$ vi core-site.xml <configuration> <property> <description>NameNode Setup </description> <name>fs.defaultFS</name> <value>hdfs://10.0.2.5:8020</value> </property> </configuration> |
Configuration for NameNode and DataNode.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
[hduser@cdhs hadoop]$ vi hdfs-site.xml <configuration> <property> <description>Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently.</description> <name>dfs.namenode.name.dir</name> <value>/data/dfs/nn</value> </property> <property> <description>list of paths on the local filesystem of a DataNode where it should store its blocks. </description> <name>dfs.datanode.data.dir</name> <value>/data/dfs/dn</value> </property> <property> <name>dfs.namenode.checkpoint.dir</name> <value>/data/dfs/snn</value> </property> <property> <description> Secondary NameNode </description> <name>dfs.namenode.secondary.http-address</name> <value>10.0.2.5:50090</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.blocksize</name> <value>134217728</value> </property> </configuration> |
Configuration for ResourceManager and NodeManager.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
[hduser@cdhs hadoop]$ vi yarn-site.xml <configuration> <property> <description>Shuffle service that needs to be set for Map Reduce applications. </description> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <description>aggregation logs</description> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <description>List of directories to store localized files in.</description> <name>yarn.nodemanager.local-dirs</name> <value>file:///data/yarn/nm-local-dir</value> </property> <property> <description>Where to store container logs.</description> <name>yarn.nodemanager.log-dirs</name> <value>file:///var/log/hadoop-yarn/containers</value> </property> <property> <description>Where to aggregate logs to.</description> <name>yarn.nodemanager.remote-app-log-dir</name> <value>hdfs://10.0.2.5:8020/var/log/hadoop-yarn/apps</value> </property> <property> <description>Classpath for typical applications.</description> <name>yarn.application.classpath</name> <value> $HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*, $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*, $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*, $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/* </value> </property> <property> <description>ResourceManager address for clients to submit jobs. </description> <name>yarn.resourcemanager.address</name> <value>10.0.2.5:8032</value> </property> <property> <description>ResourceManager Host:port for applicationMaster to talk to Scheduler to obtain resources.</description> <name>yarn.resourcemanager.scheduler.address</name> <value>10.0.2.5:8030</value> </property> <property> <description>ResourceManager Host:port for NodeManagers </description> <name>yarn.resourcemanager.resource-tracker.address</name> <value>10.0.2.5:8025</value> </property> </configuration> |
Now copy mapred-site.xml.template as file mapred-site.xml using the following command. And edit the “mapred-site.xml” file for Mapreduce Applications and JobHistory Server.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
[hduser@cdhs hadoop]$ cp mapred-site.xml.template mapred-site.xml [hduser@cdhs hadoop]$ vi mapred-site.xml <configuration> <property> <description>Execution framework set to Hadoop YARN. </description> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <description>Mapreduce job history server. </description> <name>mapreduce.jobhistory.address</name> <value>10.0.2.5:10020</value> </property> <property> <description>Mapreduce job history server WEB UI. </description> <name>mapreduce.jobhistory.webapp.address</name> <value>10.0.2.5:19888</value> </property> <property> <name>yarn.ap.mapreduce.am.staging-dir</name> <value>/user</value> </property> </configuration> |
|
1 2 |
[hduser@cdhs hadoop]$ vi slaves 10.0.2.5 |
Now we will need to create two directories to store namenode and datanode using the following commands.
|
1 2 3 4 5 6 |
[root@cdhs sf_Share_VM]# mkdir -p /data/dfs/nn [root@cdhs sf_Share_VM]# mkdir -p /data/dfs/dn [root@cdhs sf_Share_VM]# mkdir -p /data/dfs/snn [root@cdhs sf_Share_VM]# chown -R hduser:hadoop /data/dfs [root@cdhs sf_Share_VM]# mkdir -p /data/yarn/nm-local-dir [root@cdhs sf_Share_VM]# chown -R hduser:hadoop /data/yarn |
Now you will need to configure the SSH keys for your new user so that the user can securely log into hadoop without any password.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
[hduser@cdhs ~]$ ssh-keygen [hduser@cdhs ~]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/hduser/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hduser/.ssh/id_rsa. Your public key has been saved in /home/hduser/.ssh/id_rsa.pub. The key fingerprint is: SHA256:wNDLCv5DMd+kUDcJQ//+m9sb5ZgA9iyrIlMxejmNmUo hduser@cdhs The key's randomart image is: +---[RSA 2048]----+ | .o+. . | | ooo+ | | ooo..o | | . + =..o + | | . . B %S o + .| | . E @ o. o . = | | + + . o + .| | * . . . o . | | + .. =oo. | +----[SHA256]-----+ [hduser@cdhs ~]$ ssh-copy-id -i .ssh/id_rsa.pub 10.0.2.5 /bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_rsa.pub" /bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys hduser@10.0.2.5's password: Number of key(s) added: 1 |
We have now configured Hadoop to work on a single node cluster. Now we can initialize HDFS file system by formatting the namenode directory using the following command.
Now we can start Hadoop cluster, navigate to $HADOOP_HOME/sbin directory using the following command.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
[hduser@cdhs ~]$ cd hadoop/bin/ [hduser@cdhs bin]$ hdfs namenode -format --- End lines only 18/05/03 16:08:35 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 18/05/03 16:08:35 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 18/05/03 16:08:35 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 18/05/03 16:08:35 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 18/05/03 16:08:35 INFO util.GSet: Computing capacity for map NameNodeRetryCache 18/05/03 16:08:35 INFO util.GSet: VM type = 64-bit 18/05/03 16:08:35 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB 18/05/03 16:08:35 INFO util.GSet: capacity = 2^15 = 32768 entries 18/05/03 16:08:35 INFO namenode.FSImage: Allocated new BlockPoolId: BP-644753539-10.0.2.5-1525320515324 18/05/03 16:08:35 INFO common.Storage: Storage directory /data/dfs/nn has been successfully formatted. 18/05/03 16:08:35 INFO namenode.FSImageFormatProtobuf: Saving image file /data/dfs/nn/current/fsimage.ckpt_0000000000000000000 using no compression 18/05/03 16:08:35 INFO namenode.FSImageFormatProtobuf: Image file /data/dfs/nn/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds. 18/05/03 16:08:35 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 18/05/03 16:08:35 INFO util.ExitUtil: Exiting with status 0 18/05/03 16:08:35 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at cdhs/10.0.2.5 ************************************************************/ |
You can start Hadoop services by executing the following commands.
|
1 2 3 4 5 6 7 |
cd /home/hduser/hadoop/sbin [hduser@cdhs sbin]$ start-dfs.sh Starting namenodes on [cdhs] cdhs: starting namenode, logging to /home/hduser/hadoop-2.7.3/logs/hadoop-hduser-namenode-cdhs.out 10.0.2.5: starting datanode, logging to /home/hduser/hadoop-2.7.3/logs/hadoop-hduser-datanode-cdhs.out Starting secondary namenodes [cdhs] cdhs: starting secondarynamenode, logging to /home/hduser/hadoop-2.7.3/logs/hadoop-hduser-secondarynamenode-cdhs.out |
Now start YARN using the following command.
|
1 2 3 4 |
[hduser@cdhs sbin]$ start-yarn.sh starting yarn daemons starting resourcemanager, logging to /home/hduser/hadoop-2.7.3/logs/yarn-hduser-resourcemanager-cdhs.out 10.0.2.5: starting nodemanager, logging to /home/hduser/hadoop-2.7.3/logs/yarn-hduser-nodemanager-cdhs.out |
You can check the status of the services using the following command.
|
1 2 3 4 5 6 7 |
[hduser@cdhs ~]$ jdk/bin/jps 3542 ResourceManager 3098 NameNode 3648 NodeManager 3227 DataNode 3940 Jps 3392 SecondaryNameNode |
This shows that Hadoop is successfully running on the server.
You can now browse the Apache Hadoop services through your browser. By default Apache Hadoop namenode service is started on port 50070. Go to following address using your favorite browser.
To view Hadoop clusters and all applications, browse the following address into your browser.
You will see the information about NodeManager.
Secondary NameNode information avaiable via following link.
If problem in opening any of the URL above, please disable the IPTABLES service.
|
1 2 3 4 |
[root@cdhs ~]# service iptables stop Redirecting to /bin/systemctl stop iptables.service [root@cdhs ~]# service iptables disable The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl. |
Conclusion:
In this tutorial we have learnt how to install Apache Hadoop on a single node with Pseudo distribution mode.
1 Comment