Commissioning of nodes means adding new data node in cluster and decommissioning stands for removing node from cluster. You can’t directly add/remove dataNode in large and a real-time cluster as it can cause a lot of disturbance. So if you want to scale your cluster , you need commissioning  and steps are below.

Commission:

Pre-requirements:

  • Clone existing Node.
  • Change IP address and hostname –  192.168.1.155 and DN3
  • Update Hosts files on all nodes – add this entry in /etc/hosts file “192.168.1.155 DN3”
  • Make it password less

Configuration changes:

We need to update the include file on both the Resource Manager and the Namenode . If it’s not present, then create an include file on both the Nodes.

Go to your NameNode and add include file in hdfs-site.xml file.

Also update the slaves file on NameNode and add new DataNode IP address.

Edit the “yarn-site.xml” file where ResourceManager is running.

Now update the include RM.

New DataNode Set up:

Copy all configuration files from NameNode and then refresh the Nodes.

Now start the services on DataNode.

Commissioning of new DataNode is complete. Check Hadoop admin report using the command

Decommission a Node:

For decommission DataNode, exclude property need at NameNode side. Do decomm activity in non-peak hour. Any process running on this decommissioned node can fail.

Note: It’s very important to note that the include/exclude files should be mutually exclusive. Means can’t have same values in both exclude and include file.

Update

Refresh the nodes.

Hadoop Balancer:

Hadoop Balancer is a built-in property which makes sure that no datanode will be over utilized. When you run the balancer utility, it checks whether some datanode are under-utilized or over-utilized and will balance the replication factor. But make sure the Balancer should run in only off-peak hours in a real cluster, because if you run this during peak hours, it will cause a heavy load to networking, as it will transfer a large amount of data.

Hope this post was helpful in understanding about the Commissioning and Decommissioning of the datanodes in Hadoop.

Thanks

Mandy

Leave a Reply