Learn Devops tools online from the blog. Devops Institute Ahmedabad.

Install hadoop on ubuntu 10.04 LTS server installion.
Install ubuntu l0.04 LTS,
Update,upgrade and install ssh
#apt-get update
#apt-get upgrade
#apt-get install ssh

Install sun-6-java jdk
# See https://launchpad.net/~ferramroberto/

$ sudo apt-get install python-software-properties
$ sudo add-apt-repository ppa:ferramroberto/java

# Update the source list
$ sudo apt-get update

# Install Sun Java 6 JDK
$ sudo apt-get install sun-java6-jdk

$ sudo update-java-alternatives -s java-6-sun

Create new user hadoop and group
$ sudo addgroup hadoop 
$ sudo adduser --ingroup hadoop hduser

Generate ssh key to auto login to manager
#su - hduser
#ssh-keygen -t rsa -P ""
#cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys 

Disable ipv6 and reboot 
#vim /etc/modprobe.d/blacklist
add new line in it
blacklist ipv6  
Or
#disable ipv6 
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
You have to reboot your machine in order to make the changes take effect.
You can check whether IPv6 is enabled on your machine
$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
 0 means IPv6 is enabled, a value of 1 means 
disabled (that’s what we want).

Hadoop Distributed File System (HDFS)
Image copy by javacodegeeks.com
















Download hadoop from mirror site
$ cd /usr/local 
$ sudo tar xzf hadoop-1.0.3.tar.gz 
$ sudo mv hadoop-1.0.3 hadoop 
$ sudo chown -R hduser:hadoop hadoop

Confirm java home folder
#ls -l `whereis javac`

Modify hadoop home folder hadoop-env.sh 
#vim hadoop/conf/hadoop-env.sh
uncomment export JAVA_HOME and modify it
export JAVA_HOME = /usr/lib/jvm/java-6-openjdk/

Config hadoop config file 
#vim hadoop/conf/core-site.xml
add these line into it
<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/hadoop/tmp/dir/hadoop-hadoop</value>
</property>
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost</value>
</property>

Config hadoop file with hdfs-site.xml
#vim hadoop/conf/hdfs-site.xml
add these line into it
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

Config hadoop file with mapred-site.xml
#vim hadoop/conf/mapred-site.xml
add these line into it
<property>
    <name>mapred.job.tracker</name>
    <value>localhost:54311</value>
</property>

Formating the namenode
#hadoop/bin/hadoop namenode -format

Start cluster
#hadoop/bin/start-all.sh

Check hadoop process
#hadoop/bin/jps

Use netstat to check all service running status
#netstat -plten | grep java

Stop cluster
#hadoop/bin/stop-all.sh

Start cluster
#hadoop/bin/start-all.sh

Mkdir a folder for gutenberg and touch three files with contents
#mkdir /tmp/gutenberg
#cd /tmp/gutenberg
#vim 1.txt
#vim 2.txt
#vim 3.txt

Use hadoop fs copyFromLocal copy files to hdfs folder
#hadoop/bin/hadoop fs -copyFromLocal /tmp/gutenberg gutenber

Check hdfs folder content
#hadoop/bin/hadoop fs -ls 
#hadoop/bin/hadoop fs -ls gutenberg

Use java wordcount to calculate the words number
#hadoop/bin/hadoop jar hadoop-mapred-examples-0.21.0.jar 
wordcount gutenberg gutenberg-output


Hadoop Web Interfaces

    http://localhost:50070/ – web UI of the NameNode daemon
    http://localhost:50030/ – web UI of the JobTracker daemon
    http://localhost:50060/ – web UI of the TaskTracker daemon



Vishal Vyas

No comments:

Post a Comment