一、安装linux系统，拷贝2台，总共三台；

二、设置IP

虚拟机设置为桥接模式

/etc/network/interfaces

auto lo iface lo inet loopback

iface eth0 inet static

address 192.168.58.101

netmask 255.255.255.0

gateway 192.168.58.2

auto eth0

 

/etc/resolv.conf 如果文件不存在，就手动创建

# Generated by NetworkManager domain localdomain search localdomain

nameserver 192.168.58.2

sudo /etc/init.d/networking restart

route add default gw 192.168.58.2

三、设置hosts

sudo chmod 777 hosts vi /etc/hosts

192.168.58.101 sniper101

192.168.58.102 sniper102 192.168.58.103 sniper103

192.168.58.101 ubuntu  注意，这个一定要配置，否则在格式化name的时候会报错

127.0.0.1 ubuntu localhost.localdomain localhost ::1 ubuntu localhost6.localdomain6 localhost6 127.0.1.1 ubuntu

# The following lines are desirable for IPv6 capable hosts

::1     localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts

四、安装SSH ssh

1.安装openssh-server

    sudo dpkg -i openssh-client_1%3a5.5p1-4ubuntu6_i386.deb     sudo dpkg -i openssh-server_1%3a5.5p1-4ubuntu6_i386.deb         如果连网的情况下可以sudo apt-get install openssh-server进行安装     2. 启动ssh-server。 $ /etc/init.d/ssh restart service ssh restart   3. 确认ssh-server已经正常工作。 $ netstat -tlp tcp6    0    0 *:ssh                *:*                   LISTEN     - 看到上面这一行输出说明ssh-server已经在运行了。   4. 在sniper101通过ssh登录sniper102 $ ssh sniper102

五、配置SSH免密

分别登陆sniper101、sniper102、sniper103，分别执行以下命令

cd ~

ssh-keygen -t rsa

登陆sniper101

cd ~ cd .ssh cp id_rsa.pub authorized_keys scp authorized_keys sniper102:/home/grid/.ssh

ssh sniper102

cd ~/.ssh cat id_rsa.pub >> authorized_keys scp authorized_keys sniper103:/home/grid/.ssh exit

ssh sniper103

cd ~/.ssh cat id_rsa.pub >> authorized_keys scp authorized_keys sniper101:/home/grid/.ssh scp authorized_keys sniper102:/home/grid/.ssh exit

六、关闭防火墙

centos  chkconfig iptables off  不重启不会生效：  service iptables stop  不需要重启

ubuntu

 ufw disable  sudo iptables -F  sudo ufw status

七、安装JDK

 su grid

 sudo -i  passwd root

 sudo dpkg -l 

 sudo dpkg -l | grep wget

 cd /usr

 mkdir java

 tar -zxvf eclipse-SDK-4.2-linux-gtk.tar.gz

 cp ~/Desktop/jdk-6u24-linux-i586.bin /usr/java  ./jdk-6u24-linux-i586.bin

 ln -s /usr/java/jdk-xxx /usr/jdk

 vi /etc/profile 

  JAVA_HOME=/usr/jdk   CLASSPATH=$JAVA_HOME/lib   PATH=$PATH:$JAVA_HOME/bin

  export JAVA_HOME CLASSPATH PATH

 source /etc/profile    java -version

八、安装hadoop 下载hadoop-2.5.2.tar.gz，设置HADOOP_HOME

1.搭建手工切换的ha

----------------------------- namenode：sniper101和sniper102 datanode：sniper101,sniper102,sniper103 journalnode：sniper101,sniper102,sniper103 resourcemanager：sniper101 nodemanager：sniper101,sniper102,sniper103

1.1 配置文件(hadoop-env.sh、core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml、slaves)

1.1.1 hadoop-env.sh export JAVA_HOME=/usr/java/jdk

1.1.2 core-site.xml

<property>

<name>fs.defaultFS</name> <value>hdfs://sniper101</value> </property>

<property>

<name>hadoop.tmp.dir</name> <value>/home/grid/soft/hadoop/tmp</value> </property>

1.1.3 hdfs-site.xml

<property>

<name>dfs.replication</name> <value>3</value> </property>

<property>

<name>dfs.nameservices</name> <value>sniper1</value> </property>

<property>

<name>dfs.ha.namenodes.sniper1</name> <value>sniper101,sniper102</value> </property>

<property>

<name>dfs.namenode.rpc-address.sniper1.sniper101</name> <value>sniper101:9000</value> </property>

<property>

<name>dfs.namenode.http-address.sniper1.sniper101</name> <value>sniper101:50070</value> </property>

<property>

<name>dfs.namenode.rpc-address.sniper1.sniper102</name> <value>sniper102:9000</value> </property>

<property>

<name>dfs.namenode.http-address.sniper1.sniper102</name> <value>sniper102:50070</value> </property>

<property>

<name>dfs.ha.automatic-failover.enabled.sniper1</name> <value>false</value> </property>

<property>

<name>dfs.namenode.shared.edits.dir</name> <value>qjournal://sniper101:8485;sniper102:8485;sniper103:8485/cluster1</value> </property>

<property>

<name>dfs.journalnode.edits.dir</name> <value>/home/grid/soft/hadoop/tmp/journal</value> </property>

<property>

<name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property>

<property>

<name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/grid/.ssh/id_rsa</value> </property>

<property>

<name>dfs.client.failover.proxy.provider.cluster1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property>

1.1.4 yarn-site.xml

<property>

<name>yarn.resourcemanager.hostname</name> <value>sniper101</value> </property>

<property>

<name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>

1.1.5 mapred-site.xml

<property>

 <name>mapreduce.framework.name</name>  <value>yarn</value> </property>

1.1.6 slaves

sniper101 sniper102 sniper103

1.1.7 把sniper101上的hadoop文件夹复制到sniper102、sniper103节点

1.2 启动journalnode集群

  在sniper101、sniper102、sniper103上分别执行hadoop/sbin/hadoop-daemon.sh start journalnode

1.3 格式化namenode、启动namenode

  在sniper101上执行hadoop/bin/hdfs namenode -format   在sniper101上分别执行hadoop/sbin/hadoop-daemon.sh start namenode   在sniper102上执行hadoop/bin/hdfs namenode -bootstrapStandby   在sniper102上分别执行hadoop/sbin/hadoop-daemon.sh start namenode   在sniper101上执行hadoop/bin/hdfs haadmin -failover --forceactive sniper101 sniper102

1.4 启动datanode

  在sniper101上分别执行hadoop/sbin/hadoop-daemons.sh start datanode

1.5 启动resourcemanager和nodemanager

  在sniper101上执行 hadoop/sbin/start-yarn.sh start resourcemanager

测试是否正常 bin/hadoop  jar  share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar  wordcount  /README.txt   /out