Skip to content

antony@notes:~/data-platform$ cat "建置-Apache-Hadoop-3.3.3.md"

建置 Apache Hadoop 3.3.3

2022-08-16· data-platform ·資料科技平台

建置 Apache Hadoop 3.3.3

[TOC]

Create VM-US2204

Create VM-US2204 連結

Create VM-US2204 Cluster

:::spoiler 1. 啟動 US2004 虛擬主機, 執行以下設定命令

C:\Users\bigred>ssh bigboss@192.168.61.110
bigboss@192.168.61.110s password:

# 系統更新
$ sudo apt update; sudo apt upgrade; sudo apt autoremove

#　安裝套件以及 Hadoop 所需的套件
$ sudo apt-get install tree unzip net-tools openjdk-8-jdk libisal-dev libsnappy-dev libz-dev

# 允許 ssh process 的環境變數，會去讀取 ~/.ssh/environment
$ sudo nano /etc/ssh/sshd_config
.......
PermitUserEnvironment yes 

# 當我們第一次 ssh 連到別台機器時，直接收公鑰，不要再問我 yes 或 no 
$ sudo nano /etc/ssh/ssh_config
.........
StrictHostKeyChecking no

# 產生公私鑰
$ ssh-keygen -t rsa -P ''
Enter file in which to save the key (/home/bigred/.ssh/id_rsa): 按 Enter

# 複製公鑰成 authorized_keys 
$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

# 設定 /etc/hosts
$ sudo nano /etc/hosts
127.0.0.1 localhost
127.0.1.1 cvn80

192.168.66.110 dta1
192.168.66.111 dts1
192.168.66.112 dts2
192.168.66.113 dts3
192.168.66.114 dtm1
192.168.66.115 dtm2
192.168.66.120 dtw1
192.168.66.121 dtw2
192.168.66.122 dtw3
192.168.66.123 dtw4
192.168.66.124 dtw5
192.168.66.125 dtw6

# 設定 sudo 群組執行 sudo 免密碼 
# 因 bigboss 在 sudo 群組裡，在第 50 行
$ sudo nano -l /etc/sudoers
%sudo   ALL=(ALL:ALL) NOPASSWD:ALL 

# 設定 bigboss 帳號自動登入 ( 在第 40 行 )
$ sudo nano -l /lib/systemd/system/getty@.service
ExecStart=-/sbin/agetty --autologin bigboss --keep-baud 115200,38400,9600 %I $TERM

# 編輯 /etc/profile，新增以下 55 行~ 71 行的內容
$ sudo nano /etc/profile
gw=$(/sbin/route -n | grep -e "^0.0.0.0 " | head -n 1)
export GWIF=${gw##* }
ips=$(/sbin/ifconfig $GWIF | grep 'inet ')
export IP=$(echo $ips | cut -d' ' -f2 | cut -d':' -f2)
export NETID=${IP%.*}
export GW=$(/sbin/route -n | grep -e '^0.0.0.0' | tr -s \ - | cut -d ' ' -f2)
export PATH=/home/bigboss/bin:/home/bigboss/vmusdt/bin:$PATH
clear; sleep 3
v=$(cat /etc/os-release | grep 'PRETTY_NAME' | cut -d '=' -f 2)
echo "Welcome to DT Platform V 22.07 ($v)"
[ "$IP" != "" ] && echo "IP : $IP"
echo ""

alias ping='ping -c 4 '
alias dir='ls -alh '
alias poweroff='sudo poweroff'
alias reboot='sudo reboot'

# uninstall cloud-init
## Disable all services (uncheck everything except "None"):
$ sudo dpkg-reconfigure cloud-init

## Uninstall the package and delete the folders
$ sudo dpkg-reconfigure cloud-init
$ sudo apt-get purge cloud-init
$ sudo rm -rf /etc/cloud/ && sudo rm -rf /var/lib/cloud/

## Restart the computer
$ sudo reboot

# 關閉 ipv6
## 雙引號裡新增 ipv6.disable=1 , 在第 10, 11 行
$ sudo nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"
GRUB_CMDLINE_LINUX="ipv6.disable=1"

# Now update the GRUB bootloader configuration files for the changes to take effect with the following command:
$ sudo update-grub2

## Restart the computer
$ sudo reboot

## 檢查是否符合預期
$ test -f /proc/net/if_inet6 && echo "IPv6 is enabled." || echo "IPv6 is disabled."
IPv6 is disabled.

# 檢查 java 版本
$ java -version check
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~20.04-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

## 關機
$ sudo poweroff

sudo apt autoremove 解釋 sshd_config 詳細說明連結 ssh_config 詳細說明連結 Ubuntu Server 設定帳號自動登入 :::

:::spoiler 2. 將以上兩台母機 壓縮，並透過 scp 命令，複製到所在的 落地雲 主機，然後解壓縮至使用者家目錄。 參考圖片 :::

:::spoiler 3. 啟動 VMware Workstation Player, 設定 資科平台 十二部虛擬主機

   設定 虛擬主機 名稱 
      dta1, dtm1, dtm2, dts1..dts3,dtw1..6
   設定 虛擬主機 記憶體 
      dta1 -> 6G ( 6144 mb )
      dts1 -> 6G 
      dts2 -> 6G 
      dts3 -> 6G 
      dtm1 -> 4G ( 4096 mb )
      dtm2 -> 4G
      dtw1 -> 10G ( 10240 mb )
      dtw2 -> 10G 
      dtw3 -> 10G 
      dtw4 -> 10G 
      dtw5 -> 10G 
      dtw6 -> 10G 
   設定 虛擬主機 網卡
     Network Adapter -> Bridge

:::

:::spoiler 4. 啟動 資科平台 虛擬主機, 然後設定網路系統

在 Windows 命令提示字元 視窗, 執行以下命令, 登入虛擬主機
c:\>ssh bigboss@192.168.66.xx

設定固定 IP 
$ sudo nano /etc/netplan/00-installer-config.yaml 
network:
  version: 2
  renderer: networkd
  ethernets:
    ens32:
      dhcp4: no
      addresses:
        - 192.168.66.xx/24
      routes:
          - to: default
            via: 192.168.66.2
      nameservers:
          addresses: [8.8.8.8, 168.95.1.1]

測試網路
$ sudo netplan apply 

設定電腦名稱
$ sudo nano /etc/hostname
dta1

$ sudo reboot

:::

資科平台系統建置

:::spoiler 下載 資料科技 平台套件檔

在 Windows 系統的 cmd 視窗, 執行以下命令
$ ssh bigboss@<dta1 IP>

$ cd; wget http://www.oc99.org/zip/vmusdt2022.zip; unzip vmusdt2022.zip 

$ dir vmusdt      ## vmusdt (Virtual Machine Ubuntu Server Data technology)
total 20K
drwxr-xr-x 5 bigboss bigboss 4.0K Jul  4 02:45 .
drwxr-x--- 9 bigboss bigboss 4.0K Jul  5 11:35 ..
drwxr-xr-x 2 bigboss bigboss 4.0K Jul  5 11:34 bin
drwxr-xr-x 5 bigboss bigboss 4.0K May  9 18:57 hdp210
drwxr-xr-x 5 bigboss bigboss 4.0K Jul  6 03:36 hdp33

$ dir vmusdt/bin
total 28K
drwxr-xr-x 2 bigboss bigboss 4.0K Jul  5 11:34 .
drwxr-xr-x 5 bigboss bigboss 4.0K Jul  4 02:45 ..
-rwxr-xr-x 1 bigboss bigboss 1.2K Jul  5 03:21 dtconf
-rwxr-xr-x 1 bigboss bigboss  974 Jul  4 05:06 dtcopy
-rwxr-xr-x 1 bigboss bigboss  280 Jul  4 08:49 dthdpchk
-rwxr-xr-x 1 bigboss bigboss  472 Jul  4 09:08 dtsyschk
-rwxr-xr-x 1 bigboss bigboss  771 Jul  4 13:38 dtwget

# 因為有把/home/bigboss/vmusdt/bin加到PATH環境變數，所以才可以直接執行vmusdt/bin目錄裡面的命令
$ echo $PATH
/home/bigboss/bin:/home/bigboss/vmusdt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bi

$ cd ~/vmusdt; dir hdp33
total 32K
drwxr-xr-x 5 bigboss bigboss 4.0K Jul  6 03:36 .
drwxr-xr-x 5 bigboss bigboss 4.0K Jul  4 02:45 ..
drwxr-xr-x 2 bigboss bigboss 4.0K Jul  6 07:09 bin
drwxr-xr-x 7 bigboss bigboss 4.0K Jul  4 07:22 conf
-rw-r--r-- 1 bigboss bigboss  420 Jul  4 02:32 dlist
-rw-r--r-- 1 bigboss bigboss  889 Jul  4 07:35 environment
drwxrwxr-x 7 bigboss bigboss 4.0K Jul  4 02:45 opt
-rwxr-xr-x 1 bigboss bigboss 2.6K Jul  4 07:33 us.bash

us.bash –> 設定資科平台所需的環境變數 opt –> 資科平台套件的存放目錄 bin –> 存放部署或建制資科平台所需的命令 conf –> 存放資科平台套件設定檔的目錄 dlist –> 資科平台套件下載連結清單 environment –> 設定 ssh 直接執行指令時的環境變數 :::

修改 資科平台 程式 ( ~/vmusdt/bin/ )

:::spoiler 修改 dtwget

修改命令 :

$ nano -l ~/vmusdt/bin/dtwget

修改內容 :

第 5 行，更改為 [ ! -d ~/vmusdt/hdp${1} ] && echo "~/vmusdt/hdp${1} not found" && exit 1
第 6 行，更改為 [ ! -f ~/vmusdt/hdp${1}/dlist ] && echo "~/vmusdt/hdp${1}/dlist not found" && exit 1
第 27 行，更改為 sudo chown -R bigboss:bigboss ~/vmusdt/hdp${1}/opt

修改結果 :

#!/bin/bash

[ "$#" != 1 ] && echo "dtwget version" && exit 1

[ ! -d ~/vmusdt/hdp${1} ] && echo "~/vmusdt/hdp${1} not found" && exit 1
[ ! -f ~/vmusdt/hdp${1}/dlist ] && echo "~/vmusdt/hdp${1}/dlist not found" && exit 1

read -p "download DT packages ? (YES/NO) " ans
if [ "$ans" == "YES" ]; then
   [ -d ~/vmusdt/hdp${1}/opt ] && rm -r ~/vmusdt/hdp${1}/opt &>/dev/null
   mkdir -p ~/vmusdt/hdp${1}/opt

   while read line; do
     wget $line &>/dev/null
     if [ "$?" == "0" ]; then
        n=${line##*/}
        tar xvfz $n -C ~/vmusdt/hdp${1}/opt &>/dev/null
        [ "$?" == "0" ] && echo "$n ok"
        rm $n
     else
        n=${line##*/}
        echo "$n failed"
     fi
   done < ~/vmusdt/hdp${1}/dlist
fi

sudo chown -R bigboss:bigboss ~/vmusdt/hdp${1}/opt

:::

:::spoiler 修改 dtcopy

修改命令 :

$ nano -l  ~/vmusdt/bin/dtcopy

修改內容 :

第 3 行 刪除 dta2 ，新增 dts1 dts2 dts3 dtw5 dtw6
第 25 行 bigred -- > bigboss
第 28 行 bigred -- > bigboss

修改結果 :

$ nano ~/vmusdt/bin/dtcopy
#!/bin/bash

w="dta1 dts1 dts2 dts3 dtm1 dtm2 dtw1 dtw2 dtw3 dtw4 dtw5 dtw6"

[ "$#" != 1 ] && echo "dtcopy version" && exit 1

[ ! -d ~/vmusdt/hdp${1}/opt ] && echo "~/vmusdt/hdp${1}/opt not found" && exit 1

for x in $w
do
   nc -w 1 -z $x 22 &>/dev/null
   [ "$?" != "0" ] && continue

   ssh $x sudo mkdir -p /opt/bin &>/dev/null
   ssh $x sudo chmod -R 777 /opt &>/dev/null

   n=$(ssh $x ls /opt)
   for y in $n
   do
     ([ "$y" == "cni" ] || [ "$y" == "bin" ] || [ "$y" == "nfs" ]) && continue
     ssh $x sudo rm -r /opt/$y
   done

   scp -r ~/vmusdt/hdp${1}/opt/* bigboss@$x:/opt/ &>/dev/null
   [ "$?" == "0" ] && echo "$x /opt/ ok"

   scp ~/vmusdt/hdp${1}/bin/*  bigboss@$x:/opt/bin &>/dev/null
   [ "$?" == "0" ] && echo "$x /opt/bin ok"

   scp /etc/hosts $x:~ &>/dev/null
   ssh $x 'sudo mv hosts /etc/hosts'
   [ "$?" == "0" ] && echo "$x /etc/hosts ok"

   ssh $x sudo chown -R root:root /opt
   ssh $x sudo chmod -R 0777 /opt

   echo ""
done

:::

:::spoiler 修改 dtconf

修改命令 :

nano -l ~/vmusdt/bin/dtconf

修改內容 :

第 3 行 ，更改為 w="dta1 dts1 dts2 sts3 dtm1 dtm2 dtw1 dtw2 dtw3 dtw4 dtw5 dtw6"
第 19 行，更改為 [ "$?" == "0" ] && scp ~/vmusdt/hdp${1}/conf/$y/* bigboss@$x:/opt/$y/etc/hadoop/ &>/dev/null && echo "$x $y config ok" && continue
第 20 行，更改為 [ -d ~/vmusdt/hdp${1}/conf/$y ] && scp ~/vmusdt/hdp${1}/conf/$y/* bigboss@$x:/opt/$y/conf/ &>/dev/null
第 25 行，更改為 scp -r ~/vmusdt/hdp${1}/environment bigboss@$x:~/.ssh/environment &>/dev/null
第 28 行，更改為 scp -r ~/vmusdt/hdp${1}/us.bash bigboss@$x:~/us.bash &>/dev/null
第 31 行，更改為 ssh $x 'cat /etc/profile | grep '/home/bigboss/us.bash' &>/dev/null'
第 32 行，更改為 [ "$?" != "0" ] && ssh $x 'echo source /home/bigboss/us.bash | sudo tee -a /etc/profile &>/dev/null'

修改結果 :

#!/bin/bash

w="dta1 dts1 dts2 dts3 dtm1 dtm2 dtw1 dtw2 dtw3 dtw4 dtw5 dtw6"

[ "$#" != 1 ] && echo "dtconf version" && exit 1

[ ! -d ~/vmusdt/hdp${1}/opt ] && echo "~/vmusdt/hdp${1}/opt not found" && exit 1

n=$(ls ~/vmusdt/hdp${1}/opt)

for x in $w
do
   nc -w 1 -z $x 22 &>/dev/null
   [ "$?" != "0" ] && continue

   for y in $n
   do
     echo "$y" | grep 'hadoop-' &>/dev/null
     [ "$?" == "0" ] && scp ~/vmusdt/hdp${1}/conf/$y/* bigboss@$x:/opt/$y/etc/hadoop/ &>/dev/null && echo "$x $y config ok" && continue
     [ -d ~/vmusdt/hdp${1}/conf/$y ] && scp ~/vmusdt/hdp${1}/conf/$y/* bigboss@$x:/opt/$y/conf/ &>/dev/null
     [ "$?" == "0" ] && echo "$x $y config ok"
   done

   ssh $x 'sudo rm /etc/environment &>/dev/null'
   scp -r ~/vmusdt/hdp${1}/environment bigboss@$x:~/.ssh/environment &>/dev/null
   [ "$?" == "0" ] && echo "$x environment ok"

   scp -r ~/vmusdt/hdp${1}/us.bash bigboss@$x:~/us.bash &>/dev/null
   [ "$?" == "0" ] && echo "$x us.bash ok"

   ssh $x 'cat /etc/profile | grep '/home/bigboss/us.bash' &>/dev/null'
   [ "$?" != "0" ] && ssh $x 'echo source /home/bigboss/us.bash | sudo tee -a /etc/profile &>/dev/null'
   echo "$x /etc/profile ok"

   echo ""
done

:::

:::spoiler 修改 dthdpchk

修改命令 :

nano bin/dthdpchk

修改內容 :

m="dta1 dts1 dts2 dts3 dtm1 dtm2"
w="dtw1 dtw2 dtw3 dtw4 dtw5 dtw6"

修改結果 :

#!/bin/bash

m="dta1 dts1 dts2 dts3 dtm1 dtm2"
w="dtw1 dtw2 dtw3 dtw4 dtw5 dtw6"

for n in $m $w
do
   nc -w 1 -z $n 22 &>/dev/null
   if [ "$?" == "0" ]; then
      echo "[$n]"
      ssh $n 'hadoop version | head -n 1'
      ssh $n 'hadoop checknative'

      echo ""
   fi
done

:::

:::spoiler 修改 dtsyschk

修改命令 :

$ nano ~/vmusdt/bin/dtsyschk

修改內容 :

m="dtm1 dtm2 dts1 dts2 dts3"
w="dtw1 dtw2 dtw3 dtw4 dtw5 dtw6"

修改結果 :

#!/bin/bash

m="dtm1 dtm2 dts1 dts2 dts3"
w="dtw1 dtw2 dtw3 dtw4 dtw5 dtw6"

for n in $m $w
do
   nc -w 1 -z $n 22 &>/dev/null
   if [ "$?" == "0" ]; then
      ssh $n exit &>/dev/null
      [ "$?" == "0" ] && echo -n "$n ssh ok"

      ssh $n 'java -version' &>/dev/null
      [ "$?" == "0" ] && echo -n ", jdk8 ok"

      ssh $n 'which python3' &>/dev/null
      if [ "$?" == "0" ]; then
         echo ", python3 ok"
      else
         echo ""
      fi
   fi
done

:::

修改 環境變數 套件清單檔 ( ~/vmusdt/hdp33/ )

:::spoiler 修改 資科平台所需的環境變數

pig, spark, Hive 會去讀取 us.bash，hadoop 不會

修改命令 :

nano -l ~/vmusdt/hdp33/us.bash

修改內容 :

第 2 行，更改為 adminuser=bigboss
第 3 行，確認 java-8-openjdk
第 4 行，更改為 export HADOOP_HOME=/opt/hadoop-3.3.3
第 14 行，更改為 /home/bigboss/jhslog
第 27 行，更改為 export HIVE_HOME=/opt/apache-hive-3.1.3-bin
第 30 行，更改為 export PATH=/home/bigboss/vmusdt/bin:$PATH
第 33 行，更改為 export HBASE_HOME=/opt/hbase-2.4.13
第 34 行，更改為 export HBASE_CONF_DIR=/opt/hbase-2.4.13/conf

修改結果 :

#!/bin/bash
adminuser=bigboss
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk/jre  ## 資料科技平台所跑的 Java 程式的家
export HADOOP_HOME=/opt/hadoop-3.3.3
export HADOOP_LOG_DIR=/tmp
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
# export HADOOP_USER_CLASSPATH_FIRST=true
[ -z $HADOOP_USER_NAME ] && [ $SHELL == '/bin/bash' ] && declare -r HADOOP_USER_NAME=$USER

# JobHistory log file path, 會自動產生目錄
# export HADOOP_MAPRED_LOG_DIR="/home/bigboss/jhslog"

export YARN_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export YARN_LOG_DIR=/tmp

# hadoop 3.2.1 及 3.1.3 這二個版本只要執行檔案傳送命令就會出現以下訊息
# hadoop fs -put -f /etc/passwd /tmp
# SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
export HADOOP_ROOT_LOGGER="WARN,console"

export PIG_HOME=/opt/pig-0.17.0
export PIG_HEAPSIZE=512
export HIVE_HOME=/opt/apache-hive-3.1.3-bin

export PATH=/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH=/usr/lib/jvm/java-1.8-openjdk/bin:/home/bigboss/dt/bin:$PATH
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PIG_HOME/bin:$HIVE_HOME/bin

export HBASE_HOME=/opt/hbase-2.4.13
export HBASE_CONF_DIR=/opt/hbase-2.4.13/conf
export ZOOKEEPER_HOME=/opt/apache-zookeeper-3.6.3-bin
export ZOO_LOG_DIR=/tmp/logs
export PATH=$PATH:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin

#export ZEPPELIN_HOME=/opt/zeppelin-0.9.0-bin-all
#export PATH=$PATH:$ZEPPELIN_HOME/bin

export SPARK_HOME=/opt/spark-3.2.1-bin-hadoop3.2
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PYSPARK_PYTHON=/usr/bin/python3
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

if [ -d /opt/jupyter/miniconda3/bin ];then
   export ANACONDA_ROOT=/opt/jupyter/miniconda3
   export PATH=$ANACONDA_ROOT/bin:$PATH
   export PYSPARK_PYTHON=$ANACONDA_ROOT/bin/python
fi

if [ "$USER" != "" ]; then

   [ -f /home/$USER/dkc.env ] && source /home/$USER/dkc.env

   if [ ! -d metastore_db ]; then
      #hn=$(hostname)
      #if [ ${hn:0:3} == "adm" ] || [ ${hn:0:2} == "ds" ] || [ ${hn:0:8} == "zeppelin" ]; then
      echo -n "build derby database ..."
      schematool -initSchema -dbType derby &>/dev/null
      if [ "$?" == "0" ]; then
         echo " ok"
         echo "set hive.cli.print.current.db=true;" > .hiverc
         echo "set hive.metastore.warehouse.dir=/user/$USER/hive;" >> .hiverc
         echo "set hive.exec.scratchdir=/user/$USER/tmp;" >> .hiverc
      fi
   fi

   alias nano='nano -Ynone'
   alias dir='ls -alh'
   alias ssh='ssh -q'
fi

:::

:::spoiler 修改 ssh 直接執行命令時的環境變數檔案

修改命令 :

nano -l ~/vmusdt/hdp33/environment

修改內容 :

第 1 行，更改為 JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
第 2 行，更改為 HADOOP_HOME=/opt/hadoop-3.3.3
第 4 行，更改為 LD_LIBRARY_PATH=/opt/hadoop-3.3.3/lib/native/
第 5 行，更改為 HADOOP_COMMON_LIB_NATIVE_DIR=/opt/hadoop-3.3.3/lib/native
第 6 行，更改為 HADOOP_OPTS=-Djava.library.path=/opt/hadoop-3.3.3/lib/native
第 7 行，更改為 HADOOP_MAPRED_LOG_DIR=/home/bigboss/jhslog
第 8 行，更改為 YARN_HOME=/opt/hadoop-3.3.3
第 14 行，更改為 HBASE_HOME=/opt/hbase-2.4.13
第 15 行，更改為 HBASE_CONF_DIR=/opt/hbase-2.4.13/conf
第 17 行，更改為 
PATH=/root/bin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hadoop-3.3.3/bin:/opt/hadoop-3.3.3/sbin:/opt/pig-0.17.0/bin:/opt/apache-hive-3.1.3-bin/bin:/opt/spark-3.2.1-bin-hadoop3.2/bin:/opt/spark-3.2.1-bin-hadoop3.2/sbin:/opt/hbase-2.4.13/bin:/opt/apache-zookeeper-3.6.3-bin/bin

修改結果 :

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre
HADOOP_HOME=/opt/hadoop-3.3.3
HADOOP_LOG_DIR=/tmp
LD_LIBRARY_PATH=/opt/hadoop-3.3.3/lib/native/
HADOOP_COMMON_LIB_NATIVE_DIR=/opt/hadoop-3.3.3/lib/native
HADOOP_OPTS=-Djava.library.path=/opt/hadoop-3.3.3/lib/native
HADOOP_MAPRED_LOG_DIR=/home/bigboss/jhslog
YARN_HOME=/opt/hadoop-3.3.3
YARN_LOG_DIR=/tmp
PIG_HOME=/opt/pig-0.17.0/
PIG_HEAPSIZE=512
SPARK_HOME=/opt/spark-3.2.1-bin-hadoop3.2
HIVE_HOME=/opt/apache-hive-3.1.3-bin
HBASE_HOME=/opt/hbase-2.4.11
HBASE_CONF_DIR=/opt/hbase-2.4.11/conf
HBASE_LOG_DIR=/tmp
ZOO_LOG_DIR=/tmp/logs
PATH=/root/bin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hadoop-3.3.3/bin:/opt/hadoop-3.3.3/sbin:/opt/pig-0.17.0/bin:/opt/apache-hive-3.1.3-bin/bin:/opt/spark-3.2.1-bin-hadoop3.2/bin:/opt/spark-3.2.1-bin-hadoop3.2/sbin:/opt/hbase-2.4.11/bin:/opt/apache-zookeeper-3.6.3-bin/bin

:::

:::spoiler 修改 下載資科平台套件 清單檔

修改命令 :

$ nano ~/vmusdt/hdp33/dlist

修改內容 :

第 1 行，更改為
https://dlcdn.apache.org/hadoop/common/hadoop-3.3.3/hadoop-3.3.3.tar.gz

第 5 行，更改為
https://dlcdn.apache.org/hbase/2.4.13/hbase-2.4.13-bin.tar.gz

修改結果 :

https://dlcdn.apache.org/hadoop/common/hadoop-3.3.3/hadoop-3.3.3.tar.gz
https://dlcdn.apache.org/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
https://dlcdn.apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz
https://dlcdn.apache.org/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.2.tgz
https://dlcdn.apache.org/hbase/2.4.13/hbase-2.4.13-bin.tar.gz
https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz

:::

修改 hadoop 設定檔 ( ~/vmusdt/hdp33/conf/hadoop-3.3.3/ )

:::spoiler 修改 設定檔目錄名稱

$ mv ~/vmusdt/hdp33/conf/hadoop-3.3.2/ ~/vmusdt/hdp33/conf/hadoop-3.3.3/

$ mv ~/vmusdt/hdp33/conf/apache-hive-3.1.2-bin/ ~/vmusdt/hdp33/conf/apache-hive-3.1.3-bin/

$ mv ~/vmusdt/hdp33/conf/hbase-2.4.12 ~/vmusdt/hdp33/conf/hbase-2.4.13

檢查設定檔和套件檔的目錄名稱要一致

bigboss@dta1:~/vmusdt$ dir hdp33/conf
total 28K
drwxr-xr-x 7 bigboss bigboss 4.0K Jul  8 12:51 .
drwxr-xr-x 5 bigboss bigboss 4.0K Jul  8 13:04 ..
drwxr-xr-x 2 bigboss bigboss 4.0K Jul  8 12:59 apache-hive-3.1.3-bin
drwxr-xr-x 2 bigboss bigboss 4.0K Feb  6 16:00 apache-zookeeper-3.6.3-bin
drwxr-xr-x 2 bigboss bigboss 4.0K Jul  8 12:28 hadoop-3.3.3
drwxr-xr-x 2 bigboss bigboss 4.0K Jul  8 12:55 hbase-2.4.13
drwxr-xr-x 2 bigboss bigboss 4.0K May  9 18:54 spark-3.2.1-bin-hadoop3.2
bigboss@dta1:~/vmusdt$ dir /opt
total 32K
drwxrwxrwx  8 root root 4.0K Jul  4 05:28 .
drwxr-xr-x 18 root root 4.0K Jul  3 05:46 ..
drwxrwxrwx 10 root root 4.0K Jul  4 05:27 apache-hive-3.1.3-bin
drwxrwxrwx  6 root root 4.0K Jul  4 05:27 apache-zookeeper-3.6.3-bin
drwxrwxrwx  2 root root 4.0K Jul  4 05:28 bin
drwxrwxrwx 10 root root 4.0K Jul  4 05:28 hadoop-3.3.3
drwxrwxrwx 16 root root 4.0K Jul  4 05:28 pig-0.17.0
drwxrwxrwx 13 root root 4.0K Jul  4 05:28 spark-3.2.1-bin-hadoop3.2

:::

:::spoiler 修改 hadoop-env.sh

修改命令

nano -l ~/vmusdt/hdp33/conf/hadoop-3.3.3/hadoop-env.sh

修改第 54 行，java-11-openjdk-amd64 –> java-8-openjdk-amd64

修改結果 :

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/

Hadoop的程式，會到 hadoop-env.sh 這個檔案來讀取設定檔

:::

:::spoiler 修改 hdfs-site.xml

修改命令 :

$ nano -l ~/vmusdt/hdp33/conf/hadoop-3.3.3/hdfs-site.xml

修改內容 :

第 10 行，更改為 <value>file:/home/bigboss/nn</value>
第 14 行，更改為 <value>file:/home/bigboss/dn</value>
第 80 行，更改為 <value>/opt/hadoop-3.3.3/etc/hadoop/hdfs.allow</value>
第 84 行，更改為 <value>/opt/hadoop-3.3.3/etc/hadoop/hdfs.decommission</value>

修改結果 :

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/home/bigboss/nn</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/home/bigboss/dn</value>
  </property>
  <property>
    <name>dfs.permissions.superusergroup</name>
    <value>bigboss</value>
  </property>

  <property>
    <name>dfs.namenode.acls.enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>false</value>
    <description>
     1. webhdfs 功能有關閉, 但 Port 50070 不會被關掉
     2. NameNode 內定會啟動 WebHDFS 網站
     3. webhdfs client 必須要能連接到所有 Datanode, 才能執行 HDFS 功能
    </description>
  </property>

  <!--
  <property>
    <name>dfs.namenode.accesstime.precision</name>
    <value>3600000</value>
    <description>
      The access time for HDFS file is precise up to this value.
      The default value is 1 hour. Setting a value of 0 disables
      access times for HDFS.
    </description>
  </property>
  -->
  <!--
  <property>
    <name>dfs.nfs3.dump.dir</name>
    <value>/tmp/.hdfs-nfs</value>
  </property>

  <property>
    <name>dfs.nfs.exports.allowed.hosts</name>
    <value>* rw</value>
  </property>

  <property>
    <name>nfs.export.point</name>
    <value>/user</value>
  </property>
  -->
  <!--
  <property>
    <name>dfs.namenode.heartbeat.recheck-interval</name>
    <value>100000</value>
  </property>
  -->

  <!--
  <property>
      <name>net.topology.script.file.name</name>
      <value>/opt/bin/rack-awareness.sh</value>
  </property>
  -->

  <!--
  <property>
    <name>dfs.hosts</name>
    <value>/opt/hadoop-3.3.3/etc/hadoop/hdfs.allow</value>
  </property>
  <property>
    <name>dfs.hosts.exclude</name>
    <value>/opt/hadoop-3.3.3/etc/hadoop/hdfs.decommission</value>
  </property>
  -->

</configuration>

:::

:::spoiler 修改 mapred-site.xml

修改命令 :

$ nano -l ~/vmusdt/hdp33/conf/hadoop-3.3.3/mapred-site.xml

第 11 行內容修改為以下 :

<value>/opt/hadoop-3.3.3/share/hadoop/mapreduce/*,/opt/hadoop-3.3.3/share/hadoop/mapreduce/lib/*,/opt/hadoop-3.3.3/share/hadoop/common/*,/opt/hadoop-3.3.3/share/hadoop/common/lib/*,/opt/hadoop-3.3.3/share/hadoop/yarn/*,/opt/hadoop-3.3.3/share/hadoop/yarn/lib/*,/opt/hadoop-3.3.3/share/hadoop/hdfs/*,/opt/hadoop-3.3.3/share/hadoop/hdfs/lib/*</value>

修改結果 :

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
   <property>
     <name>mapreduce.framework.name</name>
     <value>yarn</value>
   </property>
   <property>
      <name>mapreduce.application.classpath</name>
      <value>/opt/hadoop-3.3.3/share/hadoop/mapreduce/*,/opt/hadoop-3.3.3/share/hadoop/mapreduce/lib/*,/opt/hadoop-3.3.3/share/hadoop/common/*,/opt/hadoop-3.3.3/share/hadoop/common/lib/*,/opt/hadoop-3.3.3/share/hadoop/yarn/*,/opt/hadoop-3.3.3/share/hadoop/yarn/lib/*,/opt/hadoop-3.3.3/share/hadoop/hdfs/*,/opt/hadoop-3.3.3/share/hadoop/hdfs/lib/*</value>
   </property>

   <property>
      <name>yarn.app.mapreduce.am.resource.mb</name>
      <value>384</value>
   </property>
   <property>
      <name>yarn.app.mapreduce.am.command-opts</name>
      <value>-Xmx256m</value>
   </property>

   <property>
      <name>mapreduce.reduce.memory.mb</name>
      <value>512</value>
   </property>
   <property>
      <name>mapreduce.reduce.java.opts</name>
      <value>-Xmx384m</value>
   </property>
   <property>
      <name>mapreduce.map.memory.mb</name>
      <value>384</value>
   </property>
   <property>
      <name>mapreduce.map.java.opts</name>
      <value>-Xmx256m</value>
   </property>

   <!-- Disable Speculative Execution -->
   <!--
   <property>
      <name>mapred.map.tasks.speculative.execution</name>
      <value>false</value>
   </property>
   <property>
      <name>mapred.reduce.tasks.speculative.execution</name>
      <value>false</value>
   </property>
   -->
   <!--
   <property>
     <name>mapreduce.input.fileinputformat.split.minsize</name>
     <value>2048</value>
   </property>

   <property>
     <name>mapreduce.input.fileinputformat.split.maxsize</name>
     <value>2048</value>
   </property>
   -->
</configuration>

:::

:::spoiler 修改 yarn-site.xml

修改命令 :

$ nano -l ~/vmusdt/hdp33/conf/hadoop-3.3.3/yarn-site.xml

修改內容 :

第 23 行，更改為 <value>/home/bigboss/yarn</value>
第 75 行，更改為 <value>/opt/hadoop-3.3.3/etc/hadoop/yarn.allow</value>
第 80 行，更改為 <value>/opt/hadoop-3.3.3/etc/hadoop/yarn.decommission</value>
第 90 行，更改為 <value>bigboss yarnadmin</value>

修改結果 :

<?xml version="1.0"?>
<configuration>

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>dtm2:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>dtm2</value>
  </property>
  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/home/bigboss/yarn</value>
  </property>

  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>dtm2:10020</value>
  </property>

  <property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
  </property>

  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
  </property>
  <property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>8</value>
  </property>

  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>384</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>896</value>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-vcores</name>
    <value>1</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-vcores</name>
    <value>1</value>
  </property>

  <!--
  <property>
    <name>yarn.scheduler.increment-allocation-mb</name>
    <value>384</value>
  </property>
  -->
  <!--
  <property>
    <name>yarn.resourcemanager.nodes.include-path</name>
    <value>/opt/hadoop-3.3.3/etc/hadoop/yarn.allow</value>
  </property>

  <property>
    <name>yarn.resourcemanager.nodes.exclude-path</name>
    <value>/opt/hadoop-3.3.3/etc/hadoop/yarn.decommission</value>
  </property>
  -->
  <!--
  <property>
    <name>yarn.acl.enable</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.admin.acl</name>
    <value>bigboss yarnadmin</value>
  </property>
  -->

  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
  </property>

</configuration>

:::

修改 hive 設定檔 (~/vmusdt/hdp33/conf/apache-hive-3.1.3-bin/)

:::spoiler 修改 hive-site.xml.template

修改命令 :

$ nano -l hdp33/conf/apache-hive-3.1.3-bin/hive-site.xml.template

修改內容 :

第 21 行，更改為 <value>bigboss</value>

修改結果 :

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
 <property>
   <name>javax.jdo.option.ConnectionURL</name>
   <value>jdbc:mariadb://mdb:3306/metastore</value>
 </property>
 <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>org.mariadb.jdbc.Driver</value>
 </property>
 <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hiveuser</value>
 </property>
 <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hivepassword</value>
 </property>
 <property>
    <name>hive.users.in.admin.role</name>
    <value>bigboss</value>
 </property>

 <property>
    <name>hive.server2.enable.doAs</name>
    <value>true</value>
 </property>

 <property>
    <name>hive.security.authorization.enabled</name>
    <value>false</value>
 </property>
</configuration>

:::

修改 資科平台 系統程式 (~/vmusdt/hdp33/bin/)

:::spoiler 修改 dtest

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/dtest

修改內容 :

第 21 行，更改為 
hadoop jar /opt/hadoop-3.3.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.3.jar pi 2 1000 &> /tmp/out

修改結果 :

#!/bin/bash
echo "[HDFS]"
nc -w 1 -z dtm1 22 &>/dev/null
[ "$?" != 0 ] && echo 'dtm1 exited' && exit 1
ssh -q dtm1 jps | grep 'NameNode' &>/dev/null
[ "$?" != 0 ] && echo "HDFS exited" && exit 1
hdfs dfsadmin -report &> /tmp/out
cat /tmp/out | grep 'Live datanodes'
cat /tmp/out | grep 'Name: '
echo ""

echo "[YARN]"
nc -w 1 -z dtm2 22 &>/dev/null
[ "$?" != 0 ] && echo 'dtm2 exited' && exit 1
ssh dtm2 jps | grep 'ResourceManager' &>/dev/null
[ "$?" != 0 ] && echo "YARN exited" && exit 1
yarn node -list -all 2>/dev/null

echo""
echo [MapReduce]
hadoop jar /opt/hadoop-3.3.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.3.jar pi 2 1000 &> /tmp/out
cat /tmp/out | grep ' Pi '

echo""
hdfs dfs -ls /tmp/spark-events &>/dev/null
[ "$?" != "0" ] && hdfs dfs -mkdir /tmp/spark-events && hdfs dfs -chmod -R 777 /tmp
echo [Spark]
spark-submit --num-executors 1 --executor-memory 512m --master yarn \
/opt/spark-3.2.1-bin-hadoop3.2/examples/src/main/python/pi.py 10 2>/dev/null

echo ""
echo [HBase]
nc -w 1 -z dtm1 22 &>/dev/null
[ "$?" != 0 ] && echo 'dtm1 exited' && exit 1
ssh -q dtm1 jps | grep 'HMaster' &>/dev/null
if [ "$?" == "0" ]; then
   echo "status" | hbase shell -n 2>/dev/null | head -n 1
fi

:::

:::spoiler 修改 hls

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/hls

修改內容 :

第 14 行，更改為 
j=$(ssh -q bigboss@$n sudo jps 2>/dev/null | grep -v Jps)

修改結果 :

#!/bin/bash

nn="dtm1"
rs="dtm2"
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

nc -w 1 -z $nn 8020 &>/dev/null
[ "$?" != 0 ] && echo "pls start HDFS first or fomathdfs" && exit 1

for n in $nn $rs $w
do
   nc -w 1 -z $n 22 &>/dev/null
   if [ "$?" == "0" ]; then
      j=$(ssh -q bigboss@$n sudo jps 2>/dev/null | grep -v Jps)
      if [ "$j" != "" ]; then
         echo -e "[$n]\n$j"
         echo ""
      fi
   fi
done

:::

:::spoiler 修改 lake.schema

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/lake.schema

修改內容 :

第 2 行，更改為
owner=bigboss

第 5 行，更改為
ssh bigboss@dtm1 jps 2>/dev/null | grep NameNode &>/dev/null

修改結果 :

#!/bin/bash
owner=bigboss
group=bigboss

ssh bigboss@dtm1 jps 2>/dev/null | grep NameNode &>/dev/null
[ "$?" != "0" ] && echo "start HDFS first" && exit 1

lakeschema="/raw/ /elt/ /dataset/ /apps/ /metadata/"

for dn in $lakeschema
do
   hdfs dfs -ls $dn &>/dev/null
   if [ "$?" != "0" ]; then
       hdfs dfs -mkdir $dn
       hdfs dfs -chown $owner:$group $dn
       hdfs dfs -chmod 755 $dn
       echo "$dn ($owner:$group,755)"
   else
       echo "$dn existed"
   fi
done

hdfs dfs -ls /tmp/hadoop-yarn &>/dev/null
[ "$?" != "0" ] && hdfs dfs -mkdir -p /tmp/hadoop-yarn/staging && echo "/tmp/hadoop-yarn/staging created (YARN)"
hdfs dfs -ls /tmp/hive &>/dev/null
[ "$?" != "0" ] && hdfs dfs -mkdir /tmp/hive/ && echo "/tmp/hive created (Jupyter)"
hdfs dfs -ls /tmp/spark-events &>/dev/null
[ "$?" != "0" ] && hdfs dfs -mkdir /tmp/spark-events/ && echo "/tmp/spark-events created (Spark)"

hdfs dfs -chmod -R 777 /tmp/ && echo "/tmp (777)"

echo ""
echo "執行完成"
exit 0

:::

:::spoiler 修改 lake.users

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/lake.users

修改內容 :

第 3 行，更改為
ssh bigboss@dtm1 jps 2>/dev/null | grep NameNode  &>/dev/null

第 9 行，更改為
for us in rbean gbean ybean zbean zp01 zp02 zp03 bigboss

修改結果 :

#!/bin/bash

ssh bigboss@dtm1 jps 2>/dev/null | grep NameNode  &>/dev/null
[ "$?" != "0" ] && echo "start HDFS first" && exit 1

hdfs dfs -mkdir /user/ &>/dev/null
[ "$?" == "0" ] && echo -e "[HDFS]\n/user created"

for us in rbean gbean ybean zbean zp01 zp02 zp03 bigboss
do
    hdfs dfs -mkdir /user/$us/ &>/dev/null
    if [ "$?" == "0" ]; then
       hdfs dfs -chmod 755 /user/$us/
       hdfs dfs -chown $us:$us /user/$us/
       echo "/user/$us created"
    else
       echo "/user/$us existed"
    fi
done

i=101
while [ $i -le 200 ]
do
    cat /etc/hosts | grep "ds$2" &>/dev/null
    [ "$?" != 0 ] && i=$(($i+1)) && continue
    nc -w 1 -z "ds$i" 22
    [ "$?" != "0" ] && break
    hdfs dfs -mkdir /user/ds$i/ &>/dev/null
    if [ "$?" == "0" ]; then
       hdfs dfs -chmod 755 /user/ds$i/
       hdfs dfs -chown ds$i:ds$i /user/ds$i/
       echo "/user/ds$i created"
    else
       echo "/user/ds$i existed"
    fi
    i=$(($i+1))
done

echo ""
echo "執行完成"
exit 0

:::

:::spoiler 修改 starthba

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/starthba

修改內容 :

第 9 行，更改為
ssh bigboss@dtm2 zkServer.sh start &>/dev/null

新增以下這行
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

第 17 行，更改為
for n in $w

第 21 行，更改為
ssh bigboss@$n hbase-daemon.sh start regionserver &>/dev/null

修改結果 :

#!/bin/bash

#n=$(hostname)
#[ ${n:0:3} != "adm" ] && echo "oops, pls return to admxxx" && exit 1

w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

nc -w 5 -z dtm1 8020 &>/dev/null
[ "$?" != 0 ] && echo "pls start HDFS first" && exit 1

ssh bigboss@dtm2 zkServer.sh start &>/dev/null
[ "$?" == "0" ] && echo "dtm3: ZooKeeper started"

ssh bigboss@dtm1 hbase-daemon.sh start master &>/dev/null
[ "$?" == "0" ] && echo "dtm3: Hbase Master started"

for n in $w
do
   nc -w 3 -z $n 22 &>/dev/null
   if [ "$?" == "0" ]; then
      ssh bigboss@$n hbase-daemon.sh start regionserver &>/dev/null
      [ "$?" == "0" ] && echo "$n: Region Server started"
   fi
done

:::

:::spoiler 修改 starthdfs

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/starthdfs

修改內容 :

第 6 行，更改為 ssh bigboss@"$m" hadoop-daemon.sh start namenode &>/dev/null
第 11 行，更改為 ssh bigboss@"$m" hadoop-daemon.sh start secondarynamenode &>/dev/null
第 18 行，更改為 ssh bigboss@"$n" hadoop-daemon.sh start datanode &>/dev/null

修改結果 :

#!/bin/bash

m="dtm1"
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

ssh bigboss@"$m" hadoop-daemon.sh start namenode &>/dev/null
sleep 10; nc -w 5 -z $m 8020 &>/dev/null
[ "$?" != 0 ] && echo "pls formathdfs first" && exit 1
echo "$m: Name Node Started"

ssh bigboss@"$m" hadoop-daemon.sh start secondarynamenode &>/dev/null
[ "$?" == "0" ] && echo "$m: Secondary Name Node started"

for n in $w
do
   nc -w 5 -z $n 22 &>/dev/null
   if [ "$?" == "0" ]; then
      ssh bigboss@"$n" hadoop-daemon.sh start datanode &>/dev/null
      [ "$?" == "0" ] && echo "$n: Data Node started"
   fi
done

:::

:::spoiler 修改 startjps

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/startjps

修改內容 :

新增以下這行
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

第 5 行，更改為
for n $w

修改結果 :

#!/bin/bash

w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

for n in $w
do
  ssh $n 'nohup jpslog.sh &>/dev/null &'
done

:::

:::spoiler 修改 startmaxctn

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/startmaxctn

修改內容 :

新增以下這行
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

第 5 行，更改為
for n in $w

修改結果 :

#!/bin/bash

w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

for n in $w
do
  ssh $n 'nohup /opt/bin/maxjps.sh &>/dev/null &'
done

:::

:::spoiler 修改 startyarn

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/startyarn

修改內容 :

第 5 行，更改為
yn=5

第 15 行，更改為
ssh bigboss@"$rs" yarn-daemon.sh start resourcemanager &>/dev/null

第 17 行，更改為
ssh bigboss@"$rs" mr-jobhistory-daemon.sh start historyserver &>/dev/null

第 24 行，更改為
ssh bigboss@$n yarn-daemon.sh start nodemanager &>/dev/null

修改結果 :

#!/bin/bash

nn="dtm1"
rs="dtm2"
yn=5
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)


#n=$(hostname)
#[ ${n:0:3} != "adm" ] && echo "oops, pls return to admxxx" && exit 1

nc -w 5 -z $nn 8020 &>/dev/null
[ "$?" != 0 ] && echo "pls start HDFS first" && exit 1

ssh bigboss@"$rs" yarn-daemon.sh start resourcemanager &>/dev/null
[ "$?" == "0" ] && echo "$rs: Resource Manager started"
ssh bigboss@"$rs" mr-jobhistory-daemon.sh start historyserver &>/dev/null
[ "$?" == "0" ] && echo "$rs: Job History Server started"

## 多 if 判斷是因為，剩的 VM 可以給 Hbase 的程式跑
c=0
for n in $w
do
   if [ "$c" -le "$yn" ]; then
      ssh bigboss@$n yarn-daemon.sh start nodemanager &>/dev/null
      [ "$?" == "0" ] && echo "$n: Node Manager started"
      c=$(( c+1 ))
   fi
done

:::

:::spoiler 修改 stophba

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/stophba

修改內容 :

新增以下這行
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

第 12 行，更改為
ssh bigboss@$n hbase-daemon.sh stop regionserver &>/dev/null

第 17 行，更改為
ssh bigboss@dtm2 hbase-daemon.sh stop master &>/dev/null

第 20 行，更改為
ssh bigboss@dtm1 zkServer.sh stop &>/dev/null

修改結果 :

#!/bin/bash

#n=$(hostname)
#[ ${n:0:3} != "adm" ] && echo "oops, pls return to admxxx" && exit 1

w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

for n in $w
do
   nc -w 1 -z $n 22 &>/dev/null
   if [ "$?" == "0" ]; then
      ssh bigboss@$n hbase-daemon.sh stop regionserver &>/dev/null
      [ "$?" == "0" ] && echo "$n: Region Server stoped"
   fi
done

ssh bigboss@dtm2 hbase-daemon.sh stop master &>/dev/null
[ "$?" == "0" ] && echo "dtm1: Hbase Master stoped"

ssh bigboss@dtm1 zkServer.sh stop &>/dev/null
[ "$?" == "0" ] && echo "dtm2: ZooKeeper stoped"

:::

:::spoiler 修改 stophdfs

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/stophdfs

修改內容 :

第 13 行，更改為
ssh bigboss@$n hadoop-daemon.sh stop datanode &>/dev/null

第 18 行，更改為
ssh bigboss@"$m" hadoop-daemon.sh stop secondarynamenode &>/dev/null

第 20 行，更改為
ssh bigboss@"$m" hadoop-daemon.sh stop namenode &>/dev/null

修改結果 :

#!/bin/bash

m="dtm1"
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

#n=$(hostname)
#[ ${n:0:3} != "adm" ] && echo "oops, pls return to admxxx" && exit 1

for n in $w
do
   nc -w 1 -z $n 22 &>/dev/null
   if [ "$?" == "0" ]; then
      ssh bigboss@$n hadoop-daemon.sh stop datanode &>/dev/null
      [ "$?" == "0" ] && echo "$n: Data Node stoped"
   fi
done

ssh bigboss@"$m" hadoop-daemon.sh stop secondarynamenode &>/dev/null
[ "$?" == "0" ] && echo "$m: Secondary Name Node stoped"
ssh bigboss@"$m" hadoop-daemon.sh stop namenode &>/dev/null
[ "$?" == "0" ] && echo "$m: Name Node stoped"

:::

:::spoiler 修改 stopjps

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/stopjps

修改內容 :

新增以下這行
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

第 5 行，更改為
for n in $w

修改結果 :

#!/bin/bash

w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

for n in $w
do
  ssh $n 'touch /tmp/jlog.stop'
  ssh $n "cat /tmp/jps-$n.log"
  echo ""
done

:::

:::spoiler 修改 stopmaxctn

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/stopmaxctn

修改內容 :

新增以下這行
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

第 5 行，更改為
for n in $w

修改結果 :

#!/bin/bash

w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

for n in $w
do
  ssh $n 'touch /tmp/maxjlog.stop'
  ssh $n "cat /tmp/jps-$n.log | uniq"
  echo ""
done

:::

:::spoiler 修改 stopyarn

修改命令 :

$ nano -l ~/vmusdt/hdp33/bin/stopyarn

修改內容 :

第 5 行，更改為
yn=5

第 15 行，更改為
ssh bigboss@$n yarn-daemon.sh stop nodemanager &>/dev/null

第 21 行，更改為
ssh bigboss@"$rs" mr-jobhistory-daemon.sh stop historyserver &>/dev/null

第 23 行，更改為
ssh bigboss@"$rs" yarn-daemon.sh stop resourcemanager &>/dev/null

修改結果 :

#!/bin/bash

nn="dtm1"
rs="dtm2"
yn=5
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

#n=$(hostname)
#[ ${n:0:3} != "adm" ] && echo "oops, pls return to admxxx" && exit 1

c=0
for n in $w
do
   if [ "$c" -le "$yn" ]; then
      ssh bigboss@$n yarn-daemon.sh stop nodemanager &>/dev/null
      [ "$?" == "0" ] && echo "$n: Node Manager stoped"
      c=$(( c+1 ))
   fi
done

ssh bigboss@"$rs" mr-jobhistory-daemon.sh stop historyserver &>/dev/null
[ "$?" == "0" ] && echo "$rs: Job History Server stoped"
ssh bigboss@"$rs" yarn-daemon.sh stop resourcemanager &>/dev/null
[ "$?" == "0" ] && echo "$rs: Resource Manager stoped"

:::

下載資科平台套件 dtwget

dtwget 程式功能 :

下載 DT 套件檔至 ~/vmausdt/hdp33/opt 這目錄

下載命令 :

$ dtwget 33

命令結果 :

download DT packages ? (YES/NO) YES
hadoop-3.3.3.tar.gz ok
apache-hive-3.1.3-bin.tar.gz ok
pig-0.17.0.tar.gz ok
spark-3.2.1-bin-hadoop3.2.tgz ok
hbase-2.4.13-bin.tar.gz ok
apache-zookeeper-3.6.3-bin.tar.gz ok

複製 資科平台 套件 dtcopy

程式功能 :

主要是複製 dta1 主機 /home/bigboss/vmusdt/hdp33/opt 這目錄至 dts1, dts2, dts3, dtm1, dtm2, dtw1, dtw2, dtw3, dtw4, dtw5, dtw6 這十一部主機的 /opt 目錄

命令 :

$ ./bin/dtcopy 33

:::spoiler 命令結果

dta1 /opt/ ok
dta1 /opt/bin ok
dta1 /etc/hosts ok

dts1 /opt/ ok
dts1 /opt/bin ok
dts1 /etc/hosts ok

dts2 /opt/ ok
dts2 /opt/bin ok
dts2 /etc/hosts ok

dts3 /opt/ ok
dts3 /opt/bin ok
dts3 /etc/hosts ok

dtm1 /opt/ ok
dtm1 /opt/bin ok
dtm1 /etc/hosts ok

dtm2 /opt/ ok
dtm2 /opt/bin ok
dtm2 /etc/hosts ok

dtw1 /opt/ ok
dtw1 /opt/bin ok
dtw1 /etc/hosts ok

dtw2 /opt/ ok
dtw2 /opt/bin ok
dtw2 /etc/hosts ok

dtw3 /opt/ ok
dtw3 /opt/bin ok
dtw3 /etc/hosts ok

dtw4 /opt/ ok
dtw4 /opt/bin ok
dtw4 /etc/hosts ok

dtw5 /opt/ ok
dtw5 /opt/bin ok
dtw5 /etc/hosts ok

dtw6 /opt/ ok
dtw6 /opt/bin ok
dtw6 /etc/hosts ok

:::

複製 資科平台 設定檔 dtconf

命令 :

$ dtconf 33

將每台電腦重新開機 ( a1 最後 )

$ ssh dts1 sudo reboot
$ ssh dts2 sudo reboot
$ ssh dts3 sudo reboot
$ ssh dtm1 sudo reboot
$ ssh dtm2 sudo reboot
$ ssh dtw1 sudo reboot
$ ssh dtw2 sudo reboot
$ ssh dtw3 sudo reboot
$ ssh dtw4 sudo reboot
$ ssh dtw5 sudo reboot
$ ssh dtw6 sudo reboot
$ ssh dta1 sudo reboot

:::spoiler 小確幸程式

程式功能 :

  1. 將所有 dt VM 重新開機
  2. 將所有 dt VM 關機
  3. 確認 dt 內網狀況
#!/bin/bash

# var
an=$(cat /etc/hosts | tr -s ' ' | grep -E ' dta[0-9]' | cut -d " " -f 2)
sn=$(cat /etc/hosts | tr -s ' ' | grep -E ' dts[0-9]' | cut -d " " -f 2)
mn=$(cat /etc/hosts | tr -s ' ' | grep -E ' dtm[0-9]' | cut -d " " -f 2)
wn=$(cat /etc/hosts | tr -s ' ' | grep -E ' dtw[0-9]' | cut -d " " -f 2)

usage() {
  cat<<EOF
Usage:
  $(basename "${BASH_SOURCE[0]}") [options]

Available options:
net      check ssh connection
reboot   reboot Hadoop Cluster
stop     poweroff Hadoop Cluster
EOF
  exit
}

case $1 in
net)
  for n in $an $sn $mn $wn
  do
    if ! nc -w 1 -z "$n" 22 &>/dev/null; then
      echo "${n} ssh Error !" && contiue
    else
      echo "${n} ssh OK !"
    fi
  done
  ;;
reboot)
  for r in $wn $mn $sn $an
  do
    ssh "$r" 'sudo reboot'
  done
  ;;
stop)
  for p in $wn $mn $sn $an
  do
    ssh "$p" 'sudo poweroff'
  done
  ;;
*)
  usage
  ;;
esac

:::

檢測 資料科技 平台

dthdpchk 檢查

命令 :

$ dthdpchk

:::spoiler 命令結果

[dta1]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dts1]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dts2]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dts3]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtm1]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtm2]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtw1]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtw2]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtw3]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtw4]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtw5]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtw6]
Hadoop 3.3.3
Native library checking:
hadoop:  true /opt/hadoop-3.3.3/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

:::

格式化 HDFS 分散檔案系統

命令 :

$ formathdfs

執行結果 :

Are you sure ? (YES/NO) YES
formathdfs ok

檢查 formathdfs

bigboss@dta1:~/vmusdt$ ssh dtm1 tree nn
nn
└── current
    ├── fsimage_0000000000000000000
    ├── fsimage_0000000000000000000.md5
    ├── seen_txid
    └── VERSION

1 directory, 4 files

$ ssh dtm1 tree dn
dn [error opening dir]

0 directories, 0 files

啟動 資科平台

啟動 HDFS 分散式檔案系統

命令 :

$ starthdfs

命令結果 :

dtm1: Name Node Started
dtm1: Secondary Name Node started
dtw1: Data Node started
dtw2: Data Node started
dtw3: Data Node started
dtw4: Data Node started
dtw5: Data Node started
dtw6: Data Node started

執行 hls 檢查 是否符合預期

$ hls
[dtm1]
1155 SecondaryNameNode
996 NameNode

[dtw1]
1191 DataNode

[dtw2]
1193 DataNode

[dtw3]
1192 DataNode

[dtw4]
1186 DataNode

[dtw5]
1194 DataNode

[dtw6]
1192 DataNode
$ ssh dtw1 tree dn
dn
├── current
│   ├── BP-1285703481-192.168.66.114-1657012142491
│   │   ├── current
│   │   │   ├── dfsUsed
│   │   │   ├── finalized
│   │   │   ├── rbw
│   │   │   └── VERSION
│   │   ├── scanner.cursor
│   │   └── tmp
│   └── VERSION
└── in_use.lock

6 directories, 5 files

啟動 YARN 分散式運算系統

命令 :

$ startyarn

命令結果 :

dtm2: Resource Manager started
dtm2: Job History Server started
dtw1: Node Manager started
dtw2: Node Manager started
dtw3: Node Manager started
dtw4: Node Manager started
dtw5: Node Manager started
dtw6: Node Manager started

執行 hls ，檢查 是否符合預期

$ hls
[dtm1]
1157 SecondaryNameNode
1004 NameNode

[dtm2]
1077 ResourceManager
1199 JobHistoryServer

[dtw1]
1461 NodeManager
1191 DataNode

[dtw2]
1194 DataNode
1404 NodeManager

[dtw3]
1191 DataNode
1402 NodeManager

[dtw4]
1195 DataNode
1405 NodeManager

[dtw5]
1194 DataNode
1406 NodeManager

[dtw6]
1193 DataNode
1404 NodeManager

測試 資科平台

命令 :

$ dtest

命令結果 :

[HDFS]
Live datanodes (6):
Name: 192.168.66.120:9866 (dtw1)
Name: 192.168.66.121:9866 (dtw2)
Name: 192.168.66.122:9866 (dtw3)
Name: 192.168.66.123:9866 (dtw4)
Name: 192.168.66.124:9866 (dtw5)
Name: 192.168.66.125:9866 (dtw6)

[YARN]
Total Nodes:6
         Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
      dtw6:42613                RUNNING         dtw6:8042                                  0
      dtw1:41169                RUNNING         dtw1:8042                                  0
      dtw3:36145                RUNNING         dtw3:8042                                  0
      dtw4:41433                RUNNING         dtw4:8042                                  0
      dtw5:40741                RUNNING         dtw5:8042                                  0
      dtw2:38177                RUNNING         dtw2:8042                                  0

[MapReduce]
Estimated value of Pi is 3.14400000000000000000

[Spark]
Pi is roughly 3.140400

[HBase]
tags: 資料科技平台
建置靜宜大學落地雲系統Linux Data Process