antony@notes:~/data-platform$ cat "Hadoop-程式解析.md"

Hadoop 程式解析

2022-05-15· data-platform ·資料科技平台

Hadoop 程式解析

確認組內dt系列的虛擬機器是否有在線上

#!/bin/bash
export  tt=003
for i in dta1 dta2 dtm1 dtm2 dtw1 dtw2 dtw3 dtw4
do
  nc -w 1 -z ${i} 22
  [ $? = 0 ] && echo "$i online" || echo "$i not online"
  ssh ${i} echo $tt
done

更改ssh client的設定檔，在ssh連到別人時，如果一開始第一次不想回答yes，才下載公鑰，就做下面的更改

sudo nano /etc/ssh/ssh_config
...
StrictHostKeyChecking no ##把ask刪掉改成no
...

允許OpenSSH Server可以讀取Environment檔

bigred@dtw4:~$ sudo nano /etc/ssh/sshd_config
PermitUserEnvironment yes  ##把註解拿掉，並將no改成yes

env

PATH環境變數，透過ssh查看，與在本機查看不同 ==用ssh登入系統時，一定會做身份認證，所以代表有一個程式一定會被執行，這支程式就是/etc/profile==

bigred@dta2:~$ env
...
PATH=/home/bigred/bin:/home/bigred/bin:/home/bigred/vmusdt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
...

bigred@dta2:~$ env | wc -l
27

bigred@dtw4:~$ ssh dta2 env
...
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
...

bigred@dtw4:~$ ssh dta2 env | wc -l
17

如果用ssh指令登入dta2這台機器，進到/etc/profile裡面修改內容

bigred@dta2:~$ sudo nano /etc/profile
export moon="go or no go"

bigred@dta2:~$ exit

登出以後，再用ssh指令登入dta2這台機器，就可以看到剛剛宣告全域的環境變數

bigred@dta2:~$ env | grep moon
moon=go or no go

==下面這行指令是，OpenSSH Server在執行，與上面直接用ssh指令登入不同，上面是bash貝殼程式在執行，所以執行env完的結果會不一樣==

bigred@dtw4:~$ ssh dta2 env
...
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
...

可以藉由以下命令得知，OpenSSH Server讀到的env是在/etc/environment這個檔案裡面

bigred@dtw4:~$ ssh dta2 cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

ssh dta2更改/etc/environment的內容

bigred@dta2:~$ sudo nano /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
moon="go or no go"  ## 新增這行

bigred@dta2:~$ exit
logout
Connection to dta2 closed.

可以看到moon這個環境變數有成功新增

bigred@dtw4:~$ ssh dta2 env
SHELL=/bin/bash
moon=go or no go ##剛剛在/etc/environmentvpu 新增的內容
...

備份到GCP

scp dt33pkg.zip sreantony@34.81.39.139:/home/sreantony/dt

us.bash內容

bigred@dtw4:~$ cat ~/vmusdt/hdp33/us.bash
#!/bin/bash
adminuser=bigred
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/     ## 把Java的JVM放在這個目錄
export HADOOP_HOME=/opt/hadoop-3.3.2  ## hadoop-3.3.2套件裡面所有的內容
export HADOOP_LOG_DIR=/tmp            ## log檔存放區
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/   ## hadoop 設定檔
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/   ## native，像是bzip2這個演算法的程式放在裡面
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
# export HADOOP_USER_CLASSPATH_FIRST=true
[ -z $HADOOP_USER_NAME ] && [ $SHELL == '/bin/bash' ] && declare -r HADOOP_USER_NAME=$USER

# JobHistory log file path, 會自動產生目錄
# export HADOOP_MAPRED_LOG_DIR="/home/bigred/jhslog"

export YARN_HOME=$HADOOP_HOME  
export HADOOP_YARN_HOME=$HADOOP_HOME
export YARN_LOG_DIR=/tmp

# hadoop 3.2.1 及 3.1.3 這二個版本只要執行檔案傳送命令就會出現以下訊息
# hadoop fs -put -f /etc/passwd /tmp
# SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
export HADOOP_ROOT_LOGGER="WARN,console"

export PIG_HOME=/opt/pig-0.17.0
export PIG_HEAPSIZE=512     ## Heap 是JVM裡面的區塊，是Java程式用來處理記憶體的一個空間，也就代表pig處理資料的記憶體大小
export HIVE_HOME=/opt/apache-hive-3.1.3-bin

export PATH=/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH=/home/bigred/vmusdt/bin:$PATH ## 前面這個目錄裡面放的是老師開發的程式，像是dtconf、dtcopy、dtwget...等
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PIG_HOME/bin:$HIVE_HOME/bin   ##這行代表我們可以在任何目錄執行Hadoop、Pig和Hive的指令

export HBASE_HOME=/opt/hbase-2.4.12
export HBASE_CONF_DIR=/opt/hbase-2.4.12/conf
export ZOOKEEPER_HOME=/opt/apache-zookeeper-3.6.3-bin
export ZOO_LOG_DIR=/tmp/logs    ##/tmp目錄裡面的檔案，只要系統重新開機，裡面的內容就會清空，這邊把紀錄檔放在/tmp是因為硬體設備不足
export PATH=$PATH:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin

#export ZEPPELIN_HOME=/opt/zeppelin-0.9.0-bin-all
#export PATH=$PATH:$ZEPPELIN_HOME/bin

export SPARK_HOME=/opt/spark-3.2.1-bin-hadoop3.2
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PYSPARK_PYTHON=/usr/bin/python3
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

if [ -d /opt/jupyter/miniconda3/bin ];then
   export ANACONDA_ROOT=/opt/jupyter/miniconda3
   export PATH=$ANACONDA_ROOT/bin:$PATH
   export PYSPARK_PYTHON=$ANACONDA_ROOT/bin/python
fi

if [ "$USER" != "" ]; then

   [ -f /home/$USER/dkc.env ] && source /home/$USER/dkc.env

   if [ ! -d metastore_db ]; then       ## 設定derby資料庫系統所需的設定
      #hn=$(hostname)
      #if [ ${hn:0:3} == "adm" ] || [ ${hn:0:2} == "ds" ] || [ ${hn:0:8} == "zeppelin" ]; then
      echo -n "build derby database ..."
      schematool -initSchema -dbType derby &>/dev/null
      if [ "$?" == "0" ]; then
         echo " ok"
         echo "set hive.cli.print.current.db=true;" > .hiverc
         echo "set hive.metastore.warehouse.dir=/user/$USER/hive;" >> .hiverc
         echo "set hive.exec.scratchdir=/user/$USER/tmp;" >> .hiverc
      fi
   fi

   alias nano='nano -Ynone'    ## 在編輯檔案時，不會有顏色
   alias dir='ls -alh'
   alias ssh='ssh -q'          ## -q 是ssh不會問我們問題
fi

bigred@dtw4:~/vmusdt$ cat ~/vmusdt/hdp33/environment
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/
HADOOP_HOME=/opt/hadoop-3.3.2
HADOOP_LOG_DIR=/tmp
LD_LIBRARY_PATH=/opt/hadoop-3.3.2/lib/native/
HADOOP_COMMON_LIB_NATIVE_DIR=/opt/hadoop-3.3.2/lib/native
HADOOP_OPTS=-Djava.library.path=/opt/hadoop-3.3.2/lib/native
HADOOP_MAPRED_LOG_DIR=/home/bigred/jhslog
YARN_HOME=/opt/hadoop-3.3.2
YARN_LOG_DIR=/tmp
PIG_HOME=/opt/pig-0.17.0/
PIG_HEAPSIZE=512
SPARK_HOME=/opt/spark-3.2.1-bin-hadoop3.2
HIVE_HOME=/opt/apache-hive-3.1.3-bin
HBASE_HOME=/opt/hbase-2.4.12
HBASE_CONF_DIR=/opt/hbase-2.4.11/conf
HBASE_LOG_DIR=/tmp
PATH=/root/bin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hadoop-3.3.2/bin:/opt/hadoop-3.3.2/sbin:/opt/pig-0.17.0/bin:/opt/apache-hive-3.1.3-bin/bin:/opt/spark-3.2.1-bin-hadoop3.2/bin:/opt/spark-3.2.1-bin-hadoop3.2/sbin:/opt/hbase-2.4.12/bin:/opt/apache-zookeeper-3.6.3-bin/bin
ZOO_LOG_DIR=/tmp/logs

裡面放的是設定檔，不是*.jar檔

bigred@dtw4:~/vmusdt$ dir ~/vmusdt/hdp33/conf/
total 28K
drwxr-xr-x 7 bigred bigred 4.0K May  9 16:15 .
drwxr-xr-x 5 bigred bigred 4.0K May 10 05:39 ..
drwxr-xr-x 2 bigred bigred 4.0K Feb  6 16:00 apache-hive-3.1.2-bin
drwxr-xr-x 2 bigred bigred 4.0K Feb  6 16:00 apache-zookeeper-3.6.3-bin
drwxr-xr-x 2 bigred bigred 4.0K May  9 18:36 hadoop-3.3.2
drwxr-xr-x 2 bigred bigred 4.0K Feb  6 16:00 hbase-2.4.12
drwxr-xr-x 2 bigred bigred 4.0K May  9 18:54 spark-3.2.1-bin-hadoop3.2

dtconf檔內容

bigred@dtw4:~/vmusdt$ cat bin/dtconf
#!/bin/bash

w="dta1 dta2 dtm1 dtm2 dtw1 dtw2 dtw3 dtw4"

[ "$#" != 1 ] && echo "dtconf version" && exit 1

[ ! -d ~/vmusdt/hdp${1}/opt ] && echo "~/vmusdt/hdp${1}/opt not found" && exit 1

n=$(ls ~/vmusdt/hdp${1}/opt)

## 偵測要處理的電腦是否在線上，不在則跳過
for x in $w
do
   nc -w 1 -z $x 22 &>/dev/null
   [ "$?" != "0" ] && continue       

## 拷貝Hadoop、Hbase和Hive...等的設定檔
   for y in $n              
   do
     echo "$y" | grep 'hadoop-' &>/dev/null
     [ "$?" == "0" ] && scp ~/vmusdt/hdp${1}/conf/$y/* bigred@$x:/opt/$y/etc/hadoop/ &>/dev/null && echo "$x $y config ok" && continue
     [ -d ~/vmusdt/hdp${1}/conf/$y ] && scp ~/vmusdt/hdp${1}/conf/$y/* bigred@$x:/opt/$y/conf/ &>/dev/null
     [ "$?" == "0" ] && echo "$x $y config ok"
   done

## 以下這三行指令是專門給Ubuntu Linux設定的，Alpine Linux就不用，因為它沒有/etc/environment
   ssh $x 'sudo rm /etc/environment &>/dev/null'  #刪除根目錄中的 environment ，openssh server才不會先讀到它
## 把dta1的bigred使用者家目錄下面的vmusdt目錄下面的hdp${1}目錄下面的environment這個檔案，傳到dt系列的虛擬機器的bigred使用者的家目錄下面的.ssh目錄下 
   scp -r ~/vmusdt/hdp${1}/environment bigred@$x:~/.ssh/environment &>/dev/null    
   [ "$?" == "0" ] && echo "$x environment ok"

## 把us.bash傳到每台dt系列機器的家目錄
   scp -r ~/vmusdt/hdp${1}/us.bash bigred@$x:~/us.bash &>/dev/null
   [ "$?" == "0" ] && echo "$x us.bash ok"

## 將source /home/bigred/us.bash 加到/etc/profile檔案的最後一行
   ssh $x 'cat /etc/profile | grep '/home/bigred/us.bash' &>/dev/null'
   [ "$?" != "0" ] && ssh $x 'echo source /home/bigred/us.bash | sudo tee -a /etc/profile &>/dev/null'
   echo "$x /etc/profile ok"

   echo ""
done

bigred@dtw4:~/vmusdt$ sudo nano hdp33/bin/formathdfs
#!/bin/bash

m="dtm1"
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

read -p "Are you sure ? (YES/NO) " ans
[ $ans != "YES" ] && echo "abort format HDFS" && exit 1

ssh "$m" rm -r nn/* sn/* &>/dev/null
[ "$?" == "0" ] && echo "$m rm ok"

for n in $w
do
   nc -w 1 -z $n 22 &>/dev/null
   [ "$?" == "0" ] && ssh $n rm -r dn/* &>/dev/null && echo "$n rm ok"
done

ssh $m 'hdfs namenode -format -clusterID cute' &>/dev/null
[ "$?" != "0" ] && echo "formathdfs failure" && exit 1
echo "formathdfs ok"

starthdfs

bigred@dtw4:~/vmusdt$ sudo nano hdp33/bin/starthdfs

m="dtm1"
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)

## 透過openssh server在dtm1直接執行啟動name node的命令
ssh bigred@"$m" hadoop-daemon.sh start namenode &>/dev/null
sleep 10; nc -w 5 -z $m 8020 &>/dev/null
[ "$?" != 0 ] && echo "pls formathdfs first" && exit 1
echo "$m: Name Node Started"

## 透過openssh server在dtm1直接執行啟動secondary namenode的命令
ssh bigred@"$m" hadoop-daemon.sh start secondarynamenode &>/dev/null
[ "$?" == "0" ] && echo "$m: Secondary Name Node started"

for n in $w
do
   nc -w 5 -z $n 22 &>/dev/null
   if [ "$?" == "0" ]; then
      ssh bigred@$n hadoop-daemon.sh start datanode &>/dev/null
      [ "$?" == "0" ] && echo "$n: Data Node started"
   fi
done

startyarn

bigred@dtw4:~/vmusdt$ sudo nano hdp33/bin/startyarn
#!/bin/bash

nn="dtm1"
rs="dtm2"
yn=4
w=$(cat /etc/hosts | grep -e ' dtw[0-9]' | cut -d' ' -f2)


#n=$(hostname)
#[ ${n:0:3} != "adm" ] && echo "oops, pls return to admxxx" && exit 1

nc -w 5 -z $nn 8020 &>/dev/null
[ "$?" != 0 ] && echo "pls start HDFS first" && exit 1

ssh bigred@"$rs" yarn-daemon.sh start resourcemanager &>/dev/null
[ "$?" == "0" ] && echo "$rs: Resource Manager started"
ssh bigred@"$rs" mr-jobhistory-daemon.sh start historyserver &>/dev/null
[ "$?" == "0" ] && echo "$rs: Job History Server started"
## 上面這行是11:33

c=0
for n in $w
do
   if [ "$c" -le "$yn" ]; then
      ssh bigred@$n yarn-daemon.sh start nodemanager &>/dev/null
      [ "$?" == "0" ] && echo "$n: Node Manager started"
      c=$(( c+1 ))
   fi
done

tags: `資料科技平台`

如何建立/開始/停止/刪除 6台資料科技的VM Datatechnology 資料科技 (舊版)

Hadoop 程式解析

Hadoop 程式解析

env

dtconf檔內容

starthdfs

startyarn

tags: 資料科技平台

tags: `資料科技平台`