antony@notes:~/data-platform$ cat "手動建制資科平台.md"

手動建制資科平台

2022-07-06· data-platform ·資料科技平台

手動建制資科平台

[TOC]

資科平台叢集建置

下載 US2004.CVN80.zip 虛擬主機壓縮檔, 然後解壓縮至使用者家目錄
- Open jdk-8 已安裝
- libisal-dev libsnappy-dev libz-dev
- 設定 SSH /etc/ssh/sshd_config -> PermitUserEnvironment yes /etc/ssh/ssh_config -> StrictHostKeyChecking no 設定好環境後，再複製其他機器，可降低人為的錯誤以及達成快速部署
啟動 US2004.CVN80 虛擬主機, 執行以下設定命令

C:\Users\bigred>ssh bigred@192.168.61.144
bigred@192.168.61.144s password:

## 系統更新
$ sudo apt update; sudo apt upgrade; sudo apt autoremove

## 安裝 Hadoop 所需的套件 (本次練習已先做好設定)
$ sudo apt install libisal-dev libsnappy-dev libz-dev

## 設定 SSH  (本次練習已先做好設定)
$ sudo nano /etc/ssh/sshd_config
.......
PermitUserEnvironment yes 

$ sudo nano /etc/ssh/ssh_config
.........
StrictHostKeyChecking no

## 產生公私鑰，並複製公鑰成 authorized_keys 
$ ssh-keygen -t rsa -P ''
Enter file in which to save the key (/home/bigred/.ssh/id_rsa): 按 Enter

$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

$ sudo nano /etc/hosts
127.0.0.1 localhost
127.0.1.1 cvn80
120.96.143.xx dta1
120.96.143.xx dtm1
120.96.143.xx dtm2
120.96.143.xx dtw1
120.96.143.xx dtw2
120.96.143.xx dtw3

## 關機
$ poweroff

在 Windows 命令提示字元視窗, 執行以下命令

C:\Users\bigred>mkdir dtvm
C:\Users\bigred>xcopy US2004.CVN80\US2004.vmwarevm dtvm\us.dta1\ /E /Y > nul
C:\Users\bigred>xcopy US2004.CVN80\US2004.vmwarevm dtvm\us.dtm1\ /E /Y > nul
C:\Users\bigred>xcopy US2004.CVN80\US2004.vmwarevm dtvm\us.dtm2\ /E /Y > nul
C:\Users\bigred>xcopy US2004.CVN80\US2004.vmwarevm dtvm\us.dtw1\ /E /Y > nul
C:\Users\bigred>xcopy US2004.CVN80\US2004.vmwarevm dtvm\us.dtw2\ /E /Y > nul
C:\Users\bigred>xcopy US2004.CVN80\US2004.vmwarevm dtvm\us.dtw3\ /E /Y > nul

xcopy ，複製檔案和目錄（包括子目錄）。 /E ，複製所有子目錄（即使它們是空的）。 /Y ，隱藏提示以確認您想要覆寫現有的目的地檔案。 > nul ， STDOUT 標準輸出重新導向至 NUL 裝置，錯誤訊息仍會列印到畫面。因為錯誤訊息通常會傳送至標準錯誤資料流程，而不是標準輸出資料流程。

啟動 VMware Workstation Player, 設定資科平台六部虛擬主機
- 設定虛擬主機名稱 (dta1,dtm1,dtm2,dtw1,dtw2,dtw3)
- 設定虛擬主機記憶體 dta1 -> 8G (8192) dtm1 -> 8G dtm2 -> 8G dtw1 -> 10G (10240) dtw2 -> 10G dtw3 -> 10G
- 設定虛擬主機網卡 Network Adapter -> Bridged
壓縮資科平台六部虛擬主機目錄, 使用 scp 命令, 複製到組員的落地雲主機
啟動資科平台虛擬主機, 然後設定網路系統

在 Windows 命令提示字元視窗, 執行以下命令, 登入虛擬主機

c:\>ssh bigred@120.96.143.xx

設定固定 IP 
$ sudo nano /etc/netplan/00-installer-config.yaml 
network:
  version: 2
  renderer: networkd
  ethernets:
    ens32:
      dhcp4: no
      addresses:
        - 120.96.143.xx/25
      gateway4: 120.96.143.254
      nameservers:
          addresses: [8.8.8.8, 1.1.1.1]

設定電腦名稱
$ sudo nano /etc/hostname
dta1

$ sudo reboot

每台虛擬機器上面跑的Java程式

dta1 hdfs,yarn,pig,hive
dtm1 name node, secondary name node
dtm2 resource manager, Job history server
dtw1 Datamode, node manager
dtw2 Datamode, node manager
dtw3 Datamode, node manager
dtw4 Datamode, node manager

資科平台系統建置

下載資料科技平台套件檔在 Windows 系統的 cmd 視窗, 執行以下命令

$ ssh bigred@<dta1 IP>

$ cd; wget http://www.oc99.org/zip/vmusdt2022.zip; unzip vmusdt2022.zip 

$ dir vmusdt      ## vmusdt (Virtual Machine Ubuntu Server Data technology)
total 341M
drwxr-sr-x 4 bigred bigred 4.0K Mar 21 20:44 .
drwxr-sr-x 9 bigred bigred 4.0K Mar 21 22:41 ..
drwxr-sr-x 2 bigred bigred 4.0K Mar 21 22:17 bin
-rw-r--r-- 1 bigred bigred 341M Mar 21 20:42 hdp210
drwxr-sr-x 4 bigred bigred 4.0K Mar 21 22:15 hdp33

bigred@dta1:~$ dir vmusdt/bin
total 28K
drwxr-xr-x 2 bigred bigred 4.0K May 11 06:57 .
drwxr-xr-x 5 bigred bigred 4.0K May 11 14:02 ..
-rwxr-xr-x 1 bigred bigred 1.2K May 10 07:40 dtconf
-rwxr-xr-x 1 bigred bigred  942 May 11 05:51 dtcopy
-rwxr-xr-x 1 bigred bigred  255 Apr 25 07:07 dthdpchk
-rwxr-xr-x 1 bigred bigred  452 Jul 31  2021 dtsyschk
-rwxr-xr-x 1 bigred bigred  771 May  9 15:35 dtwget

## 因為有把/home/bigred/vmusdt/bin加到PATH環境變數，所以才可以直接執行vmusdt/bin目錄裡面的命令
$ echo $PATH
/home/bigred/bin:/home/bigred/bin:/home/bigred/vmusdt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

$ cd ~/vmusdt; dir hdp33
total 28K
drwxr-sr-x 4 bigred bigred 4.0K Mar 21 22:15 .
drwxr-sr-x 4 bigred bigred 4.0K Mar 21 20:44 ..
-rwxr-xr-x 1 bigred bigred 2.6K Mar 20 16:52 us.bash
drwxr-sr-x 2 bigred bigred 4.0K Mar 21 20:36 bin
drwxr-sr-x 7 bigred bigred 4.0K Mar 20 16:41 conf
-rw-r--r-- 1 bigred bigred  420 Mar 20 16:10 dlist
-rw-r--r-- 1 bigred bigred  891 Mar 20 16:51 environmen

us.bash #設定資科平台所需的環境變數 opt #資科平台套件的存放目錄 bin #存放部署或建制資科平台所需的命令 conf #存放資科平台套件設定檔的目錄 dlist #資科平台套件下載連結清單 environment #設定 ssh 直接執行指令時的環境變數

檢視下載資科平台套件清單檔

$ cat hdp33/dlist
https://dlcdn.apache.org/hadoop/common/hadoop-3.3.2/hadoop-3.3.2.tar.gz
https://dlcdn.apache.org/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
https://dlcdn.apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz
https://dlcdn.apache.org/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.2.tgz
https://dlcdn.apache.org/hbase/2.4.12/hbase-2.4.12-bin.tar.gz
https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz

開始下載資科平台套件

$ dtwget 33
download DT packages ? (YES/NO) YES
hadoop-3.3.2.tar.gz ok
apache-hive-3.1.3-bin.tar.gz ok
pig-0.17.0.tar.gz ok
spark-3.2.1-bin-hadoop3.2.tgz ok
hbase-2.4.12-bin.tar.gz ok
apache-zookeeper-3.6.3-bin.tar.gz ok

[註] dtwget 主要是下載 DT 套件檔至 ~/vmausdt/hdp33/opt 這目錄

$ ./bin/dtcopy 33
dta1 /opt/ ok
dta1 /opt/bin ok
dta1 /etc/hosts ok

dtm1 /opt/ ok
dtm1 /opt/bin ok
dtm1 /etc/hosts ok

dtm2 /opt/ ok
dtm2 /opt/bin ok
dtm2 /etc/hosts ok

dtw1 /opt/ ok
dtw1 /opt/bin ok
dtw1 /etc/hosts ok

dtw2 /opt/ ok
dtw2 /opt/bin ok
dtw2 /etc/hosts ok

dtw3 /opt/ ok
dtw3 /opt/bin ok
dtw3 /etc/hosts ok

......

[註] dtcopy 主要是覆製 dta1 主機 /home/bigred/vmalpdt/hdp33/opt 這目錄至 dtm1, dtm2, dtw1, dtw2, dtw3 這三部主機的 /opt 目錄

複製資料科技平台設定檔

$ cat ~/vmalpdt/hdp33/us.bash
#!/bin/bash
adminuser=bigred
export JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk/jre  ## 資料科技平台所跑的Java程式的家
export HADOOP_HOME=/opt/hadoop-3.3.2
export HADOOP_LOG_DIR=/tmp
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop/
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
# export HADOOP_USER_CLASSPATH_FIRST=true
[ -z $HADOOP_USER_NAME ] && [ $SHELL == '/bin/bash' ] && declare -r HADOOP_USER_NAME=$USER

# JobHistory log file path, 會自動產生目錄
# export HADOOP_MAPRED_LOG_DIR="/home/bigred/jhslog"

export YARN_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export YARN_LOG_DIR=/tmp

# hadoop 3.2.1 及 3.1.3 這二個版本只要執行檔案傳送命令就會出現以下訊息
# hadoop fs -put -f /etc/passwd /tmp
# SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
export HADOOP_ROOT_LOGGER="WARN,console"

export PIG_HOME=/opt/pig-0.17.0
export PIG_HEAPSIZE=512
export HIVE_HOME=/opt/apache-hive-3.1.2-bin

export PATH=/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH=/usr/lib/jvm/java-1.8-openjdk/bin:/home/bigred/dt/bin:$PATH
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PIG_HOME/bin:$HIVE_HOME/bin

export HBASE_HOME=/opt/hbase-2.4.11
export HBASE_CONF_DIR=/opt/hbase-2.4.11/conf
export ZOOKEEPER_HOME=/opt/apache-zookeeper-3.6.3-bin
export ZOO_LOG_DIR=/tmp/logs
export PATH=$PATH:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin

#export ZEPPELIN_HOME=/opt/zeppelin-0.9.0-bin-all
#export PATH=$PATH:$ZEPPELIN_HOME/bin

export SPARK_HOME=/opt/spark-3.2.1-bin-hadoop3.2
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PYSPARK_PYTHON=/usr/bin/python3
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

if [ -d /opt/jupyter/miniconda3/bin ];then
   export ANACONDA_ROOT=/opt/jupyter/miniconda3
   export PATH=$ANACONDA_ROOT/bin:$PATH
   export PYSPARK_PYTHON=$ANACONDA_ROOT/bin/python
fi

if [ "$USER" != "" ]; then

   [ -f /home/$USER/dkc.env ] && source /home/$USER/dkc.env

   if [ ! -d metastore_db ]; then
      #hn=$(hostname)
      #if [ ${hn:0:3} == "adm" ] || [ ${hn:0:2} == "ds" ] || [ ${hn:0:8} == "zeppelin" ]; then
      echo -n "build derby database ..."
      schematool -initSchema -dbType derby &>/dev/null
      if [ "$?" == "0" ]; then
         echo " ok"
         echo "set hive.cli.print.current.db=true;" > .hiverc
         echo "set hive.metastore.warehouse.dir=/user/$USER/hive;" >> .hiverc
         echo "set hive.exec.scratchdir=/user/$USER/tmp;" >> .hiverc
      fi
   fi

   alias nano='nano -Ynone'
   alias dir='ls -alh'
   alias ssh='ssh -q'
fi

$ cat ~/vmalpdt/hdp33/environment
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre
HADOOP_HOME=/opt/hadoop-3.3.2
HADOOP_LOG_DIR=/tmp
LD_LIBRARY_PATH=/opt/hadoop-3.3.2/lib/native/
HADOOP_COMMON_LIB_NATIVE_DIR=/opt/hadoop-3.3.2/lib/native
HADOOP_OPTS=-Djava.library.path=/opt/hadoop-3.3.2/lib/native
HADOOP_MAPRED_LOG_DIR=/home/bigred/jhslog
YARN_HOME=/opt/hadoop-3.3.2
YARN_LOG_DIR=/tmp
PIG_HOME=/opt/pig-0.17.0/
PIG_HEAPSIZE=512
SPARK_HOME=/opt/spark-3.2.1-bin-hadoop3.2
HIVE_HOME=/opt/apache-hive-3.1.2-bin
HBASE_HOME=/opt/hbase-2.4.11
HBASE_CONF_DIR=/opt/hbase-2.4.11/conf
HBASE_LOG_DIR=/tmp
ZOO_LOG_DIR=/tmp/logs
PATH=/root/bin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hadoop-3.3.2/bin:/opt/hadoop-3.3.2/sbin:/opt/pig-0.17.0/bin:/opt/apache-hive-3.1.2-bin/bin:/opt/spark-3.2.1-bin-hadoop3.2/bin:/opt/spark-3.2.1-bin-hadoop3.2/sbin:/opt/hbase-2.4.11/bin:/opt/apache-zookeeper-3.6.3-bin/bin

降版Java

## 關閉Hadoop
bigred@dta1:~$ stopyarn ; stophdfs
## 安裝jdk8
bigred@dta1:~$ sudo apt-get install openjdk-8-jdk

## 尋找Java程式
bigred@dta1:~/vmusdt$ which java
/usr/bin/java

## /usr/bin/java是連結檔
bigred@dta1:~/vmusdt$ ls -al /usr/bin/java
lrwxrwxrwx 1 root root 22 May 11 12:12 /usr/bin/java -> /etc/alternatives/java

## /etc/alternatives/java還是捷徑檔
bigred@dta1:~/vmusdt$ ls -al /etc/alternatives/java
lrwxrwxrwx 1 root root 46 May 11 12:12 /etc/alternatives/java -> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

## JVM(java)程式真正的家
bigred@dta1:~/vmusdt$ ls -al /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
-rwxr-xr-x 1 root root 14632 Nov  3  2021 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

## 檢查
bigred@dta1:~$ java -version check
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~20.04-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

bigred@dta1:~$ nano ~/vmusdt/hdp33/us.bash
#!/bin/bash
...
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre  ##把原本的java-11改成java-8
...

==在設定一次的原因是因為只要是Hadoop的程式，都是到hadoop-env.sh這個檔案來讀取設定檔，不會去讀取us.bash（只有pig, spark, Hive才會去讀取）==

bigred@dta1:~/vmusdt$ nano hdp33/conf/hadoop-3.3.2/hadoop-env.sh
...
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre
...

還有要改

bigred@dta1:~/vmusdt$ cat hdp33/environment
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre

設定檔和套件檔的目錄名稱要一致

設定檔
bigred@dta1:~/vmusdt$ dir hdp33/conf
total 28K
drwxr-xr-x 7 bigred bigred 4.0K May 11 05:45 .
drwxr-xr-x 5 bigred bigred 4.0K May 12 06:51 ..
drwxr-xr-x 2 bigred bigred 4.0K Feb  6 16:00 apache-hive-3.1.2-bin
drwxr-xr-x 2 bigred bigred 4.0K Feb  6 16:00 apache-zookeeper-3.6.3-bin
drwxr-xr-x 2 bigred bigred 4.0K May 12 06:49 hadoop-3.3.2
drwxr-xr-x 2 bigred bigred 4.0K Feb  6 16:00 hbase-2.4.12
drwxr-xr-x 2 bigred bigred 4.0K May  9 18:54 spark-3.2.1-bin-hadoop3.2

套件檔
bigred@dta1:~/vmusdt$ dir /opt
total 40K
drwxrwxrwx 10 root root 4.0K May 12 06:00 .
drwxr-xr-x 20 root root 4.0K May 11 12:29 ..
drwxrwxrwx 10 root root 4.0K May 12 05:59 apache-hive-3.1.3-bin
drwxrwxrwx  6 root root 4.0K May 12 05:59 apache-zookeeper-3.6.3-bin
drwxrwxrwx  2 root root 4.0K May 12 06:00 bin
drwxrwxrwx  3 root root 4.0K Apr 13  2021 cni
drwxrwxrwx 10 root root 4.0K May 12 05:59 hadoop-3.3.2
drwxrwxrwx  7 root root 4.0K May 12 05:59 hbase-2.4.12
drwxrwxrwx 16 root root 4.0K May 12 06:00 pig-0.17.0
drwxrwxrwx 13 root root 4.0K May 12 06:00 spark-3.2.1-bin-hadoop3.2

更改目錄名稱
bigred@dta1:~/vmusdt$ mv hdp33/conf/apache-hive-3.1.2-bin/ hdp33/conf/apache-hive-3.1.3-bin/

拷貝設定檔到dt系列的虛擬機器

bigred@dta1:~/vmusdt$ dtconf 33

將每台電腦重新開機(a1 最後開)

bigred@dta1:~/vmusdt$ ssh dtm1 sudo reboot
bigred@dta1:~/vmusdt$ ssh dtm2 sudo reboot
bigred@dta1:~/vmusdt$ ssh dtw1 sudo reboot
bigred@dta1:~/vmusdt$ ssh dtw2 sudo reboot
bigred@dta1:~/vmusdt$ ssh dtw3 sudo reboot
bigred@dta1:~/vmusdt$ ssh dta1 sudo reboot

dthdpchk檢查

bigred@dta1:~$ dthdpchk
[dta1]
Hadoop 3.3.2
Native library checking:
hadoop:  true /opt/hadoop-3.3.2/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtm1]
Hadoop 3.3.2
Native library checking:
hadoop:  true /opt/hadoop-3.3.2/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtm2]
Hadoop 3.3.2
Native library checking:
hadoop:  true /opt/hadoop-3.3.2/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtw1]
Hadoop 3.3.2
Native library checking:
hadoop:  true /opt/hadoop-3.3.2/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtw2]
Hadoop 3.3.2
Native library checking:
hadoop:  true /opt/hadoop-3.3.2/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

[dtw3]
Hadoop 3.3.2
Native library checking:
hadoop:  true /opt/hadoop-3.3.2/lib/native/libhadoop.so
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   true /lib/x86_64-linux-gnu/libisal.so.2
PMDK:    false The native code was built without PMDK support.

formathdfs

bigred@dta1:~/vmusdt$ formathdfs
Are you sure ? (YES/NO) YES
formathdfs ok

檢查formathdfs

bigred@dta1:~/vmusdt$ ssh dtm1 tree nn
nn
└── current
    ├── fsimage_0000000000000000000
    ├── fsimage_0000000000000000000.md5
    ├── seen_txid
    └── VERSION

1 directory, 4 files

bigred@dta1:~/vmusdt$ ssh dtm1 tree dn
dn [error opening dir]

0 directories, 0 files

開啟HDFS

bigred@dta1:~$ starthdfs
dtm1: Name Node Started
dtm1: Secondary Name Node started
dtw1: Data Node started
dtw2: Data Node started
dtw3: Data Node started

bigred@dta1:~/vmusdt$ hls
[dtm1]
3257 NameNode
3482 SecondaryNameNode

[dtw1]
3104 DataNode

[dtw2]
2890 DataNode

[dtw3]
3006 DataNode

檢查

bigred@dta1:~$ ssh dtw1 tree dn
dn
├── current
│   ├── BP-832687255-120.96.143.51-1652366868764
│   │   ├── current
│   │   │   ├── finalized
│   │   │   ├── rbw
│   │   │   └── VERSION
│   │   ├── scanner.cursor
│   │   └── tmp
│   └── VERSION
└── in_use.lock

6 directories, 4 files

啟動YARN

bigred@dta1:~/vmusdt$ startyarn
dtm2: Resource Manager started
dtm2: Job History Server started
dtw1: Node Manager started
dtw2: Node Manager started
dtw3: Node Manager started

## 檢查
bigred@dta1:~/vmusdt$ hls
[dtm1]
3257 NameNode
3482 SecondaryNameNode

[dtm2]
3637 JobHistoryServer
3485 ResourceManager

[dtw1]
3104 DataNode
3947 NodeManager

[dtw2]
3649 NodeManager
2890 DataNode

[dtw3]
3780 NodeManager
3006 DataNode

檢查

bigred@dta1:~/vmusdt$ dtest
[HDFS]
Live datanodes (3):
Name: 120.96.143.53:9866 (dtw1)
Name: 120.96.143.54:9866 (dtw2)
Name: 120.96.143.55:9866 (dtw3)

[YARN]
Total Nodes:3
         Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
      dtw2:44893                RUNNING         dtw2:8042                                  0
      dtw3:46197                RUNNING         dtw3:8042                                  0
      dtw1:46179                RUNNING         dtw1:8042                                  0

[MapReduce]
Estimated value of Pi is 3.14400000000000000000

[Spark]
Pi is roughly 3.143560

[HBase]

tags: `資料科技平台`

Hadoop 實戰篇資科平台重要資訊

手動建制資科平台

手動建制資科平台

資科平台叢集建置

資科平台系統建置

複製 資料科技平台 設定檔

降版Java

tags: 資料科技平台

複製資料科技平台設定檔

tags: `資料科技平台`