antony@notes:~/data-platform$ cat "實測資科平台.md"
實測資科平台
實測資科平台
第一題
YARN最大運算總量
bigred@dta1:~/vmusdt$ hdp33/bin/startmaxctn
hadoop jar /opt/hadoop-3.3.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.2.jar pi 50 300000
bigred@dta1:~/vmusdt$ hdp33/bin/stopmaxctn第二題
請各組 3 位同學在 dta2，使用不同的帳號登入，同時執行以下 3 個命令，告知執行結果的平均時間
time hive -S -e "select substring(dt,12,8) as t, sum(adult)/10000 from daily_mask where substring(dt,1,10)='2020/04/17' group by substring(dt,12,8) order by t"
—————————
time hive -S -e "select substring(dt,12,8) as t, sum(adult)/10000 from twmcompact where substring(dt,1,10)='2020/04/17' group by substring(dt,12,8) order by t" 2>/dev/null
—————————
time hive -S -e "select substring(dt,12,8) as t, sum(adult)/10000 from twmpp where substring(dt,1,10)='2020/04/17' and month='04' group by substring(dt,12,8) order by t" 2>/dev/null第一支程式
bigred@dta1:~$ time hive -S -e "select substring(dt,12,8) as t, sum(adult)/10000 from daily_mask where substring(dt,1,10)='2020/04/17' group by substring(dt,12,8) order by t"
real 2m5.104s
user 0m23.854s
sys 0m4.920s第二支程式 compact
bigred@dta1:~$ time hive -S -e "select substring(dt,12,8) as t, sum(adult)/10000 from twmcompact where substring(dt,1,10)='2020/04/17' group by substring(dt,12,8) order by t" 2>/dev/null
real 1m31.348s
user 0m21.857s
sys 0m3.571sparquet
bigred@dta1:~$ time hive -S -e "select substring(dt,12,8) as t, sum(adult)/10000 from twmparquet where substring(dt,1,10)='2020/04/17' group by substring(dt,12,8) order by t" 2>/dev/null
real 0m58.534s
user 0m21.175s
sys 0m3.968s第三支程式
bigred@dta1:~$ time hive -S -e "select substring(dt,12,8) as t, sum(adult)/10000 from twmpp where substring(dt,1,10)='2020/04/17' and month='04' group by substring(dt,12,8) order by t" 2>/dev/null
real 0m56.604s
user 0m21.122s
sys 0m3.884s