antony@notes:~/data-platform$ cat "ELK-cluster.md"

ELK-cluster

2022-08-07· data-platform ·系統工程

ELK-cluster

[TOC]

Getting Started

Get Elasticsearch_2.13-2.6.0.tgz
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.0-linux-x86_64.tar.gz
$ tar zxvf elasticsearch-7.4.0-linux-x86_64.tar.gz
$ cd ~/elasticsearch-7.4.0
Modify The Elasticsearch Config
Modify the config
$ vim ~/elasticsearch-7.4.0/config/elasticsearch.yml

## 新增一筆資料
$ curl -X PUT "localhost:9200/my-index/_doc/1?pretty" -H 'Content-Type: application/json' -d'{"user" : "Mack","age" : "20","interests"
 : "music"}'

## 檢查資料
$ curl -X GET "localhost:9200/my-index/_search?pretty"

建立索引為tibame類別employee的文件，並產生三筆資料，資料內容有first_name,last_name,age,about,interests

匯入資料

curl -H 'Content-Type: application/json' -X POST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
curl -H 'Content-Type: application/json' -X POST 'localhost:9200/bank/shakespeare/_bulk?pretty' --data-binary @shakespeare_6.0.json
curl -H 'Content-Type: application/json' -X POST 'localhost:9200/bank/hfp/_bulk?pretty' --data-binary @prod.json

#查詢所有 index
GET _cat/indices

#刪除 index
DELETE my-index-001

#創建 index 並賦予自動型別判定
PUT my-index-001
{
 "mappings": {
   "dynamic_templates":[
    {
      "longs_as_strings":{
        "match_mapping_type": "string",
        "match": "long_*",
        "unmatch": "*_text",
        "mapping": {
          "type": "long"
        }
      }
    }]
 }
}


# #寫入一筆資料
PUT my-index-001/_doc/1
{
    "long_num": "5",
    "long_text": "foo"
}


#確認 index 的資料
GET my-index-001/_search?pretty
{
  "query": {
    "match_all": {}
  }
}

PUT tibame/
{"mappings" : {
      "properties" : {
        "long_num" : {
          "type" : "long"
        },
        "long_text" :{
          "type" : "text",
          "fields": {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }

        }
      }
}
  
}

Field名稱為is開頭皆設定為布林(bollean)類型

PUT my-index-001
{
 "mappings": {
   "dynamic_templates":[
    {
      "longs_as_strings":{
        "match_mapping_type": "boolean",
        "match": "is_*",
        "unmatch": "*_text",
        "mapping": {
          "type": "long"
        }
      }
    }]
 }
}

GET /shakespeare/_search?pretty
{
  "query": {
    "match_all": {}
  }
}

假設資料流不是自己的，就可以用空查詢來確認

Query Filter

⽤查詢某個特定欄位，如match，term，range

注意，term 不會被分詞，得到的結果只有符合與不符合

命令

GET /bank/_search?pretty
{
  "query": {
    "term": {
      "firstname": {
        "value": "Amber"
      }
    }
  }
}

結果：為何找不到？

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

透過預設分詞器查看 Amber 怎麼被分詞的

命令

GET /_analyze
{
  "analyzer": "standard",
  "text": ["Amber"]
}

“Amber” 被預設的 Standard 分詞器分詞時，結果如下

{
  "tokens" : [
    {
      "token" : "amber",
      "start_offset" : 0,
      "end_offset" : 5,
      "type" : "<ALPHANUM>",
      "position" : 0
    }
  ]
}

如果要搜尋大寫的 Amber，命令如下

GET /bank/_search?pretty
{
  "query": {
    "term": {
      "firstname.keyword": {
        "value": "Amber"
      }
    }
  }
}

在 filed 後面加 “.keyword” keyword 代表不會被分詞器分詞

range

GET /bank/_search?pretty
{
  "query": {
    "range": {
      "account_number": {
        "gte": 20,
        "lte": 40
      }
    }
  }
}

match

GET /bank/_search?pretty
{
  "query": {
    "match": {
      "lastname": "Molina"
    }
  }
}

Must, must_not, should

⽤於合併其他的查詢或複合查詢語句，也就是說複合語句之間可以巢狀，⽤來表⽰⼀個複雜的單⼀查詢。

must：必須符合條件
must_not：必須不符合條件
should：如果可能符合條件

must

GET /shakespeare/_search?pretty
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "speaker": "KING"
          }
        },{
          "match": {
            "text_entry": "Have"
          }
        }
      ]
    }
  }
}

Query Aggregate

Metric(指標聚合)
- 會去計算最大最小 or 平均值的匯總函數。
Bucket(桶聚合)
- 想成SQL的 group by ，滿足一些統計，ex. 男女比例或是國家人口…等。
Matrix(矩陣聚合)
- 用於計算每一組文件欄中的統計資訊。
Pipeline(管道聚合)
- 將上一次的聚合的結果，再進行一次聚合分析

一般來說 Bucket 會配合 Matrix

語法解釋

GET bank/_search?pretty
{
  "aggs" [聚合]: {
    "NAME":[自訂整合名稱]{
      "AGG_TYPE" [聚合總類]: {
        field : column
      }
    }
  }
}

練習

GET bank/_search?pretty
{
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword"
      }
    }
  }
}

# size: 0 可以關掉值的顯示
GET bank/_search?pretty
{ "size": 0, 
  "aggs": {
    "group_by_age": {
      "range": {
        "field": "age"
      }
    }
  }
}

GET bank/_search?pretty
{ "size": 0, 
  "aggs": {
    "group_by_age": {
      "range": {
        "field": "age",
        "ranges": [
          {
            "to": 20
          },
          {
            "from": 20,
            "to": 30
          },
          {
            "from": 30,
            "to": 40
          },
          {
            "key": "大於等於40",
            "from": 40
          }
        ]
      }
    }
  }
}

GET bank/_search?pretty
{
  "size": 0, 
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword"
      },
      "aggs": {
        "group_by_gender": {
          "terms": {
            "field": "gender.keyword"
          }
        }
      }
    }
  }
}

Query Metrics

計算平均存款

GET bank/_search?pretty
{
  "size": 0,
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword"
      },
      "aggs": {
        "avg_balance": {
          "avg": {
            "field": "balance"
          }
        }
      }
    }
  }
}

索引名稱bank，以欄位state進⾏分組，統計各州數量，再依年齡計算不重複的數字共有幾個 Distinct

GET bank/_search?pretty
{
  "size": 0,
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword"
      },
      "aggs": {
        "age_distinct": {
          "cardinality": {
            "field": "age"
          }
        }
      }
    }
  }

Query Pipline

索引bank，以欄位state進⾏分組，計算各州的平均存款，並找出平均存款最低的州。

GET bank/_search?pretty
{
  "size": 0,
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword"
      },
      "aggs": {
        "avg_balance": {
          "avg": {
            "field": "balance"
          }
        }
      }
    },
    "min_balance_state":{
      "min_bucket": {
        "buckets_path": "group_by_state>avg_balance"
      }
    }
  }
}

索引logs*，以欄位@timestamp進⾏分組，依每⼩時為分組將bytes進⾏加總計。

GET logs*/_search?pretty
{
  "size": 0
  , "aggs": {
    "per_hour": {
      "date_histogram": {
        "field": "@timstomp",
        "interval": "hour"
      },
      "aggs": {
        "sum_of_bytes": {
          "sum": {
            "field": "bytes"
          }
        },
        "cul_sum_of_bytes":{
          "cumulative_sum": {
            "buckets_path": "sum_of_bytes"
          }
        }
      }
    }
  }
}

GET /shakespeare/_search?pretty
{
  "query": {
    "query_string": {
      "default_field": "text_entry",
      "query": "channel OR blood"
    }
  }
}

Logstash

## 下載 Logstash
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-7.4.0.tar.gz

## 解壓縮 Logstash
$ tar -xzf logstash-7.4.0.tar.gz

## 切換目錄
$ cd ~/logstash-7.4.0

## 開啟 Logstash
$ ~/logstash-7.4.0/bin/logstash -e 'input { stdin { } } output { stdout {} }’


$ cat sample1.conf
input {
  stdin {
  }
}
filter {
  kv {
    value_split => "="
    field_split => "&?"
  }
}
output {
 stdout {
  }
}

## 測試 `-t`
$ ../bin/logstash -f sample1.conf -t

Beats

## 下載 Beats
$ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.4.0-linux-x86_64.tar.gz

## 解壓縮 Beats
$ tar -xzf filebeat-7.4.0-linux-x86_64.tar.gz

## 切換目錄進 beats 的目錄
$ cd filebeat-7.4.0-linux-x86_64/

tags: `系統工程`

hadoop 設定檔 ELK-Overview

ELK-cluster

ELK-cluster

Getting Started

Field名稱為is開頭皆設定為布林(bollean)類型

Query Filter

range

match

Must, must_not, should

must

Query Aggregate

Query Metrics

Query Pipline

Logstash

Beats

tags: 系統工程

tags: `系統工程`