antony@notes:~/data-platform$ cat "Minio-Architecture.md"

Minio Architecture

2024-09-22· data-platform

Minio Architecture

認識 MinIO

MinIO was designed from the beginning to be a fully compatible alternative to Amazon’s S3 storage API. They claim to be the most compatible S3 alternative while also providing comparable performance and scalability.

MinIO also provides a variety of deployment options. It can run as a native application on most popular architectures and can also be deployed as a containerized application using Docker or Kubernetes.

Additionally, MinIO is open-source software. Organizations are free to use it under the terms of the AGPLv3 license. Just beware that this option comes with no support aside from online documentation and the MinIO user community. For larger enterprises, paid subscriptions with dedicated support are also available.

Minio 有分成 3 個架構

Single-Node Single-Drive (SNSD or “Standalone”) A single MinIO server with a single storage volume or folder. SNSD deployment provides failover protections. Drive-level reliability and failover depends on the underlying storage volume. SNSD deployments are best suited for evaluation and initial development of applications using MinIO for object storage.
Single-Node Multi-Drive (SNMD or “Standalone Multi-Drive”) A single MinIO server with four or more storage volumes. SNMD deployments provide drive-level reliability and failover only.
Multi-Node Multi-Drive (MNMD or “Distributed”) Multiple MinIO servers with at least four drives across all servers. The distributed MNMD topology supports production-grade object storage with drive and node-level availability and resiliency.

Minio 系統中的多顆硬碟可以分為 Data shard 和 Parity shard 兩個種類

連接 MinIO Server

在 Taroko K8S 管理主機執行以下命令 : 
$ mc config host add mios http://miniosnsd.s3-system:9000 minio minio123
.........
Added `mios` successfully.

[註] mios 是連接名稱, 不是 Bucket 名稱

$ mc config host ls
..........
mios
  URL       : http://miniosnsd.kube-system:9000
  AccessKey : minio
  SecretKey : minio123
  API       : s3v4
  Path      : auto
..........

管理 MinIO Bucket

# 建立 haha bucket
$ mc mb mios/haha
Bucket created successfully `mios/haha`.

# 透過 dd 命令產生 500 Mib 的檔案
$ dd if=/dev/zero  of=/tmp/500M bs=10485760 count=50
50+0 records in
50+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 2.09069 s, 251 MB/s

# 將檔案複製進 Minio
$ mc cp /tmp/500M  mios/haha
/tmp/500M:       500.00 MiB / 500.00 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 220.76 MiB/s 

# 檢查是否符合預期
$ mc ls mios/haha
[2024-09-22 09:31:18 CST] 500MiB STANDARD 500M

# 刪除 haha bucket 中的檔案
$ mc rm mios/haha/500M
Removed `mios/haha/500M`.

# 刪除 haha bucket
$ mc rb mios/haha
Removed `mios/haha` successfully.

Minio 本身在儲存的時候不是檔案系統作業，有它自己儲存資料的方式，所以像是 MySQL 這類的資料庫需要有檔案系統來寫入資料，它們就沒辦法直接儲存在 Minio 中，要透過 fuse 多墊一層檔案系統，才能讓 MySQL 資料庫的資料儲存在 Minio 中。

fuse 可以用 s3fs-fuse，1.93 以前的版本不要用，1.94 有穩住，但還是沒辦法儲存高吞吐量的資料，像 MySQL 如果透過 s3fs 來將資料儲存在 Minio 中，就會遇到 I/O 不足的挑戰，s3fs 比較適合靜態的檔案儲存。

第二種 fuse 可以選擇 JuiceFS，這專案就會比 s3fs 還要適合儲存高吞吐量的資料。

Ceph 也有自己做出來的檔案系統，讓使用者能將資料寫入 Ceph 的儲存體。

MinIO & Python

安裝 Python MinIO 套件

$ sudo mv /usr/lib/python3.12/EXTERNALLY-MANAGED /usr/lib/python3.12/EXTERNALLY-MANAGED.old

$ pip3 install minio
Defaulting to user installation because normal site-packages is not writeable
Collecting minio
  Downloading minio-7.2.4-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi (from minio)
  Downloading certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Collecting urllib3 (from minio)
  Downloading urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting argon2-cffi (from minio)
  Downloading argon2_cffi-23.1.0-py3-none-any.whl.metadata (5.2 kB)
Collecting pycryptodome (from minio)
............

$ mkdir -p ~/wulin/pyminio

使用 python 或 c # …程式語言將資料寫入 Minio 就可以不用多一層 fuse ，因為這些語言都有提供連接、呼叫和此用 Minio 的 library

撰寫 MinIO Python 程式

$ echo 'from minio import Minio
from minio.error import S3Error
def main():
    client = Minio(
        "miniosnsd.s3-system:9000",
        secure=False, access_key="minio", secret_key="minio123"
    )
    found = client.bucket_exists("mypy")
    if not found:
        client.make_bucket("mypy")
        print("Created bucket", "mypy")
    else:
        print("Bucket mypy already exists")

if __name__ == "__main__":
    try:
        main()
    except S3Error as exc:
        print("error occurred.", exc) ' >~/wulin/pyminio/mkbucket.py

執行 MinIO Python 程式

$ python ~/wulin/pyminio/mkbucket.py
Created bucket mypy

$ mc ls mios
[2024-03-18 14:57:58 CST]     0B mypy/

$ mc rb mios/mypy

# 離開 Taroko 管理主機
$ exit

資料儲存在 Minio + K8s 當作資料處理引擎 + Spark 做資料分析 + Iceberg 可以讓資料的欄位異動，以上就能跑出資料智慧平台。

JuiceFS & MinIO

JuiceFS 系統架構

資料能夠儲存在 Ceph 或 S3 的 Storage Meta Data (紀錄目錄的名稱、權限，目錄有哪些檔案、它們的權限…等，或是 Linux 檔案系統特定的權限 sticky bit…等) 能夠儲存在 Redis 資料庫

JuiceFS 有提供內建的 FUSE 以檔案系統的方式將資料儲存在 JuiceFS 中，在 K8s 也有提供 CSI Driver 讓 K8s 的 Pods 能夠將資料儲存在 JuiceFS 中

Minio YAML

kind: Service
apiVersion: v1
metadata:
  name: miniosnsd
  namespace: s3-system
  labels:
    app: miniosnsd
#  annotations:
#    metallb.universe.tf/address-pool: mlb-pool
spec:
#  allocateLoadBalancerNodePorts: false
#  type: LoadBalancer
#  loadBalancerIP: 172.22.1.150
  selector:
    app: miniosnsd
  ports:
  - name: mclient
    protocol: TCP
    port: 9000
    targetPort: 9000
  - name: mconsole
    protocol: TCP
    port: 9001
    targetPort: 9001
---
apiVersion: v1
kind: Pod
metadata:
  name: miniosnsd
  namespace: s3-system
  labels:
    app: miniosnsd
spec:
  hostname: minioserver
  volumes:
  - name: miniodata
    hostPath:
      path: /var/miniosnsd
  containers:
  - name: minioserver
    image: minio/minio
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: miniodata
      mountPath: /disk
    args:
    - server
    - --console-address
    - :9001
    - /disk
    env:
    - name: MINIO_ROOT_USER
      value: "minio"
    - name: MINIO_ROOT_PASSWORD
      value: "minio123"
    ports:
    - containerPort: 9000
      name: mclient
      protocol: TCP
      hostPort: 9000
    - containerPort: 9001
      name: mconsole
      protocol: TCP
      hostPort: 9001
  nodeSelector:
    app: taroko

Minio 的資料實際上儲存在 K8s Node 主機的 /var/miniosnsd 目錄區底下，指定在 /var 目錄底下的原因是因為 K8s 專用的作業系統，很多 root filesystem 都是 read-only ，但允許 /var 目錄區能夠做修改

Btrfs send/receive 實作 OpenZFS