雲端原生 ELK 日誌系統 - ELK on Kubernetes

# 雲端原生 ELK 日誌系統架構轉型
# Cloud-Native ELK Stack Migration to Kubernetes

Description:
針對早期單機版 EC2 部署所導致的資源擴展困難與高昂維運成本，導入 Kubernetes 架構實現 ELK 服務的容器化轉型。此專案為公司首個生產環境 K8s 應用，透過自動化編排解決大規模日誌處理的彈性伸縮與存儲管理挑戰。
To address the scaling limitations and high maintenance overhead of single-instance EC2 deployments, I led the migration of ELK services to a Kubernetes-based architecture. As the company's first production K8s project, it was designed to leverage automated orchestration for large-scale logging environments.

Tech Stack:

• AWS EKS / ECR / ELB
• Python (FastAPI / boto3 / K8s Client)
• StatefulSet / Deployment / DaemonSet
• Prometheus / cAdvisor / Grafana

Features:

• 一站式自動化建置 (One-Click Provisioning)
  開發整合前端介面與後端 API，透過 Kubernetes Python Client 調用 K8s API，實現 ELK 集群的一站式自動化建置。
  Developed an integrated platform leveraging FastAPI backend to interface with the Kubernetes API for automated ELK cluster provisioning.
• 客製化映像檔管理 (Custom Image Management)
  針對內部需求客製化 Elasticsearch 與 Logstash 映像檔，並建立 AWS ECR 私有倉庫管理機制。
  Customized Elasticsearch and Logstash container images with secure lifecycle management using AWS ECR.
• StatefulSet 狀態管理 (Stateful Workloads)
  靈活應用 StatefulSet 管理具狀態的數據節點，搭配 ConfigMap 與 LoadBalancer 服務，優化日誌系統的持久化與對外存取。
  Leveraged StatefulSets for stateful data node management with ConfigMaps and LoadBalancer services.
• Prometheus 監控方案 (Monitoring Suite)
  架構 Prometheus 與 cAdvisor 監控方案，搭配 Grafana 儀表板，即時觀測 Kubernetes 叢集健康度與 ELK 處理效能。
  Architected a monitoring suite using Prometheus and cAdvisor integrated with Grafana dashboards.

Results:

• 將 ELK 環境的建置時間從原本的手動 1 小時大幅縮短至 5 分鐘內，極大化交付速度。
  Reduced ELK environment setup time from 1 hour of manual effort to under 5 minutes.
• 透過 K8s 的自我修復 (Self-healing) 特性，將維運頻率從每月數次降低至每年僅需極少的人工介入。
  Leveraged K8s self-healing capabilities to reduce maintenance incidents to near-zero intervention annually.
• 實現資源的動態垂直擴展，能根據業務流量即時調整計算與存儲規模，確保系統穩定性。
  Enabled dynamic vertical scaling for real-time adjustment of compute and storage resources.

Architecture Overview:

sequenceDiagram
    participant User as Operator
    participant Platform as FastAPI Backend
    participant K8s as Kubernetes API
    participant ECR as AWS ECR
    participant ELB as AWS ELB
    
    User->>Platform: Create ELK Cluster Request
    Platform->>K8s: Create Namespace
    Platform->>K8s: Create Service (type: LoadBalancer)
    K8s->>ELB: Provision Load Balancer
    Platform->>K8s: Create ConfigMap (ES/Logstash Config)
    Platform->>ECR: Pull Custom ES Image
    Platform->>K8s: Create StatefulSet (Elasticsearch)
    Platform->>ECR: Pull Custom Logstash Image
    Platform->>K8s: Create StatefulSet (Logstash)
    Platform->>K8s: Create Deployment (Kibana)
    Platform->>K8s: Create Prometheus Config
    K8s-->>Platform: All Resources Ready
    ELB-->>Platform: Return Kibana ELB Endpoint
    Platform-->>User: ELK Cluster Ready (Kibana URL)