★ FEATURED ARTICLE 1. 环境准备与基础配置在Ubuntu 22.04 LTS上部署Prometheus Operator前需要确保基础环境符合要求。我推荐使用全新安装的Ubuntu 22.04系统避免已有服务造成端口冲突。1.1 系统级优化首先执行系统更新和基础工具安装sudo apt update sudo apt upgrade -y sudo apt install -y curl wget git gnupg2 software-properties-common apt-transport-https ca-certificates调整内核参数以优化Kubernetes网络性能cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.ipv4.ip_forward 1 net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 EOF sudo sysctl --system提示对于生产环境建议单独准备监控节点避免与其他服务竞争资源。Prometheus Operator对磁盘IO要求较高使用SSD存储可获得更好性能。1.2 Kubernetes集群部署使用kubeadm部署Kubernetes集群单节点测试环境配置sudo apt install -y docker.io sudo systemctl enable --now docker curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - echo deb https://apt.kubernetes.io/ kubernetes-xenial main | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt update sudo apt install -y kubelet1.28.0-00 kubeadm1.28.0-00 kubectl1.28.0-00 sudo apt-mark hold kubelet kubeadm kubectl sudo kubeadm init --pod-network-cidr10.244.0.0/16 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config安装网络插件这里选择Flannelkubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml2. Prometheus Operator核心组件部署2.1 使用Helm安装Operator添加Prometheus社区Helm仓库helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update创建监控专用命名空间kubectl create namespace monitoring安装Prometheus Operator使用自定义values优化配置cat EOF custom-values.yaml prometheus: prometheusSpec: scrapeInterval: 30s evaluationInterval: 30s retention: 15d resources: requests: memory: 2Gi cpu: 1 limits: memory: 4Gi cpu: 2 storageSpec: volumeClaimTemplate: spec: storageClassName: standard accessModes: [ReadWriteOnce] resources: requests: storage: 50Gi alertmanager: alertmanagerSpec: replicas: 2 resources: requests: memory: 512Mi cpu: 500m limits: memory: 1Gi cpu: 1 EOF helm install prometheus-operator prometheus-community/kube-prometheus-stack \ -n monitoring \ -f custom-values.yaml2.2 验证安装状态检查Pod运行状态kubectl get pods -n monitoring -w预期输出应包含以下关键组件prometheus-prometheus-operator-prometheus-0 3/3 Running prometheus-operator-operator-xxx 1/1 Running alertmanager-prometheus-operator-alertmanager-0 2/2 Running grafana-xxx 1/1 Running获取Prometheus Web UI访问地址kubectl port-forward -n monitoring svc/prometheus-operator-prometheus 9090:90903. 高级配置与优化策略3.1 自定义监控目标配置创建ServiceMonitor监控Kubernetes API Servercat EOF | kubectl apply -f - apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kube-apiserver namespace: monitoring labels: release: prometheus-operator spec: jobLabel: component endpoints: - port: https scheme: https bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token tlsConfig: insecureSkipVerify: true namespaceSelector: matchNames: - default selector: matchLabels: component: apiserver provider: kubernetes EOF3.2 资源配额与自动扩缩配置Prometheus垂直自动扩缩VPAkubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/vertical-pod-autoscaler/hack/vpa-beta1-crd.yaml kubectl apply -n monitoring -f - EOF apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: prometheus-vpa spec: targetRef: apiVersion: apps/v1 kind: StatefulSet name: prometheus-prometheus-operator-prometheus updatePolicy: updateMode: Auto EOF3.3 长期存储方案集成Thanos实现长期存储可选helm repo add bitnami https://charts.bitnami.com/bitnami helm install thanos bitnami/thanos \ --set objstoreConfigthanos-config \ --set existingObjstoreSecretthanos-objstore-secret \ -n monitoring4. 告警与可视化配置4.1 自定义告警规则创建关键告警规则示例cat EOF | kubectl apply -f - apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: kubernetes-alerts namespace: monitoring labels: release: prometheus-operator spec: groups: - name: node.rules rules: - alert: NodeCPUHigh expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{modeidle}[5m])) * 100) 80 for: 10m labels: severity: warning annotations: summary: High CPU usage on {{ $labels.instance }} description: {{ $labels.instance }} CPU usage is {{ $value }}% EOF4.2 Grafana仪表板管理导入常用Kubernetes仪表板DASHBOARD_ID315 kubectl apply -n monitoring -f - EOF apiVersion: integreatly.org/v1alpha1 kind: GrafanaDashboard metadata: name: kubernetes-cluster labels: app: grafana spec: json: | $(curl -sSL https://grafana.com/api/dashboards/${DASHBOARD_ID}/revisions/latest/download | jq -c) EOF配置Grafana数据源自动发现kubectl patch -n monitoring prometheus prometheus-operator-prometheus --type merge -p { spec: { grafana: { additionalDataSources: [{ name: Loki, type: loki, url: http://loki.monitoring.svc.cluster.local:3100, access: proxy }] } } }5. 生产环境最佳实践5.1 高可用配置配置Alertmanager集群与持久化存储kubectl patch -n monitoring alertmanager prometheus-operator-alertmanager --type merge -p { spec: { replicas: 3, storage: { volumeClaimTemplate: { spec: { storageClassName: standard, resources: { requests: { storage: 10Gi } } } } } } }5.2 安全加固启用RBAC和网络策略kubectl apply -n monitoring -f - EOF apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: prometheus-allow-only-grafana spec: podSelector: matchLabels: app.kubernetes.io/name: prometheus ingress: - from: - podSelector: matchLabels: app.kubernetes.io/name: grafana EOF5.3 性能调优优化Prometheus抓取配置kubectl patch -n monitoring prometheus prometheus-operator-prometheus --type merge -p { spec: { scrapeInterval: 1m, scrapeTimeout: 10s, evaluationInterval: 1m, retention: 30d, walCompression: true, query: { lookbackDelta: 5m, maxConcurrency: 20, timeout: 2m } } }我在实际生产环境中发现当监控目标超过5000个时需要特别注意以下配置增加--storage.tsdb.max-block-duration和--storage.tsdb.min-block-duration参数调整--query.max-samples参数防止查询超时为Prometheus配置单独的持久卷避免存储压力影响其他服务
阅读完成 · 觉得有帮助？
