Skip to main content

GKE high worker pod count

Monitor: GCP Cloud Monitoring — [Critical] GKE workers - High Pod Count Service: worker (GCP, GKE Autopilot, workers namespace)

Overview​

This monitor fires when more than 200 worker-* pods are running in the workers namespace of the GKE Autopilot cluster, sustained for 5 minutes. It counts pods across all statuses (Pending/Running/Succeeded/Failed/Unknown) by summing the kube-state-metrics kube_pod_status_phase gauge.

A high pod count usually indicates runaway pod creation — pods being created faster than they are cleaned up, or workers not terminating. On Autopilot (~2 worker pods/node) 200 pods is roughly 100 nodes of spend, so a sustained breach is both a cost and a reliability risk.

🔍 Step 1: Understand the root cause​

Find out why this many pods exist:

kubectl get pods -n workers -l app=worker --sort-by=.metadata.creationTimestamp

Correlate with the queue depth / autoscaler that drives worker creation, and with any recent deploys or spike in enqueued work.

Step 2: Decide — expected or not?​

  • If the count is legitimate (real spike in work, expected load) → raise the monitor threshold. Update pod_count_threshold in create_gke_pod_count_alarm and pulumi up.
  • If the count is NOT expected → pods aren't being cleaned up. Investigate the pod logs and backend logs to find why worker cleanup/termination isn't happening (stuck jobs, failed teardown, crash-loops), and fix the source.