A DIY Kubernetes Cost Optimization Audit with Native Tools

A one-afternoon Kubernetes cost audit with kubectl, kube-state-metrics, VPA in recommendation mode, autoscaler logs and the EKS, GKE and AKS cost views. Every tool measures, none of them decides, and that's where the audit stops.

GuideUpdated Nov 2, 2026

Cover Image for A DIY Kubernetes Cost Optimization Audit with Native Tools

A DIY Kubernetes Cost Optimization Audit with Native Tools

Run this on your most expensive cluster before reading further:

kubectl describe nodes | grep -A 8 "Allocated resources"

Then run kubectl top nodes next to it. If most nodes show 70% or more of CPU requested and 15% or less actually used, you're looking at the most common Kubernetes cost problem there is. The nodes can't be scaled away, because the scheduler thinks they're full. They're mostly air.

Block out one afternoon and you can walk out with a findings document that names every over-requested deployment, every half-empty node and every workload blocking scale-down, using only tooling that ships with Kubernetes or comes free from your cloud provider: kubectl and metrics-server, kube-state-metrics, the Vertical Pod Autoscaler in recommendation mode, cluster-autoscaler logs, and the cost views in the EKS, GKE and AKS consoles.

Part one mapped where cluster spend leaks: overprovisioned requests and limits, idle and underpacked nodes, orphaned volumes and load balancers, forgotten namespaces and missing autoscaling. This part is the audit itself, grouped by tool, because each tool answers a different question. Part three hands the verification to CloudThinker agents working through a read-only Kubernetes connection.

01Tool 1

kubectl top

Requests against live usage, pod and node, right now.

02Tool 2

kube-state-metrics

The same gap per namespace, over a full week.

03Tool 3

VPA, recommend only

What each request should be, from observed traffic.

04Tool 4

Autoscaler logs

Nodes it wants to remove and what blocks it.

05Tool 5

Cost consoles

Dollars per namespace, so findings get prioritized.

Each tool answers one question. Run them in this order and each narrows the next. Budget an afternoon per cluster.

Tool 1: kubectl top and metrics-server, the requests-versus-usage gap

The biggest Kubernetes waste source is the gap between what pods request and what they use. The scheduler and the cluster autoscaler both work from requests, not usage, so this gap is priced into every node you run.

First, confirm metrics-server is installed (EKS needs you to install it; GKE and AKS ship it by default):

kubectl get deployment metrics-server -n kube-system

Then look at the node picture:

kubectl top nodes

How to read it. kubectl top nodes shows actual CPU and memory usage per node. Now compare it with what's reserved:

kubectl describe nodes | grep -A 8 "Allocated resources"

For each node you get two very different numbers: allocated (the sum of pod requests, what the scheduler thinks is used) and live usage from top. A node at 78% CPU requested and 11% CPU used is the classic signature. That ratio, usage over requests, is the number the rest of this audit chases.

Find the worst offenders at the pod level:

kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory

And dump every workload's resource requests to compare:

kubectl get pods -A -o custom-columns=\
'NS:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory'

Cross-reference the two lists. A pod requesting 2 CPU while top shows 40m is a 50x overask. Also note every pod whose request columns come back empty. Unset requests are the opposite failure (they schedule greedily and get evicted unpredictably), and they'll skew every number in the rest of this audit.

One caveat before you file tickets: kubectl top is a point-in-time sample. A batch job that spikes nightly will look idle at 2 p.m. The next two tools fix that.

Tool 2: kube-state-metrics and Prometheus, requests over time

kube-state-metrics exports the declared state of the cluster, including every container's requests and limits, as Prometheus metrics. If you run any Prometheus-compatible stack (kube-prometheus-stack installs both), you can turn the spot check above into a seven-day view.

Requested CPU per namespace:

sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)

CPU actually used per namespace:

sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)

The number that drives the whole audit is utilization of requests, and you want its peak over a full week, not its average:

max_over_time(
  (
    sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)
    /
    sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)
  )[7d:5m]
)

Repeat with resource="memory" and container_memory_working_set_bytes (no rate()) for the memory side.

How to read it. A namespace whose peak never goes above 25% of requested CPU across a full week, busiest hour included, is carrying at least two to three times more request than it needs. Then rank by absolute waste, not by ratio. A 40-core namespace at 30% is worth more than a 2-core namespace at 5%:

topk(10,
  sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)
  -
  sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)
)

That list, in idle cores, is your remediation priority. Expect it to be short: a few services usually hold most of the idle request.

Tool 3: VPA in recommendation mode, the right-size numbers

You now know which workloads are over-requested. The Vertical Pod Autoscaler tells you what their requests should be, and with updateMode: "Off" it only recommends. It never touches a running pod. GKE has VPA built in (gcloud container clusters update CLUSTER --enable-vertical-pod-autoscaling, see the GKE docs). AKS offers it as a managed add-on. On EKS you install the upstream version:

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

Create one VPA object per suspect workload from your Tool 2 ranking:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api-recommender
  namespace: prod
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    updateMode: 'Off'

Give it a day or two of real traffic, then read the recommendation:

kubectl describe vpa checkout-api-recommender -n prod

How to read it. The Status.Recommendation block gives you four numbers per container. Target is the request VPA would set: your right-size figure. Lower Bound is the floor below which the pod is likely to be throttled or OOM-killed. Upper Bound is the ceiling worth considering for spiky workloads. Uncapped Target is what VPA wanted before any policy constraints. A deployment requesting 2000m CPU with a target of 312m is your finding, with the safe new number attached.

Two cautions. Let VPA watch at least one full weekly traffic cycle before trusting a target for anything customer-facing. And don't let VPA in Auto mode and an HPA both act on CPU for the same deployment, because they'll fight. When you lower a memory request, keep a memory limit, so a bad estimate ends in one container restart instead of node memory pressure.

Tool 4: cluster-autoscaler logs, idle nodes and scale-down blockers

Over-requests waste money inside nodes. The autoscaler tells you about waste at the node level: nodes it knows are unneeded but can't remove. On EKS with the self-managed cluster autoscaler, start with the status ConfigMap:

kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml

Then search the logs for scale-down decisions:

kubectl -n kube-system logs deploy/cluster-autoscaler --tail=2000 \
  | grep -iE "scale.?down|unneeded|no.?scale"

On GKE and AKS the autoscaler is managed and its pod isn't visible. On GKE, read the cluster autoscaler visibility events in Cloud Logging (the cluster-autoscaler-visibility log for the cluster). On AKS, use kubectl get events -A --field-selector source=cluster-autoscaler.

How to read it. Two findings live here. First, nodes marked unneeded for long stretches that never terminate: each one is a full instance of pure waste. Second, and more useful, the reasons scale-down is blocked, which the logs state outright: pods without a controller, pods using local storage, restrictive PodDisruptionBudgets, or the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation. List the PDBs that allow zero evictions, since one of them can pin a whole node:

kubectl get pdb -A -o json \
  | jq -r '.items[] | select(.status.disruptionsAllowed == 0)
    | "\(.metadata.namespace)/\(.metadata.name)\tminAvailable=\(.spec.minAvailable // "-")\tmaxUnavailable=\(.spec.maxUnavailable // "-")"'

Every blocker you fix compounds: it frees a node today, and it lets the autoscaler keep packing nodes every day after.

While you're at the node level, check bin-packing directly. If kubectl describe nodes shows most nodes stuck around half requested, your pods' request shapes don't fit your instance shape. A different node size, or a second node group for large pods, can recover whole nodes with no workload changes.

Tool 5: cloud cost views, dollars instead of cores

Everything so far is in cores and GiB. Your cloud's billing console turns it into dollars per namespace or workload, which is what turns an audit into an approved ticket.

  • EKS: turn on split cost allocation data for EKS under Billing and Cost Management → Cost Management Preferences. Within about a day, your Cost and Usage Report gains pod-level cost rows you can group by cluster, namespace and workload.
  • GKE: turn on GKE cost allocation (gcloud container clusters update CLUSTER_NAME --enable-cost-allocation), then group billing reports by namespace or cluster label. The GKE console's Clusters → Cost optimization tab also charts requested against used per cluster, with no setup.
  • AKS: turn on the AKS cost analysis add-on (az aks update -g RG -n CLUSTER --enable-cost-analysis, Standard or Premium tier), then open Cost Management → Cost analysis and pick the Kubernetes views for namespace and asset breakdowns.

How to read it. Put each namespace's monthly dollar figure next to its peak utilization from Tool 2. "Namespace X costs $4,200 a month and peaks at 22% of what it requests" is the sentence that gets a right-sizing ticket prioritized.


What each tool answers

ToolAnswersDoesn't answer
kubectl topUsage against requests, right nowWhether right now is typical
kube-state-metrics + PromQLPeak utilization per namespace over a weekWhat the new request should be
VPA in recommendation modeThe right-size request per containerWhich node it frees, or what that saves
Autoscaler logs and PDBsWhich nodes are unneeded and why they stayWhether a blocking PDB is load-bearing
Cloud cost viewsDollars per namespace and workloadWhich change to make first

Read the right-hand column top to bottom and you have the honest ceiling of the native toolchain: every tool measures, none of them joins the measurements or decides.

Where the DIY audit stops

Run this audit. It works. But know its limits before you make it your strategy.

01Cadence

A snapshot

Requests drift with the next deploy. VPA targets describe this week.

02Cost

Recurring hours

One cluster is an afternoon. Ten clusters on three clouds is headcount.

03Correlation

Joined by hand

Nothing links a VPA target to the node it would free or the dollars it saves.

04Action

PRs for others

Every finding becomes a change to someone else’s Helm values.

The native toolchain is good at measurement. It stops where verifying and acting begin.

It's a snapshot. Requests drift the day after you tune them, because every deploy can reset them. The VPA targets you collected this week describe this week's traffic.

It's engineer hours, recurring. One cluster, one afternoon. Ten clusters on three clouds, every quarter, is real headcount, and it's the first thing dropped in a busy sprint.

The tools don't talk to each other. You joined top output, PromQL, VPA objects, autoscaler logs and a billing console by hand. Nothing connects a VPA target to the node it would free and the dollars that node costs.

Nothing here remediates. Every finding becomes a PR to someone else's Helm values, and tickets without an owner are where the savings quietly expire.

The work left after the native tools is verification and judgment: is that idle namespace really abandoned, does that PDB protect something real, will the new request survive Monday's peak. Part three hands that work to CloudThinker agents, which run the same kubectl and PromQL checks through your connections, on a schedule you set, and leave every change to you.

Related reading

About the author

ST

Steve Tran

Founder & CTO, CloudThinker

Steve Tran is the founder and CTO of CloudThinker, which builds Frontier Investigation Agents for production cloud operations. He writes about incident investigation, cloud cost, and the practices teams use to safely run autonomous AI agents against production AWS, GCP, and Azure environments.

kubernetescostoptimizationfinopskubectlautoscalingdevops