Splitting a shared Kubernetes cluster's cost across teams
The CostReveal team, , 5 minute read
A shared Kubernetes cluster is efficient to run and hard to pay for. The cloud bill says what each node cost. It cannot say which team's workloads ran on it, because the cluster, not the bill, decides where each pod goes. So the cluster's cost lands on whoever owns the account, and every team that uses it pays nothing. This post covers how to divide a cluster's cost fairly, the choices that make one split fairer than another, and how CostReveal makes each choice visible.
Start from the bill, not a price list
A cluster priced from a list price per core and per gigabyte produces a number that looks right and adds up to nothing on the invoice. The fair split starts from what was actually billed: each node's own line on the bill, at the price you paid, with your discounts, Savings Plans and Spot prices in it. CostReveal finds each node on the bill by a tag its nodes carry, and divides only what was billed, so the cluster's teams add up to the cluster's cost to the cent.
Requested or used? The greater of the two
A workload costs what it holds away from others. Kubernetes reserves what a container requests, whether it uses it or not, and a container can use more than it requested when the node has room. Each choice alone is unfair:
- By requests alone, a workload that bursts above its request uses capacity it never pays for.
- By usage alone, a workload that requests a large reservation and leaves it idle pays nothing for the room it denied everyone else.
So a workload takes the greater of what its containers requested and what they used, of CPU and of memory, each over the day. Over-requesting costs the team that does it, and bursting costs the team that bursts.
Node by node, at that node's price
Nodes are not alike. A core on a large memory-optimized instance costs more than one on a general purpose instance, and a GPU node costs far more than either. Averaging the cluster hides that: a team on the costly pool would be charged the cheap pool's price.
So each node's line is divided among the workloads that ran on that node, by that node's capacity. A workload running on an expensive node pays that node's price. Lines that belong to no node, such as disks, load balancers and the control plane, are divided across the whole cluster.
GPUs priced as GPUs
On a GPU node, most of the price is the GPU. Divided by CPU and memory alone, a GPU node's cost would land on whatever ran beside the model, not on the model. So on a node with GPUs, the GPUs carry most of its cost, divided by the GPUs each workload requested for as long as it ran, NVIDIA's and AMD's. The share is a setting of the cluster, and so is how CPU and memory split the rest.
Idle stays in the open
Capacity no workload took is idle. It is real cost: you paid for the node. Spreading it across teams in proportion to their use makes every team look more expensive than it is, and hides the waste from the people who could remove it.
So idle is its own row, per node and per cluster. Days the cluster sent no metrics are unmeasured, also their own row, and never divided. The team that runs the cluster sees exactly what is unused, and can right-size the pool.
Owners from the labels you already have
A pod labelled team: payments is the payments team's, read with the same tag keys as your cloud resources. A pod's own label comes first, then its workload's, then its namespace's, then its node's, so a team that labels its Deployments, its namespaces or a node pool needs to label nothing else. Tags still come first: a node line already tagged for a team stays that team's, and the cluster divides only what tags and named owners leave unknown.
What workloads explain is shown as allocated, split by measured use. A workload with no owner label stays unknown on that dimension, shown as such and never spread across teams.
What you can do with it
Explore groups by cluster, namespace and workload, with idle and unmeasured days as rows of their own, and crosses each with the provider, account, region and service. The Kubernetes page shows each cluster's cost: what its nodes' own lines explain, what is shared across the cluster, and what is idle. The same figures flow into budgets, chargeback and reports, because a team's cluster cost is part of the team's spend like any other.
Frequently asked questions
What runs in my cluster? The OpenTelemetry Collector, installed from its official chart with the values CostReveal gives you, or one more exporter on a Collector you already run. It sends what each node offers and each container requested and used, of CPU, memory and GPUs, and the labels that name owners, with each Node and Pod object cut to those fields before it leaves the cluster.
Why the greater of requests and usage? A reservation denies capacity to others even when unused, and usage above a request takes capacity no one paid for. The greater of the two charges both.
What happens to idle capacity? It stays its own row, so the waste is visible to whoever can remove it.
How are GPUs handled? On a GPU node, the GPUs carry most of the cost, divided by the GPUs each workload requested.
Can a team label a namespace instead of every pod? Yes. A pod's own label comes first, then its workload's, its namespace's and its node's.
Keep reading
- Connecting Kubernetes to CostReveal: the Collector, and the tag that finds each node on the bill.
- Allocation rules for shared costs: splitting what the cluster does not.
- Why CostReveal shows the spend it cannot explain: why idle and unknown stay in the open.