CPU utilization lies: autoscaling a single-threaded service

The service was slow. Not down, just slow: p95 latency climbing well past where users notice, requests piling up, the kind of degradation that generates support tickets instead of alerts. And the autoscaler, the whole point of which is to add capacity when a service is under strain, sat there doing nothing.

The metric it was watching said everything was fine. Average CPU utilization on the tasks was hovering around 30 percent, nowhere near the scale-out threshold. The dashboard was calm. The users were not. Both were right, and the gap between them is one of the most common autoscaling traps on a container platform.

This is the first article in a series on running a multi-tenant SaaS on AWS at team scale. It is about a metric that lies, quietly, by design.

Why 30 percent CPU meant 100 percent busy

The service was a single-threaded application. A Node.js API, in this case, but the same is true of any process that does its real work on one thread: a classic Python or Ruby worker, most single-process runtimes.

A single-threaded process can, by definition, saturate exactly one CPU core. The task it was running on had four vCPUs. So the arithmetic that matters is brutally simple:

one core fully pegged / four vCPUs on the task = ~25% task-average CPU

At full saturation, the busiest that process can ever make the task look is about 25 percent. Add a little async I/O overhead spread across the runtime and you land around 30 percent. That is not a service with headroom. That is a service redlining on the only core it can use, while three cores sit idle and drag the average down to a number that reads as “barely working.”

The autoscaling policy was tracking average CPU across the task’s cores. For a workload that can only ever use one of them, that average is not a measure of load. It is a measure of load divided by four.

The metric was answering a different question

This is the real lesson, and it is not specific to AWS or ECS. Average CPU utilization answers “how much of the machine’s total compute is in use.” Autoscaling needs the answer to a different question: “is this service keeping up with its work.”

For a multi-threaded service that spreads across all its cores, those two questions have nearly the same answer, which is why CPU is the default scaling metric and why it usually works. For a single-threaded service, they diverge completely. The service can be fully saturated and failing its latency targets while the CPU metric, honestly and correctly, reports 30 percent.

Scaling on the wrong metric is worse than not scaling, because it comes with a dashboard that actively reassures you. Everything looks healthy right up until a human notices it is not.

Three ways to fix it

There is a quick lever and a couple of durable ones.

Size the task to one vCPU. If the process can only use one core, stop giving it four. On a one-vCPU task, “one core saturated” is 100 percent CPU, and the autoscaler can finally see the truth. This is the cheapest fix and it doubles as a cost cleanup: you were paying for three idle cores per task. Standardising the fleet to one vCPU per service is not a downgrade here, it is making the reported metric mean something again.

Scale on a signal that reflects the actual constraint. The workload is latency-bound, so scale on latency or on request pressure, not CPU. On this stack that means target-tracking on the load balancer’s target response time, or on request count per target, rather than CPU utilization. A custom metric works too: for an event-loop runtime, event-loop lag is a direct measure of “this process is behind.” Scale on the thing that hurts, not on a proxy that averages it away.

Run one worker per core if you must use big tasks. If a task really needs four vCPUs, then run four workers on it (cluster mode, one process per core, a process manager, or four smaller tasks instead of one big one). Now all four cores can be busy, and average CPU becomes an honest metric again. The rule underneath: CPU utilization is only meaningful when your app can actually use all the cores it is measured against.

The immediate mitigation, while you decide, is to scale out by hand: add tasks and watch latency. If more tasks fix the latency, you have just confirmed the service was capacity-starved all along, and you can lower the scaling threshold (or switch metrics) with confidence.

The takeaway

Autoscaling is only as good as the metric it watches, and the default metric quietly assumes your app uses every core it is given. A single-threaded service breaks that assumption: it maxes out one core, reports a fraction of the task’s total CPU, and sails under a CPU-based scale-out threshold while its latency falls apart.

Match the metric to how the app actually consumes resources. Size single-threaded tasks so one core is the whole task and CPU tells the truth, or scale on latency and request pressure instead. And treat a calm dashboard next to unhappy users as a bug in your monitoring, not a coincidence. The metric that says everything is fine is the first thing to distrust.