From Alerting to Inference: Metrics Never Stopped Mattering
In this post
Your LLM is slow, users are complaining and the queues keep growing. Someone on the team is already profiling the model, looking at batch sizes, considering a bigger GPU. Nine times out of ten, the answer is already in the metrics.
I’ve spent most of my career staring at metrics, on bare-metal servers, Kubernetes clusters and managed services on public cloud. The lesson I keep re-learning is that the infrastructure is lying to you and you’re not asking it the right questions. LLM serving is the same lesson in a different domain.
How the job of metrics changed#
When I started doing infrastructure work, metrics had exactly one purpose: tell me when something is broken. A threshold gets crossed, an alert fires, someone wakes up.
Over the years I watched metrics go from passive alarms to something that drives how systems behave, each step built on top of the previous one. In the first two steps a person reads the number and acts on it (amber); in the last two the system does it by itself (green).
something broke"] --> B["Capacity planning
how much do we need?"] B --> C["Auto-scaling
the system reacts"] C --> D["Intelligent routing
per-request decisions"] classDef human fill:#40361d,stroke:#eec35e,stroke-width:1.5px,color:#eceae5 classDef system fill:#343621,stroke:#a8c373,stroke-width:2px,color:#eceae5 class A,B human class C,D system
None of this was planned. Nobody sat in a room in 2012 and said “one day we’ll route individual inference requests based on GPU memory pressure.” It happened one problem at a time, and the foundation never changed: expose a number, collect it, act on it.
Alerting, and how Prometheus became the standard#
Before Prometheus, monitoring was a mess. Every tool had its own data model, its own query language and its own way of defining “something is wrong.”
I’ve operated Nagios, Icinga and Zabbix. They all worked, kind of, but they were built around host-based checks: is this server up? Is this disk full? Is this process running? When you moved to dynamic environments (containers, microservices, autoscaling groups), that model fell apart fast.
Prometheus got a few things right that the others didn’t:
Pull-based collection. The monitoring system scrapes targets, not the other way around. Targets come and go (hello, Kubernetes pods), and Prometheus handles discovery.
Dimensional data model. Labels instead of hierarchical metric names. http_requests_total{method="GET", status="200", service="api"} is way more flexible than servers.api01.http.get.200.count.
PromQL. A real query language. You can ask it more than “what’s the value now?”, for example:
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket{service="api"}[5m])
)
That gives you the p99 latency of your API over the last 5 minutes. Try doing that with Nagios.
Prometheus became a standard because it solved a real problem in a way that made sense, not because the CNCF said so (though it was the second project to graduate, right after Kubernetes). Cloud providers adopted the same exposition format and remote-write protocol: Google Cloud has Managed Service for Prometheus, Azure has Prometheus-compatible monitoring, and AWS has it too. They don’t run Prometheus’s TSDB under the hood, but they speak the same language. When everyone converges on the same data model, it’s usually the right one.
(Prometheus’s own storage story is a different matter. If you’ve ever tried to keep 6 months of metrics in a single Prometheus instance, you know why Thanos and Mimir exist. But that’s not the point here.)
The alerting pipeline looks like this: blue is the metric, orange is the bad news, and amber is where a person gets involved.
/metrics endpoint"] --> P["Prometheus
scrapes"] P --> R["PromQL rule
evaluated"] R --> AL["Alert fires"] AL --> AM["Alertmanager"] AM --> N["PagerDuty, Slack
or email"] classDef metric fill:#2d3128,stroke:#83a598,stroke-width:1.5px,color:#eceae5 classDef bad fill:#d65d0e,stroke:#d65d0e,color:#1a170f classDef human fill:#40361d,stroke:#eec35e,stroke-width:1.5px,color:#eceae5 class T,P metric class AL bad class N human
Once you have all this data collected continuously and queryable in real time, the obvious next question is why a human is the only thing acting on it.
When CPU metrics lie#
I was working with a team running a fairly standard web application on Kubernetes. They had the textbook setup: Horizontal Pod Autoscaler watching CPU utilization, scale up at 70%, scale down at 30%.
Except users were getting 503s and latency was climbing, while the dashboards said CPU was fine, sitting happily at 40-50%.
The problem had several layers, and it took a while to peel them all back.
CPU was the wrong signal#
CPU utilization is a terrible scaling signal for most applications. It tells you how busy the processor is, which says little about how well the application is serving users. An app can have low CPU and still be completely stuck waiting on database connections, DNS resolution or downstream services. The HPA was scaling on a metric with almost no correlation to user experience.
New pods took too long to start#
When the HPA did eventually trigger (because CPU finally caught up), new pods took too long to become ready: a cold JVM, dependency initialization, health check delays. By the time the new capacity was online, requests had already queued up and timed out.
The metrics we added#
We instrumented the application to expose what actually mattered:
- Connection pool utilization (are we running out of database connections?)
- Response time percentiles per endpoint (p95 and p99, not averages)
- SQL query execution duration (is the database the bottleneck?)
Then we rebuilt the scaling signal. Instead of CPU, we used a composite: gateway-measured latency per user session, average application response time, database load and pod readiness time. The HPA now scaled on what users were experiencing.
Tracing pointed at DNS#
We still had latency spikes that the new metrics didn’t fully explain, so we turned on distributed tracing. Service-to-service calls had a consistent overhead on external endpoints that shouldn’t have been there, and the traces pointed straight at DNS.
Kubernetes sets ndots:5 by default in /etc/resolv.conf inside pods. Any hostname with fewer than 5 dots gets the cluster search domains appended first, so when the app tried to resolve api.partner.com, the resolver first tried api.partner.com.default.svc.cluster.local, then .svc.cluster.local, then .cluster.local.
Some of those lookups resolved to stale records pointing at wrong endpoints. The app would connect, get a bad response or a timeout, retry, and only then fall back to the correct resolution. And because the DNS cache TTL on those records was effectively zero, it went through this whole dance on every single call.

It’s always DNS.
Once we set dnsConfig.options: [{name: ndots, value: "2"}] on the pods talking to external services and added trailing dots to FQDNs in the configuration, the stale resolutions stopped and the latency spikes disappeared with them.
System metrics (CPU, memory) tell you the server is on, application metrics tell you if it works, and tracing tells you why it doesn’t. You need all three: with just Prometheus dashboards, we would never have found the ndots issue.
Auto-scaling: metrics become actuators#
Back from the DNS detour. The bigger change in that story is that metrics stopped being something humans stare at and started being something systems consume.
The Kubernetes HPA was a first step: crude, but it established the pattern. Then came the Prometheus Adapter and KEDA, and suddenly you could scale on any metric: queue depth in RabbitMQ, lag in a Kafka consumer group, custom application latency. Whatever your application exposes, the system can react to it.
Same colors as before: blue is the metric, green is the system acting on it, and the dashed line closes the loop.
exposes metric"] --> P["Prometheus
scrapes"] P --> HPA["HPA / KEDA
evaluates target"] HPA --> SCALE["Scale decision
+/- pods"] SCALE --> SCHED["Scheduler
places pods"] SCHED --> STAB["Metric
stabilizes"] STAB -.->|loop| P classDef metric fill:#2d3128,stroke:#83a598,stroke-width:1.5px,color:#eceae5 classDef system fill:#343621,stroke:#a8c373,stroke-width:2px,color:#eceae5 class APP,P,STAB metric class SCALE,SCHED system linkStyle 5 stroke:#b8b5ad,stroke-width:1.5px,stroke-dasharray:5 4,color:#b8b5ad
Cloud environments have the same pattern under different names. Google Cloud has the HPA in GKE (using Cloud Monitoring or Managed Prometheus as signal source), custom metrics for MIG autoscaling, and Cloud Run scaling on request concurrency. Always the same loop: metric, evaluation, action, feedback.
At some point it clicks: monitoring is input to the control plane, not overhead. Without it, your HPA is a coin flip and your capacity planning is a spreadsheet someone updates quarterly.
I’ve had this conversation with enough customers at Google Cloud to know how it usually goes. The teams that instrument early build systems that adapt. The rest build systems that page people at 3 AM and then blame the cloud provider.
The number of times I’ve seen “we need to migrate to another cloud because this one is slow” when the actual problem was zero instrumentation and a misconfigured HPA is… concerning. That’s a rant for another article.
Intelligent routing: metrics drive every request#
The Gateway API Inference Extension is a CNCF project that extends Kubernetes gateways (Envoy Gateway, kgateway, GKE Gateway) into inference-aware routing proxies. It takes the same pattern one step further: instead of “should we add more pods?”, the question becomes “which pod should handle this request?”
Traditional load balancing doesn’t work for LLM serving, because round-robin across GPU endpoints ignores everything that matters. One GPU might have a full request queue while another is idle. One might already have the LoRA adapter for your fine-tuned model loaded in memory, while another would have to load it from scratch. One sits at 90% KV-cache utilization, another at 20%.
The Endpoint Picker queries real-time metrics from the model servers (blue) and passes each request through a chain of filters until it lands on an endpoint (green).
(Envoy / GKE)"] GW --> EPP["Endpoint Picker
queries model server metrics"] EPP --> F1["Filter: queue depth
is this GPU busy?"] F1 --> F2["Filter: KV-cache
memory headroom?"] F2 --> F3["Filter: LoRA affinity
adapter already loaded?"] F3 --> F4["Filter: criticality
production or batch?"] F4 --> GPU["Optimal GPU
endpoint"] classDef metric fill:#2d3128,stroke:#83a598,stroke-width:1.5px,color:#eceae5 classDef system fill:#343621,stroke:#a8c373,stroke-width:2px,color:#eceae5 class EPP metric class GPU system
The cascading filters are smart about priority. A critical production request can tolerate a queue depth up to 50 and higher KV-cache utilization, because it has to be served. A sheddable batch request gets strict thresholds: queue depth under 5, KV-cache under 80%. If nothing meets the criteria, the request gets shed rather than degrading everyone else’s experience.
Loading a fine-tuned LoRA adapter onto a GPU is expensive: it eats memory and takes time. If an endpoint already has the adapter loaded from a previous request, routing there avoids that overhead entirely. No traditional load balancer can do that, because it requires knowing what’s happening inside the model server.
That knowledge comes from the model servers themselves. vLLM and Triton expose metrics on a /metrics endpoint, and the Endpoint Picker polls them on a tight interval (50ms by default) and makes its routing decisions on that cached state. Same pattern as before, just running way faster.
What hasn’t changed#
Twenty years ago, metrics paged someone at 3 AM. Ten years ago, they scaled pods. Now they route individual inference requests to specific GPUs based on memory pressure and adapter affinity.
The pattern didn’t change, it just got applied to harder problems. Prometheus ended up giving us the common language that made this possible, and autoscalers, routing proxies and inference gateways all speak it now.
I still see teams treating monitoring as an afterthought. Deploy without instrumentation, set up HPA on CPU and call it a day, skip tracing because “we’ll add it later.” Then something breaks in production and nobody can explain why, because there’s nothing to look at.
The GPUs got faster and the workloads got weirder, but the job hasn’t changed: put a number on it, collect it, do something useful with it.
The opinions expressed here are my own and don’t represent my employer’s positions.