Skip to main content
Updated September 6, 2026 · Worked example, not a measured benchmark GPU scale-to-zero saves money when the avoidable cost of keeping a replica warm exceeds the cost of restarting it, and the application can tolerate the resulting delay. Scaling the application to zero does not make the entire cloud account free. Cluster services, storage, networking, and other retained resources can continue to incur charges. The decision therefore has two tests: does it reduce the bill, and does the first request still meet the product’s requirements?

​
Start with the time between bursts

A daily average request rate hides the information needed for this decision. Two endpoints can receive the same number of requests while one is continuously busy and the other is idle for hours between bursts. Record the distribution of idle gaps, the cooldown before a replica is removed, and the number of restarts. Also measure startup with an empty cache and with a reusable cache. Image fetches, model loading, engine initialization, and instance provisioning can contribute differently to each restart. For a synchronous assistant, include the time a user’s request waits for capacity. An asynchronous job can tolerate a different startup budget; the job queues guide explains that execution model.

​
Calculate a simple break-even

For one idle gap, use these quantities: Under a simplified billing model:
The break-even expression assumes C is positive and the gap is long enough to trigger scale-down. R includes billed startup work and incremental data transfer. If you estimate it from a startup duration multiplied by a compute rate, do not add that compute cost a second time. Costs identical in both policies cancel in this comparison but still belong in the total service budget.

​
A worked example

Assume an avoidable warm-replica cost of 3.60perhour,afive−minutecooldown,and3.60 per hour, a five-minute cooldown, and 0.12 of incremental restart cost. These are hypothetical inputs, not current AWS or Tensorfuse prices. The cost per second is 0.001.Thebreak−evenidlegapistherefore300+0.12/0.001=420seconds,orsevenminutes.Fora20−minuteidlegap,thesavingis0.001. The break-even idle gap is therefore 300 + 0.12 / 0.001 = 420 seconds, or seven minutes. For a 20-minute idle gap, the saving is 0.001 × 900 − 0.12=0.12 = 0.78. This calculation assumes scale-down actually removes the billed capacity. If a GPU node remains allocated for another workload, removing one pod might not reduce the bill at all. Billing minimums and capacity sharing can also make the cost discontinuous. Measure avoidable cost at the resource that is billed.

​
Apply the latency constraint separately

A financially attractive restart can still violate the application deadline. Suppose the measured time to a usable response after scale-up is 45 seconds, while the product requires a first token within two seconds. Positive savings do not make that policy suitable for the interactive path. Consider retaining a warm replica, using a predictable prewarming schedule, or moving delay-tolerant work to a queue. Benchmark those choices with the same traffic trace. A readiness check should report whether the model server can accept work; starting a container process is an earlier milestone. Kubernetes distinguishes startup and readiness probes for these different lifecycle checks.

​
Apply this to Tensorfuse

The deployment configuration documents min_scale, max_scale, and concurrency. A minimum of zero permits scale-to-zero; a nonzero minimum retains replicas. Measure the resulting resource lifetime and bill before assuming a particular savings percentage. Use the cold-start guide to investigate startup phases, and read why lazy loading can still delay the first request. Evaluate the final policy using cost per successful request, including requests that time out during restarts.