GPU Capacity Planning for SaaS With Spiky AI Demand

Traditional SaaS infrastructure scales on demand. A web server farm that handles 1,000 requests per second can provision additional servers in minutes when 2,000 requests per second arrives. The cost of additional capacity is modest, the provisioning time is short, and the marginal cost per request decreases as fixed infrastructure costs are amortized.

Self-hosted AI inference does not have these properties. A GPU inference server handling 100 requests per minute cannot be supplemented in minutes — GPU instances may have availability constraints, setup time is measured in tens of minutes, and the cost of a single GPU instance represents a meaningful capacity increment. Demand spikes that a traditional SaaS stack absorbs through automatic scaling will create inference queuing and latency degradation in a self-hosted AI stack unless the capacity architecture is designed specifically for spike behavior.

Why AI Demand Spikes Differently Than SaaS Traffic

SaaS traffic spikes correlate with product usage patterns: more users logging in, more API calls, more data fetched. These patterns are relatively predictable and scale linearly with user count.

AI inference demand has a different structure. The same user action can generate dramatically different inference loads depending on what the user is doing:

• A user asking "what is my account balance?" triggers one short inference call (low cost)
• A user asking "analyze all my transactions from the past year and identify spending anomalies" triggers multiple long-context inference calls with potentially hours of processing (high cost)

Both actions are one user action from the traffic perspective, but they represent 50–100× different inference demand. This non-linearity means that inference demand can spike even when the number of concurrent users is stable, if those users happen to be using high-inference features simultaneously.

The four demand spike patterns in AI SaaS each require different mitigation strategies:

Pattern 1: Time-of-day business hour concentration Enterprise AI products see 70–80% of their daily inference volume compressed into an 8–9 hour business day window. The spike factor (peak hour vs. off-peak hour demand) is typically 4–8×. This pattern is predictable and can be planned for; it requires provisioning capacity for the peak hour rate, not the average hour rate.

Pattern 2: Feature-triggered cascade spikes When a product feature triggers multiple sequential or parallel inference calls, each user action multiplies inference demand. A "comprehensive report" feature that calls the model 8 times generates 8× the inference of a "quick summary" feature. If a batch of users clicks the comprehensive report button simultaneously, inference demand spikes proportionally. These spikes are product-design-dependent and visible in per-feature usage analytics.

Pattern 3: Batch processing window concentration Scheduled jobs — nightly data refreshes, end-of-week reports, monthly compliance documents — create predictable demand spikes at specific times. When many customers' scheduled jobs overlap (all running at midnight, or all running on Monday morning), the combined spike can saturate capacity for a predictable window. Staggering batch processing windows across customer cohorts reduces peak concentration.

Pattern 4: Viral or event-driven surges A product mention in a major newsletter, a social media post by an influential user, or integration into a popular workflow tool can drive 5–20× normal traffic within hours. These surges are unpredictable in timing but not in character — the hybrid capacity architecture (base + on-demand + API fallback) is the only effective response to event-driven surges that exceed planning assumptions.

The Capacity Sizing Framework

Step 1: Build the Demand Distribution

Collect 60–90 days of hourly inference request counts and token throughput. Calculate the distribution of hourly demand:

• P50 (median) demand hour
• P90 demand hour
• P99 demand hour
• Peak (maximum observed) demand hour

The ratio of P99 to P50 indicates how spiky the demand pattern is. A ratio of 3× or less indicates relatively smooth demand; a ratio of 10× or more indicates severe spiking that requires careful capacity architecture.

Step 2: Model the Inference Cost Per Request

For each major feature type, calculate the average inference demand:

• Average input token count
• Average output token count
• Average number of model calls per user action
• Average time per model call

This enables translating user action counts into inference resource requirements: requests per minute × average tokens per request = throughput requirement.

Step 3: Calculate Capacity Requirements by Percentile

Map throughput requirements to GPU capacity:

Required GPU Count = Throughput Requirement (tokens/min) 
                   ÷ GPU Throughput Capacity (tokens/min per GPU)
                   × Safety Factor (1.2–1.3)

GPU throughput capacity depends on the model size, GPU type, and batch size. Benchmark this empirically with load tests on your specific model and GPU combination.

Calculate required GPU count at P50, P90, and P99 demand levels. The difference between P50 and P90 determines the burst capacity requirement; the difference between P90 and P99 determines how much of the load can be handled by on-demand provisioning vs. API fallback.

Step 4: Design the Three-Layer Architecture

Base capacity (reserved): Size for the P50–P70 demand level. Reserved instance pricing provides 40–60% discounts over on-demand for 1–3 year commitments. Base capacity handles the majority of typical demand at the lowest cost per hour.

Burst capacity (on-demand): Size for the P70–P95 demand level. On-demand GPU instances are more expensive per hour but can be provisioned in 5–15 minutes and released when no longer needed. Burst capacity handles predictable spikes — business hour peaks, batch processing windows — without the commitment cost of reserved instances.

API fallback: For demand above the P95 level — unpredictable surges, underestimated batch windows, product events. Configure the inference routing layer to fall back to an external API provider when the on-demand GPU pool is at capacity. API fallback is the most expensive per-inference option but provides an unconstrained capacity ceiling.

Practical Architecture: Implementing the Hybrid Model

The routing layer sits between the application layer and the inference infrastructure:

Application → Routing Layer → [Check capacity availability]
                              → Base GPU Pool (reserved)  
                              → Burst GPU Pool (on-demand)
                              → API Provider (fallback)

The routing decision logic:

def route_inference_request(request):
    base_pool_utilization = get_utilization(base_pool)
    burst_pool_utilization = get_utilization(burst_pool)
    
    if base_pool_utilization < 0.80:
        return route_to(base_pool, request)
    elif burst_pool_utilization < 0.80:
        # Auto-provision burst if pool is small
        if burst_pool.size < burst_pool.max_size:
            burst_pool.provision_additional()
        return route_to(burst_pool, request)
    else:
        return route_to(api_fallback, request)

The burst pool auto-provisioning trigger (provisioning additional capacity when utilization exceeds 80%) ensures that burst capacity grows ahead of demand rather than reacting to it — a 10-minute provisioning delay is too long to provision capacity in response to real-time demand; proactive provisioning avoids the queue.

Staggering Batch Jobs to Reduce Peak Concentration

Batch processing jobs from different customers create coordinated spikes when they run simultaneously. Staggering these jobs across a longer window reduces peak demand without changing total work:

Random jitter: Add a random offset (0–60 minutes) to each customer's batch job schedule. Simple to implement; reduces peak by 30–50% for typical batch windows.

Load-aware scheduling: Schedule batch jobs based on current inference capacity availability. Jobs waiting to run are queued and dispatched when capacity is below the 70th percentile utilization threshold.

Customer cohort staggering: Divide customers into cohorts and assign batch windows: Cohort A runs midnight–1am, Cohort B runs 1am–2am, etc. More predictable than random jitter; requires communication to customers about their batch window.

According to OpenView Partners' infrastructure benchmarks for AI SaaS, companies that implement batch job staggering reduce peak-to-average demand ratios by 35–55%, enabling smaller base capacity for the same throughput.

For the economic context of the self-hosting decision, see The Breakeven Math on Self-Hosting vs API Inference. For how inference costs appear in COGS, see AI-Native SaaS Gross Margin Decomposition.

Conclusion

GPU capacity planning for AI-native SaaS is a systems engineering problem that requires understanding both the technical properties of GPU inference and the behavioral patterns of AI product demand. The hybrid architecture — base reserved capacity, burst on-demand capacity, and API fallback — handles spiky demand economically by matching the cost structure of each capacity tier to the frequency and duration of the demand it handles.

The capacity sizing discipline — building the demand distribution, calculating requirements by percentile, sizing each tier appropriately — is not a one-time exercise. As the product and customer base evolve, the demand distribution changes. Quarterly capacity reviews ensure the architecture remains calibrated to actual demand patterns rather than assumptions that were correct at an earlier stage of growth.