Three ongoing cycles
Forecasting, evaluation, and decision refresh have different cadences. A module's interval controls new forecasts, evaluations wait for target times, and the Operator keeps asking what replicas should do now.
The sample interval determines time per observation; the history window determines how far back a method sees; the horizon determines how far forward it predicts; cooldown limits action frequency. These are distinct settings. Observe mode stops at collection and forecasting without producing execution recommendations.
What happens in a forecast cycle
- Select due modules. The task center checks enabled modules and respects individual run intervals.
- Prepare history. Render metric queries for the workload, query Prometheus, align units and grids, and record gaps and freshness.
- Forecast. Disabled or No prediction uses defaults. Fixed mode runs one selected method; dynamic mode runs centrally enabled methods. Insufficient-history and failed candidates record reasons.
- Produce advice. Eligible forecasts enter resource and replica calculations with schedules, bounds, look-ahead consensus, and safeguards.
- Store results. Save forecasts, run information, and versioned, expiring recommendations for views, admission, and the Operator. Observe mode performs no resource changes.
Baselines remain eligible. Candidates need promotion for the corresponding version. Parameter changes can alter model identity, so previous promotion does not automatically qualify a new configuration.
Why ask for replicas ahead of demand?
New Pods take time to become available. The platform uses measured or configured readiness time, decision wait, and recent lateness allowance to establish a lead window, then combines the available methods' required replicas in that window.
Scale-up considers demand within the lead window. Scale-down requires stronger agreement over the whole horizon and current demand fitting into fewer replicas. Observed load above forecasts and existing capacity may trigger emergency scale-up, still within replica ceilings.
Follow load between forecasts
When the Operator polls, the platform checks decision freshness. Refresh reuses fresh stored forecasts, collects current observations, and recomputes advice. Algorithms run again only when the forecasts themselves are also stale.
Refresh is serialized per module. Requests wait for a configured period while background refresh may continue, letting later polls read the new version. Modules with individual intervals retain their own cadence.
Advice does not immediately become capacity
For example, a workload has four replicas and advice asks for ten. With a two-minute cooldown and a maximum increase of one, it may hold during cooldown and first move to five once execution is allowed. These are example settings, not universal defaults.
The Operator also checks target, permissions, conflicts, readiness, and bounds. Emergency scale-up may bypass ordinary upward cooldown and step limits; scale-down never takes that path. Successful execution means settings were submitted, while Ready replicas establish available capacity.
Where safeguards apply
| Mechanism | Effect |
|---|---|
| OOM score weighting | When enabled, penalizes memory scores in underallocated evaluation windows, affecting comparison rather than directly lowering memory forecasts. |
| OOM memory floor | Uses recent OOM events and memory observations to prevent overly low advice. |
| JVM heap pressure | Contributes a replica constraint when configured with the necessary heap metrics. |
| Data or model degradation | Without eligible forecasts, follows schedules, fallback, or unchanged-value rules and records the chosen source. |
| Platform outage | The Operator uses cache, hold, degradation, and stable recovery states, distinct from an individual model failure. |
How outcomes inform the next cycle
Once forecast target times arrive, actuals are aligned to compute resource error, peak underestimation, and coverage and update scores and rankings. Rolling backtests compare candidates with baselines for eligibility; actual-value evaluation explains predictions that have already occurred.
Scale-up outcomes compare Ready replicas and later demand to assess timeliness. Evaluated readiness and lateness inform later lead windows. Effect reports summarize accuracy, stability, Pod-hours, and capacity SLO.