FORGE did not begin with a search for the most powerful cluster. It began with a recurring delivery problem: a scheduled ML pipeline, a memory-sensitive API, an edge workload, and a conventional service all needed reliable infrastructure, promotion, rollback, and observability, but they did not need the same runtime.
How could those workloads share an operating model without forcing each of them into Kubernetes? FORGE answers with contracts around infrastructure intent, immutable artifacts, reviewed deployment state, health, and operator feedback. EKS is one implementation surface inside that system, alongside managed application and edge runtimes.
Three facts kept the design grounded. Demand is currently concentrated across the eastern and southern United States, making a compact regional foundation reasonable. The product is growing quickly, so online services and deadline-bound workflows need independent room to scale. And because most surrounding services already run on AWS, reusing its identity, delivery, networking, metrics, and cost controls is simpler than introducing another hosting control plane.
The AWS Well-Architected Framework provides useful external vocabulary: reliability, security, performance efficiency, cost optimization, and operational excellence have to be reviewed together. FORGE turns those qualities into repeatable contracts rather than treating any one runtime as evidence of platform maturity.
Standardize contracts, not every runtime
What should these workloads actually share? The responsibilities that operators repeat, not necessarily the scheduler that runs them.
FORGE defines six shared contracts: infrastructure declaration, artifact production, deployment-state review, reconciliation, health and metrics, and operator notification. A workload can implement those contracts as a scheduled Kubernetes job, a memory-sensitive API, an edge component, or a conventional managed service.
This keeps EKS in its proper place: useful where Kubernetes scheduling and operations create leverage, but not the definition of the platform. A new workload can inherit delivery and operational conventions without inheriting infrastructure it does not need.
That established the unit of reuse. The next problem was separating changes that should not share a release boundary.
Separate what changes at different speeds
Why did a small release sometimes demand a large review? Application source and environment selection were changing in the same place.
| Plane | Owns | Changes when |
|---|---|---|
| Infrastructure intent | Network, compute, identity, shared services | Platform requirements change |
| Application artifact | Tested executable or container | Application code changes |
| Deployment state | Environment-selected artifact and configuration | Release promotion or rollback |
The useful move was to separate three things that change for different reasons. Infrastructure changes when the platform needs a new capability. Application artifacts change when code changes. Deployment state changes when an environment promotes or rolls back a tested artifact. Once those concerns stopped sharing one change surface, a release became easier to read.
Terraform declares cloud infrastructure, OIDC-based CI identity avoids long-lived build credentials, and managed builds publish immutable artifacts. A dedicated state repository records what each environment should run; Argo CD reconciles Kubernetes workloads toward that reviewed intent. This follows the core SRE release-engineering principle that reproducible construction and consistent rollout are properties of the release system, not operator memory.
The review questions become narrower: “What does the software do?” belongs to the application change; “What should this environment run?” belongs to deployment state. Promotion copies a tested choice forward, and rollback selects a previous known artifact. Git records intent and history, while runtime checks still verify whether reconciliation actually succeeded.
The separation also reduces environment-specific delivery logic. Build once, select the artifact through reviewed state, and let the reconciler converge each environment on that selection.
A single repository reduces coordination overhead and can be the right starting point. It became the weaker boundary once application changes and environment promotion needed different review, rollback, and access semantics. A dedicated state repository adds coordination, but makes an environment selection a small, explicit change. Versioned contracts and state-update automation keep that separation from moving coupling into operator procedure.
Kubernetes was a portfolio decision
Why operate Kubernetes at all when AWS already offers simpler hosting products? The answer has to come from the workload portfolio, not a preference for one control plane.
| Runtime | Best fit | Limitation for this portfolio |
|---|---|---|
| Lambda | Sporadic, bounded control work | Poor fit for persistent state or long processing |
| ECS/Fargate | Isolated long-running services | Less reuse of the established Kubernetes operating model |
| Managed AI workflow or hosting | Standard training and model-endpoint lifecycles | A narrower control plane for a portfolio that also includes custom services and edge delivery |
| EKS | Mixed services, scheduled jobs, specialized compute | Higher platform and upgrade burden |
| Managed application runtime | Conventional services needing little control | Less suitable for specialized scheduling |
Lambda remains useful for bounded control work within its 15-minute invocation limit. ECS with Fargate is viable for isolated long-running containers, but it is not the Recsys compute model. A conventional managed application runtime can be simpler still.
Managed AI platforms are attractive for a standard train-and-host loop; SageMaker AI, for example, manages training infrastructure and model hosting (AWS documentation). FORGE also had to deliver custom pipelines, a Go decision service, experiment integration, edge artifacts, and conventional applications. A separate ML-only lifecycle would have duplicated promotion, identity, and observability contracts.
EKS became reasonable because scheduled jobs, always-on services, specialized placement, GitOps, and observability could amortize one operating model and integrate with the existing AWS estate. Its control-plane fee, worker costs, and upgrade burden remain real. AWS's container decision guide likewise treats runtime selection as a workload decision, not a hierarchy.
Processor choice follows the same rule. Compatible edge workloads use Arm64, where the runtime and dependency chain support it; general-purpose and accelerator-backed pools keep the architecture their workloads require. Cost efficiency is evaluated as effective compute cost divided by measured completed work. AWS provides Graviton migration and price-performance guidance for that benchmark-driven choice (AWS Graviton resources).
The resulting rule is intentionally reversible: use Kubernetes while portfolio reuse exceeds its operating cost; choose a managed runtime whenever it does not.
Runtime selection answers where a workload runs. Environment policy and scaling determine whether it remains controlled as demand changes.
Environment isolation is a policy, not a diagram
Environment isolation is easier to reason about when identity, reachability, data, and operator access are reviewed as separate controls.
Compute operates within controlled cloud-network boundaries. Production minimizes externally reachable surface and operational access. Development permits controlled integration access where required, with authentication, isolation, non-production data, and monitoring. Credentials and deployment permissions are scoped by environment.
Public reachability, application authorization, and control-plane access are different concerns. A service can have a public entry point and still enforce application identity; a private network path does not remove the need for authorization. Likewise, deployment configuration expresses intended controls, while runtime evidence is still needed to verify that the intended boundary holds.
This prevents two common shortcuts: treating a private path as authorization, or allowing a development integration need to become a production default.
The same reasoning applies to region count. A second active region could improve latency for a dispersed audience and reduce dependence on one regional failure domain, but it would duplicate compute and state, complicate data consistency and release promotion, and require continuously tested failover. Concentrated demand does not justify that operating tax today. Meaningful geographic demand shift, data-residency requirements, latency objectives, or recovery targets that one region cannot meet would change the decision.
Autoscaling has two control loops
What does “autoscaling” mean when adding a node does not decide how many application replicas should exist? There are two control loops, and only one is automated here today.
Scheduled orchestration creates bounded pods with task-specific resource requests, while reviewed deployment state sets replica intent for long-running services. A general HPA or KEDA policy is not part of the current serving state: node autoscaling does not imply application autoscaling. Kubernetes documents the same separation between horizontal workload scaling and node provisioning for unschedulable pods.
Fixed replica intent is defensible for the current long-running service because an availability floor and measured memory-sensitive capacity dominate a relatively compact demand envelope. It also makes snapshot duplication and rollout headroom predictable. The trade-off is reserved headroom and a reviewed rather than automatic response to sustained demand. If request variability, queueing, freshness, or tail latency begins to consume that margin, the next controller should scale on the signal that represents work; CPU alone is often weak for a memory-bound service.
The infrastructure loop adds allocatable capacity when desired units cannot be placed. A launched machine is not yet a ready workload: image pull, addressing, volume attachment, topology, and initialization still precede scheduling convergence.
Placement is limited by the tightest resource, not aggregate utilization. Memory after rollout headroom, specialized hardware, topology, address capacity, and scale-up delay each impose independent bounds.
The Recsys cost case study shows how local and managed state change the application-capacity boundary. FORGE owns the controller interaction and shared placement boundary.
Three failures became platform contracts
The platform became more coherent when failures stopped being treated as isolated fixes and started changing shared contracts. Three examples mattered most.
Memory pressure
The first instinct was to give different ML stages a uniform resource shape. The workloads disagreed: encoding, evaluation, ranking, and orchestration each reached peak memory differently. Streaming large data, chunking evaluation, reusing cached encodings, limiting work to the intended scope, and recording resident-memory checkpoints made both failures and costs easier to explain. The shared lesson was task-aware sizing, not a larger default pod.
Network-address pressure
A scale-up once looked successful from the compute side while placement was still blocked by network-address availability. That exposed a blind spot: an instance can exist without making a pod schedulable. Address capacity now sits beside CPU, memory, and specialized hardware in capacity reviews, and diagnostics distinguish “instance launched” from “pod network ready.”
Ambiguous job outcomes
A quiet input window revealed a semantic problem. If success is inferred from output volume, a valid no-work run looks much like a broken job. Pipeline contracts now distinguish success, skip, and failure. Deliberate no-work can be reported without a false alarm, while malformed or missing required input still fails closed.
Together they changed the platform contract: size by task, model addresses explicitly, and make terminal state semantic.
An alert is part of the system interface
When does a metric become operational? When it reaches an owner with enough context to decide what to do next.
FORGE uses Prometheus-compatible collection, managed metric storage, Grafana, alert policy, and Slack notification. Workloads emit application meaning; the platform makes those signals queryable and routable.
CloudWatch remains the natural source for AWS service alarms, but using it as the only application metric model would split Kubernetes workloads from the Prometheus instrumentation already common in their libraries and dashboards. Fully self-hosting Prometheus would preserve that model while adding storage durability, scaling, and upgrade work to the platform. Prometheus-compatible collection with managed storage is the middle ground: one workload-facing metric language and less stateful monitoring infrastructure, at the cost of another managed service and a deliberate bridge to native AWS alarms.
The metric contract spans request behavior, artifact freshness, batch duration and resident memory, restarts, scheduling pressure, autoscaling, and reconciliation health. Workloads define meaning; the platform supplies collection, retention, dashboards, and delivery.
An actionable notification identifies impact, ownership, diagnostic context, and the next action. Rules also need volume guards, time windows, and resolved behavior: a percentage without a denominator can page on one event, while a firing notification without resolution leaves recovery ambiguous.
Cost case study: route sparse alerts on demand
One modest build-versus-buy decision illustrates the platform's cost discipline. The managed-alert path uses a small Lambda formatter between SNS and Slack rather than an always-on notification service. It handles both Prometheus Alertmanager and CloudWatch alarm payloads with 256 MB of memory, a 10-second failure ceiling, and 30 days of log retention.
The buy-versus-build reference was a fixed $25 per month alert-to-Slack subscription. At the configured memory and maximum duration, Lambda compute alone would need roughly 600,000 fully billed, timeout-length invocations to reach $25. The timeout is a failure ceiling, not an assumed average duration. This compares only transport and formatting. A subscription that also provides on-call schedules, escalation, or incident management is a different product.
The complete path also includes SNS, secret retrieval, and log ingestion or storage. SNS is request-priced with no minimum commitment, so those terms remain volume-driven too. Cost controls in the implementation are simple: Alertmanager groups related events, inhibits a warning while the matching critical alert is active, and limits repeat notifications; scheduled Airflow runs suppress routine success messages; and log retention is bounded. These choices reduce both spend and alert fatigue without removing resolved-state delivery.
One platform, different runtime shapes
Reuse is visible in shared contracts, while each workload preserves its natural execution model.
| Workload | Character | Reused platform capability |
|---|---|---|
| Recsys learning | Scheduled, artifact-producing ML | Reproducible orchestration and resource isolation |
| Recsys serving | Always-on and memory-sensitive | Controlled rollout, health, scaling, monitoring |
| Presort/computer vision | Cloud-to-edge lifecycle | Artifact delivery and remote observability |
| Translation | Conventional managed service | Simpler runtime with shared delivery standards |
| Model development | Intermittent tracking and human annotation | Optional workspace with durable artifacts and scoped access |
The Treverse Recsys case study is the most detailed consumer. Presort adds edge artifact lifecycle and remote health; translation shows that a conventional service can reuse delivery standards without joining Kubernetes.
Tracking and annotation are deliberately optional runtime shapes. Experiment artifacts and reviewed labels persist in platform storage, while interactive services and their backing resources can be disabled between active experiments or annotation campaigns. This avoids paying idle compute for tools that are valuable during development but unnecessary on the production request path.
This is the platform test that matters: a new shape should reuse the contracts and only introduce the runtime machinery its workload requires.
Keep the platform reversible
The next improvements reduce operator inference and environment-specific logic.
The platform needs more application-aware scaling, better capacity forecasting, less environment-specific deployment logic, clearer reusable workload contracts, and stronger automated deployment verification. Geographic concentration makes a compact foundation reasonable now. The same contracts can reduce application-specific work in a future regional expansion, but state replication, traffic policy, failure semantics, and recovery testing would still be new engineering, not a configuration-only change.
Managed runtimes should remain preferable whenever Kubernetes adds no material value. FORGE succeeds when the next workload is cheaper to reason about, not merely easier to place on EKS, and when its most complex runtime remains optional.