Treverse Recsys: Temporal Recommendations from Learning to Serving

What should a recommender learn when an item may leave the auction before its behavioral history becomes useful?

That question shaped Treverse Recsys. Newly eligible items arrive cold, older items disappear from the candidate set, and user histories range from rich to nearly absent. The clock changes both what the model can know and what it is still allowed to recommend.

The system answers by learning from time-safe relational evidence, doing expensive ranking work offline, and publishing a verified decision artifact. The online service then has a smaller job: load a complete release, choose the correct recommendation path, record exposure, and fail predictably.

Five review questions recur throughout the design. They turn broad production ideals, such as the AWS Well-Architected pillars and Google SRE's service-objective discipline, into checks that can change an implementation.

Quality	Question used in this system
Correctness	Does learning and evaluation preserve time and eligibility?
Consistency	Can a request observe only a complete, identified release?
Availability	Does failure retain a useful path and a last-known-good state?
Efficiency	Does the complete architecture meet the objective at the lowest justified complexity and cost?
Evidence	Is the decision supported by a reproducible evaluation, load gate, or controlled experiment?

Time-bounded inventory weakens ID-only learning

What changes when yesterday's useful item may not be eligible tomorrow? Cold start becomes the normal case, not an exception.

An ID-only recommender gets stronger when the same users and items repeat. Here, fresh items arrive before they can build much behavior, older items leave quickly, and user histories range from rich to nearly absent. Behavioral signals can also arrive at different speeds: a shallow action appears immediately, while a completed outcome may settle much later.

That creates three requirements.

1. The model must represent unseen inventory from context rather than waiting for an item ID to become warm.
2. Evaluation must preserve time direction so future interactions cannot leak into earlier decisions.
3. Online serving must keep a predictable request path without making neural inference another request-time dependency.

The product objective matters just as much. An item open is useful diagnostic evidence, but it is not interchangeable with a bid, checkout, or realized marketplace value. Optimizing the easiest action would have produced a cleaner dashboard and potentially a weaker business result.

These were not independent requirements. A model that handles cold items but leaks future behavior is invalid; a time-safe model that makes every request wait on neural inference is operationally fragile; and a fast system optimized only for opens can still move the marketplace in the wrong direction. The design had to satisfy all three together.

Choosing a temporal graph for the shape of the problem

Which model family can use new-item context without flattening the timing and relational meaning of auction activity?

Approach	Strength	Pressure in this domain	Role
Popularity	Simple, robust, explainable	Limited personalization	Baseline and deterministic fallback
Matrix factorization or collaborative filtering	Strong repeated identity signal	Cold items have weak latent histories	Benchmark where inputs are reproducible
Two-tower retrieval	Inductive, ANN-friendly, and able to encode temporal summaries	Pair-specific event context must be compressed before scoring	Complementary reference
Sequential modeling	Ordered behavior	Histories can be short or irregular	Useful where sequence depth supports it
Temporal graph retrieval	Relations, recency, and inductive context	More complex training and evaluation	Chosen retrieval family

Classic matrix factorization remains a useful reference when repeated identities carry stable signal, but a new item has no learned latent history. A two-tower candidate generator, as in the YouTube recommendation architecture, is a stronger inductive reference because both towers can consume content and temporal summaries.

Two-tower retrieval does not inherently omit time or interaction types. Its constraint is separable scoring: pair-specific and neighborhood-specific evidence must be compressed into independent user and item representations before similarity is computed. That matters for auctions, where a bid is a timestamped relationship between a participant and an evolving item, not merely metadata or a generic click. A temporal graph supplies a direct inductive bias for those relations. Sequential or two-tower models remain credible when compressed history preserves the needed signal; the graph earns its extra complexity only while time-safe challenger results justify it.

The implemented retriever is GraphSAGE-style rather than a literal reproduction of one paper. GraphSAGE supplies the key inductive idea: learn an aggregation function instead of one embedding per permanent node. The model combines content encoders with edge-aware neighborhood aggregation, filters sampled history by the query time, and represents recency and interaction strength in edge context. A learned gate can lean toward content when graph evidence is weak. That behavior matters for both new inventory and sparse users.

Training uses contrastive retrieval with in-batch negatives plus a hard-negative term. Evaluation can use seeded sampled negatives or exact ranking within the eligible candidate scope. Exact filtered retrieval remains the correctness reference. Approximate search is an optimization only after measured need; methods such as ScaNN trade exactness for search efficiency and therefore require recall and latency validation against that reference.

The experimented model learned from high-confidence completed outcomes and used item context when behavioral history was thin. The data contract can also represent context-rich bids and other pre-conversion events, but those signals belong to a later challenger. Keeping that boundary explicit matters: the production result later in this article belongs to the model that was actually tested, not to the more ambitious system it could become.

Move intelligence off the hot path

Once the model became more capable, the next question was where its complexity should run. A recommendation request should consume a decision, not recreate the learning system.

Databricks participates upstream in curation and validation. A versioned source contract separates that preparation from downstream model ownership. Scheduled orchestration then constructs features, trains or refreshes a candidate, evaluates the eligible release, and materializes ranked output. The serving process consumes the immutable ranked artifact.

This makes the deployed interface deliberately narrow. Live events inform measurement and later learning, but they do not mutate the active model during a request. A failed training job cannot partially update serving state, and a transient event problem cannot silently rewrite the recommendation function.

Online neural ranking would provide fresher request context and avoid materializing every decision in advance. It would also put model latency, model-runtime capacity, and partial rollout behavior on the availability path. The current candidate scope and release cadence make precomputation the better trade: the system spends compute once per release and makes request behavior deterministic. This boundary should move if request-time context produces a repeatable gain that snapshots cannot express, or if artifact freshness misses the product objective.

FORGE owns the reusable orchestration, deployment-state, scaling, and observability contracts that make this loop operational. This article owns the workload-specific modeling and release semantics.

A model release is also a data release

Reproducibility ends too early if it stops at model weights; the served decision also depends on features, eligibility, and ranked output.

The source bundle and every major artifact carry versioned contracts. Deterministic inputs and seeded evaluation make reruns comparable. Training checkpoints preserve execution state, including random number state, while evaluation rejects incompatible scope. The same eligibility definition must apply to champion and challenger; otherwise an apparent gain may only be a candidate-set mismatch.

Most releases do not need to begin with retraining. If the champion remains compatible, refreshing ranks captures catalog change without paying training cost or introducing avoidable representation variance. The trade-off is model staleness, so aligned evaluation decides when refresh is no longer enough. A challenger then has to pass quality, coverage, and scope gates. Failure keeps the champion or, if materialization itself fails, the last-known-good release.

A model registry alone would not close this loop. The served behavior also depends on the eligible candidate scope, encoded features, and materialized ranks. Release identity therefore binds those data decisions to the model instead of treating weights as the deployable unit. This is the practical lesson behind hidden technical debt in ML systems: model behavior lives in data and pipeline dependencies as well as model code.

The local runner executes the production task order with the same artifact contracts. That gives operators a reproducible way to replay a bounded release path without treating the orchestrator UI as the only source of truth.

Several implementation details made this loop practical:

• reuse cached encodings and skip encoder work on complete cache hits;
• evaluate large candidate sets in chunks;
• restrict ranking to the intended release scope;
• record stage-level resident memory checkpoints;
• stream or reference large intermediate data instead of duplicating it in orchestration metadata.

The FORGE deployment-state model explains how a tested application artifact and reviewed environment intent move independently. Here, the result is an immutable release identity that binds model behavior to the data decision actually served.

The smallest complete decision service

How can a new release become visible without exposing a half-loaded state? Build it away from readers, validate it, then publish it as one unit.

The service resolves a versioned release, streams the snapshot into replacement maps, validates required metadata and fallback material, and derives compact serving indexes. None of those structures are visible to requests while they are being assembled.

Publication happens inside one write-locked critical section. The current implementation replaces the related fields together under that lock; it does not literally exchange one atomic pointer. Readers therefore observe the old complete state or the new complete state, never a partially populated mixture.

That code-level distinction is the operational guarantee:

resolve versioned release
          ↓
build warming state ── validation failure ─→ discard
          ↓
publish under one write lock
          ↓
old state becomes collectible

If refresh fails after a valid artifact has loaded, readiness remains true and health reports stale state. If the release identity is unchanged, the service skips the snapshot reload. Liveness answers whether the process can run; readiness answers whether at least one useful serving path is available. A deterministic fallback covers missing personalized state, and its use is observable rather than silent.

The implementation favors a small, auditable decision path over additional request-time complexity. One exclusive mutex is a deliberate simplicity trade: multi-map publication is straightforward to verify, but readers can contend with one another as well as with refresh. A read/write lock, immutable aggregate behind an atomic pointer, or sharded maps would be justified by measured lock wait or tail-latency pressure. Before that evidence appears, each alternative enlarges the concurrency proof without improving product behavior.

That kept the request path small. It also made the next decision unavoidable: whether paying for duplicated memory in two replicas was still cheaper than operating shared state.

Cost case study: lower serving cost by keeping immutable state local

The tempting comparison was process memory versus Redis. The useful comparison was the complete service architecture that each option left behind.

Moving ranked rows into a managed cache would not remove release selection, experiment assignment, exposure commit, validation, fallback, or downstream assembly. It would relocate immutable state while leaving the decision service and its availability replicas in place. The comparison therefore asks about the complete architecture: refresh memory, demand, and the worker capacity a remote store can genuinely remove.

The worked example uses round, replaceable assumptions. Change active users, actions, peak factor, state size, latency objective, or worker shape, and the conclusion must be recalculated.

Size the refresh peak, not the artifact alone

The replacement snapshot makes refresh peak, not steady-state size, the memory constraint. In the worked example, a 16 GiB serving unit is the complete capacity envelope, not 16 GiB of cacheable rows. Runtime, profile state, active and warming snapshots, allocator behavior, and headroom all share it. Two such units provide 32 GiB of raw capacity; one temporary rollout unit raises the placement envelope to 48 GiB.

Turn product demand into a one-replica test

Total users describe the addressable population, but DAU drives this request estimate. The model produces the test target; it does not manufacture a throughput result. The 60% ceiling reserves 40% of tested capacity beyond the modeled 10x peak and should follow the measured saturation knee. The p99 multiplication is a conservative in-flight proxy; Little's Law formally uses mean latency. It does not imply 1 ÷ 0.250 = 4 TPS, because the Go server processes concurrent requests.

Rounded dev microbenchmark: a short synthetic run against the two-replica inference path sustained low hundreds of requests per second with p99 below 100 ms and no HTTP failures at moderate concurrency. Experiment exposure was bypassed, so this tests load balancing, routing, and in-memory lookup, not the complete gateway and downstream path. Its purpose is to show margin over the worked 48.23 QPS gate, not define production capacity.

Two replicas also establish an availability floor when topology placement keeps them on different nodes. One node can disappear while the other continues serving; one larger node cannot provide that failure boundary. The benefit is conditional on enforced placement, readiness, disruption handling, and the surviving replica passing the one-replica load gate.

Compare complete architectures

The comparison keeps two service replicas in every design and gives the managed-cache alternatives smaller workers. This credits Redis or Valkey for memory they genuinely remove without pretending either product replaces the decision service.

Rates were checked for US East (N. Virginia) on July 15, 2026 from AWS ElastiCache pricing and EC2 On-Demand pricing. The worked sensitivity uses 730 hours, 7.5 million reads, 6.5 GB of billable state, and one ECPU per read. Unit rates are $0.1632/hour for m7g.xlarge, $0.0816/hour for m7g.large, $0.084/GB-hour + $0.0023/million ECPUs for Valkey, and $0.125/GB-hour + $0.0034/million ECPUs for Redis OSS.

Worked-example architecture	Service-side capacity assumption	State assumption	Monthly estimate
Local maps on larger workers	8 vCPUs, 32 GiB raw across two m7g.xlarge workers	Replica-local	$238.27
Smaller workers plus Valkey	4 vCPUs, 16 GiB raw across two m7g.large workers	6.5 GB Valkey Serverless	$119.14 + $398.60 = $517.74
Smaller workers plus Redis OSS	4 vCPUs, 16 GiB raw across two m7g.large workers	6.5 GB Redis OSS Serverless	$119.14 + $593.15 = $712.29

Under these assumptions, both managed-cache options cost more while halving service CPU and adding a network dependency. The listed-cost break-even is approximately 1.94 GB for Valkey and 1.31 GB for Redis OSS; above it, cache storage consumes the entire $119.13 worker-downsize saving. Even below that boundary, the smaller worker must pass the same 48.23 QPS, 250 ms p99, and failover gate. When an existing node has enough unallocated memory for both active and warming state, local maps add no instance charge until the next node-size or node-count step.

S3 Standard pricing puts the same worked example at about $3.15/month, but S3 is not latency-equivalent: every decision would add object retrieval and decoding. It remains the release source, loaded once and served many times. PostgreSQL becomes relevant when transactions, joins, or durable mutation are required; Redis or Valkey becomes relevant when shared state or independent cache scaling is required.

The lesson is not “memory is always better than Redis.” It is to price the complete architecture, locate the break-even point, and test the smaller service shape before treating managed state as a saving. At the worked inputs, immutable point lookup and experiment-aware application logic favor local state. If one product open produces multiple recommendation requests, Q_peak scales by that fan-out and the conclusion must be tested again.

Define the migration trigger before the migration

A good local-state decision includes a written path for the day it stops being good.

Condition	Preferred direction	Reason
Immutable point lookup fits safely	Local maps	Lowest latency and smallest failure surface
State grows but keys partition cleanly	Sharded serving	Preserve local access while distributing memory
Replica duplication dominates or cross-replica coordination becomes necessary	Managed key-value state	Scale shared state independently while retaining the decision router
Transactions and joins dominate	Postgres	Consistency, auditability, and inspection
Search, filters, or vectors become request-time requirements	OpenSearch	Query capability rather than generic caching

Object storage remains the versioned source of truth in each case. The serving technology is a materialization choice, not the authority for a release.

Migration signals are operational: refresh peaks approach safe headroom, memory restarts recur, load time exceeds the rollout budget, tail latency crosses its objective, duplicated state costs more than the complete cost of a shared-store alternative, or point lookup stops matching the product query. The complete comparison retains router compute, experiment calls, request charges, transfer, observability, and the new failure mode introduced by a remote dependency.

With the serving boundary justified, the remaining test was whether the system improved the marketplace outcome it was built for.

Fewer opens, stronger downstream outcomes

What should the team conclude when the original engagement metric falls, but bidding, checkout, and marketplace value rise?

Metric	Relative change	95% confidence interval
Marketplace sales per exposed buyer	+12.62%	+5.67% to +20.03%
Checkout-user rate	+3.21%	+0.65% to +5.83%
Bid-action users	+2.13%	+0.57% to +3.72%
Item-open users	-4.56%	-7.64% to -1.39%

The result was not uniformly positive across the funnel. Item open was the original primary metric, and it declined. The downstream scorecard changed the business interpretation, not that historical designation. One plausible explanation is that the treatment removed curiosity clicks and helped buyers reach actionable inventory with fewer opens. The experiment is consistent with that mechanism; it does not prove every lost open was low value.

Assignment and exposure were designed to make the comparison trustworthy. The server selected a variant, routed the request, and committed exposure only after the inference service finalized its route. Final item assembly can still fall back downstream, so the contract is closer to decision delivery than a guarantee that every ranked item rendered.

The analysis used CUPED-adjusted effects and 95% confidence intervals. Holm correction was applied across the five-metric family: value, checkout, bid action, and item open remained below the adjusted threshold, while another secondary action did not. Random assignment supports a causal comparison between these tested arms, but it does not identify the mechanism, guarantee the same effect in every segment, or validate later model changes.

The experiment predates the later cold-start policy, reranker, and broader intent-history challenger. None of those additions can claim credit for this result; each needs its own controlled comparison. A trustworthy aggregate result still needs a way to investigate individual decisions.

Make the embedding space discussable, not definitive

How can engineers and leadership inspect embedding retrieval without turning a projection into causal proof?

Three internal tools answer different questions. A diagnostic board checks whether a case received the expected artifact. An experiment harness checks assignment and exposure interpretation. A profile explorer makes embedding-based retrieval inspectable to both engineers and leadership: it connects an abstract vector space to the practical observation that nearby profiles often receive similar recommendations while more separated profiles tend to diverge.

The coordinates are deterministic for a fixed artifact, which makes before-and-after investigation possible. They are a dimensionality-reduced view, not the retrieval space itself. Projection preserves useful structure only approximately: nearby points are candidates for similar high-dimensional geometry, but distant dimensions can collapse together and visible distance can be distorted. Recommendation comparison remains the check on whether two projected profiles actually behave alike.

That qualification is important in leadership communication. The map can explain that the system learns a continuous similarity structure rather than a collection of hand-named segments. It cannot prove that a visible cluster is causal, semantically correct, or stable under intervention.

Application signals cover request duration and errors, artifact freshness, refresh outcome, fallback use, and resident-memory trends. FORGE owns collection, dashboards, alert policy, and operator delivery; the application owns the meaning of those signals.

Inspectability earns trust when it shortens a concrete diagnostic loop, not when it turns latent space into decoration.

The next measured decisions

The roadmap is a sequence of hypotheses with triggers, not a promise to add complexity.

The post-experiment baseline now includes a more deliberate cold-start policy and constrained reranking. Neither is part of the result above. The next controlled decision is to test that later baseline while keeping broader time-aware intent in a separately attributable challenger. Each comparison should preserve the exposure and release contracts so its effect can be isolated.

Approximate nearest-neighbor retrieval belongs only where exact search is measured as a bottleneck. Partitioning or managed state belongs only after memory, load-time, query, or cost triggers are crossed. The standard is not maximum sophistication; it is a system whose correctness can be replayed, availability tested, cost recomputed, and product effect measured again.