Cartridge Economics & Break-even

When does the one-time cost of generating and training reusable per-document KV-cache Cartridges, plus persistent storage and loading, become cheaper than repeatedly prefilling conventional RAG text? Inspired by Cartridges at Scale (Amazon AGI Lab, arXiv:2606.04557) .

Workload

Source documents

Total source tokens

Avg source tokens/document (derived, overridable)

Compression ratio

Model (BF16 KV MiB per 1K Cartridge tokens)

KV format

Avg unique Cartridges/query

RAG queries/month

Model/Cartridge lifetime (months)

Storage replicas

One-time Cartridge construction cost

CAS trains Cartridge pools jointly, so enter the measured cost of the whole corpus run; the per-document number shown in results is an average, not a marginal cost. The paper reports H200/B200 hardware and optimizer configuration but not wall-clock time or GPU-hours — empty fields stay “benchmark required”.

Measured run Direct total GPU-hours

Training GPUs

Training wall-clock hours

Training GPU provider/rate

Self-study generation cost ($)

Other one-time cost ($)

Per-query inference savings

Use measured wall times. A 10x token reduction is not automatically a 100x reduction in GPU time — complete prefill includes projections, MLPs, kernel efficiency, batching and scheduling, not just O(n²) attention.

Planning-assumption mode (illustrative only)

Text-RAG prefill wall ms/query

Cartridge-path prefill wall ms/query

Inference GPUs/request (TP size)

Inference GPU provider/rate

Decode GPU-ms saved/query (optional)

Compute-value realization 100%

Realization 100% = elastic billing, or saved capacity eventually removes GPUs, so saved GPU time becomes cash. 0% = fixed provisioned fleet: the optimization creates headroom, not a smaller invoice.

Cartridge storage & loading

Dense-retrieval/index costs are common to Text RAG and Cartridge RAG and cancel out of the comparison. The CAS implementation still performs chunk retrieval to select document Cartridges; only the additional Cartridge-object load is modeled here.

Storage provider

Cartridge-object cache hit rate (%)

Custom storage $/GB-month (override)

Custom GET $/request (override)

Custom retrieval $/GB (override)

Custom egress $/GB (override)

Quality gate

Economics are conditional on quality parity: a cheaper result is irrelevant if accuracy fails.

Text-RAG score

Cartridge-RAG score

Allowed degradation (points)

Results

Persistent corpus KV

—

Cartridge KV/document

—

Construction cost (corpus)

—

Construction cost/document (avg)

—

Storage cost

—

GPU-time value saved

—

Realized cash saving

—

Cartridge load cost

—

Net saving/query

—

Break-even query volume

—

Monthly net at selected volume

—

Payback

—

Lifetime ROI

—

Net saving/Cartridge load (approx)

—

Break-even document loads/month

—

Per-document numbers divide query-level savings evenly between the Cartridges selected for a query — an allocation approximation. Popular documents should be trained before cold ones; a measured per-document prefill delta overrides this when available.

Lifetime cost crossover

Cartridge corpus scaling

Model

Compression

KV format

Source corpus tokens

Avg document tokens

Unique Cartridges/query

RAG queries/day

Cache hit rate (%)

Storage provider

Persistent capacity

—

Monthly storage price

—

Active Cartridge prefix/query

—

Cold Cartridge reads

—

Read traffic

—

Corpus size scaling (1M – 100B source tokens)

Daily cold-read traffic vs query volume

Paper-derived values (workload presets)

Model: Qwen3-8B; training hardware: H200/B200.
100–200 self-study questions per document.
Experiments used 80 epochs; the paper’s discussion notes 10–20 epochs often reach ~95% of best performance.
Roughly 12,400–14,160 optimizer steps depending on dataset.
Global batch size 128; packed sequence length 8,192.
LongHealth Text RAG at k=10: 9,860 context tokens/query.
LongHealth Cartridge RAG, 20x compression, k=10: 2,673 Cartridge KV tokens/query.
LongHealth Cartridge RAG, 100x compression, k=10: 566 Cartridge KV tokens/query.
k=10 means ten retrieved chunks, not ten Cartridges.
Average unique Cartridges at k=10: LongHealth 4.4, QuALITY 4.8, FinQA 8.7, TechQA 8.1.
BF16 KV footprint per 1K Cartridge tokens: Qwen3-0.6B 110 MiB, Qwen3-4B/8B 141 MiB, Qwen3-32B 250 MiB.
The paper does not report total GPU-hours or wall-clock training duration — those fields stay “benchmark required” in the paper preset.

What is measured, what is assumed

User-measured values (primary mode): Text-RAG and Cartridge-path prefill wall milliseconds, inference GPU count, and training GPU-hours. Dollar savings are never derived by squaring the token compression ratio.
Illustrative assumptions (planning mode): prefill times estimated from an assumed throughput. Prominently marked; never described as paper results.
Warning: Cartridges are model/checkpoint-specific. Every model upgrade invalidates trained Cartridges and shortens the amortization horizon — that is what the lifetime input models.
Warning: GPU-time value is not necessarily immediate cash savings. On a fixed provisioned cluster the saving first appears as extra capacity, not a smaller invoice — that is what the realization percentage models.
Warning: the economics are conditional on quality parity; the quality gate blocks a plain “profitable” verdict until parity is verified.
KV-format multipliers below BF16 are ideal payload ratios, not measured codec output sizes.

GPU pricing snapshot (checked 2026-07-17 )

Provider	Hardware	Price	Unit	Checked	Source

Storage pricing snapshot (checked 2026-07-17 )

Provider	Class	Price	Checked	Source

Prices are a static, auditable snapshot — nothing is scraped at runtime. Where a provider does not publish a transfer/request price it is marked unmodeled, never silently assumed zero.