AWS has introduced a new pattern to cut AI retrieval bills on Amazon Bedrock. If you build retrieval-augmented generation (RAG) apps, this setup can lower your token spend, but your users may wait longer for answers.
In a post published 2026-08-21 on the AWS Machine Learning Blog, AWS showed how to place a small model between retrieval and the final answer. Instead of sending long text chunks to a large model, a fast model extracts only the exact relevant sentences first.
How the setup works
The reference design runs inside an AWS Lambda function using the Amazon Bedrock Converse API. You need an AWS account, IAM permissions, and model access.
The pipeline splits the work between two models:
– Claude Haiku acts as the compressor, picking verbatim text spans that match the user’s question.
– Claude Sonnet reads only the trimmed text and writes the final response.
AWS tested three approaches across more than 500,000 documents from nine enterprise source types and 500 questions across 10 categories: baseline RAG, compression alone, and rerank plus compression.
What the numbers show
According to AWS, shrinking the context creates large savings:
– Token volume: Drops to 12% with compression (8.6x fewer tokens) and 10% with rerank plus compression (10.1x fewer tokens), compared to 100% on baseline RAG.
– Total cost: Falls to 67% (a 33% saving) and 64% (a 36% saving).
– Quality score: Stays close to the baseline (97.5% and 97.6% versus 100%), based on an LLM judge scoring correctness, completeness, citation accuracy, and conciseness.
– Hallucination rate: Drops from 51% on baseline to 44% with compression and 38% with reranking, with faithfulness tracked separately.
– Latency: Rises by +19% for compression and +12% for rerank plus compression. Speed takes a hit.
The claims you should verify yourself
These numbers come entirely from AWS’s internal tests on a single corpus evaluated by an LLM judge. Nobody outside AWS has independently checked them, and they are not a performance guarantee for other workloads, price points, or models. Adding another model call creates a new point of failure. The compressor could accidentally discard vital facts, weaken citations, or choke on malformed spans and prompt injection.
How to test this pattern
Do not change your production pipeline yet. Test your own data first.
Set up a fixed test query set with your retriever, top-k settings, chunking, and model IDs. Run queries through all three paths: baseline RAG, compression, and rerank plus compression. Log input tokens, output tokens, Lambda cost, model cost, and both p50 and p95 latency. Most importantly, review the compressed spans directly. Check what gets dropped. If tail latency spikes or key evidence goes missing, keeping your baseline RAG path remains the safer choice.