P6: Context Rot – Hamel’s Blog - Hamel Husain

Title Slide

The Rise of Long Context Windows

The Common Assumption: More Context is Better

Explaining the “Needle in a Haystack” (NIAH) Benchmark

Experiment 1: Adding Ambiguity (Semantic vs. Lexical Matching)

Implications of Ambiguity in Real-World Applications

Experiment 2: Adding Distractors

Visualizing the Distractor Setup

Results: Performance Degrades with More Distractors

Implications of Distractors in Domain-Specific Contexts

Analyzing Failure Modes: Model Hallucinations vs. Abstention

Experiment 3: Shuffling Haystack Content

Surprising Results: Models Perform Better on Shuffled Context

Experiment 4: Conversational Memory

Experiment 5: Text Replication Task

Key Takeaways

Context Engineering Example: Orchestrator and Subagents