Make generation dramatically faster
one move: reuse past keys
Open the code pane immediately, then use the Spec tab, optional design question, tests, and artifact when you want them.
Finish each move before carrying the artifact forward.
Caching prior keys and values turns repeated prefix work from quadratic into incremental work.
Deliverable: Write the mechanism in one sentence.Add a KV cache to a decoder and generate one token at a time.
Deliverable: Type the smallest runnable implementation.Cached and uncached logits must match for every generated position.
Deliverable: Record the expected output and one edge case.Measure tokens per second at growing context lengths and inspect cache memory.
Deliverable: Change one variable and explain the result.A cache-aware generation trace and speedup report.
Deliverable: Save the code, output, and a short failure note.Cached and uncached logits must match for every generated position.
Measure tokens per second at growing context lengths and inspect cache memory.
A cache-aware generation trace and speedup report.
Mapped from the supplied Build Everything PDF, source page 49. The five moves are Deriva’s implementation contract for this project.