Save a tokenized corpus to disk
one move: turn bytes into batches
Open the code pane immediately, then use the Spec tab, optional design question, tests, and artifact when you want them.
Finish each move before carrying the artifact forward.
BPE merges frequent byte pairs so rare words remain representable as subwords.
Deliverable: Write the mechanism in one sentence.Download a permitted corpus, tokenize it, and write a compact binary dataset.
Deliverable: Type the smallest runnable implementation.Round-trip samples and token counts must match the manifest.
Deliverable: Record the expected output and one edge case.Vary vocabulary size, sequence packing, and streaming batch size.
Deliverable: Change one variable and explain the result.A dataset manifest with tokenizer version, token count, and split hashes.
Deliverable: Save the code, output, and a short failure note.Round-trip samples and token counts must match the manifest.
Vary vocabulary size, sequence packing, and streaming batch size.
A dataset manifest with tokenizer version, token count, and split hashes.
Mapped from the supplied Build Everything PDF, source page 39. The five moves are Deriva’s implementation contract for this project.