Make raw text batchable by defining a deterministic lowercase whitespace tokenizer before adding a learned vocabulary.
Required APIdef tokenize(text):
return token_listThink through the mechanism first if you want the extra reasoning step. It never blocks the editor.
tokenize('Hello WORLD')the same text should produce the same token sequence
tokenize(' ')empty documents create no training tokens
2 hidden edge tests run after the visible contract passes.
Run Tests to see the contract verdicts here.