Deriva — Derive the Algorithm

← Projectslexical-search / L3BM25-Style Score~18 min

one move: measure behavior

Count what token follows each token. This is the complete sufficient statistic for a tiny bigram language model before probabilities or sampling. In the Lexical Search system, implement this level as a deterministic contract before composing it with the next service.

Required API

def bigram_counts(tokens):
    return {token: {next_token: count}}

Behavior

look only at adjacent pairs
create a row for each source token
increment repeated transitions

Examples — tap to reveal

Constraints

Use only the Python standard library.
Return deterministic values so every run can be compared with the contract.
Do not mutate caller-owned inputs.

Opening…