See experts specialize by token
one move: route tokens to experts
Open the code pane immediately, then use the Spec tab, optional design question, tests, and artifact when you want them.
Finish each move before carrying the artifact forward.
Mixture-of-Experts replaces one FFN with routed experts plus a load-balancing objective.
Deliverable: Write the mechanism in one sentence.Implement top-k routing, expert capacity, combine weights, and auxiliary balancing loss.
Deliverable: Type the smallest runnable implementation.Every token must route within capacity and outputs must preserve shape.
Deliverable: Record the expected output and one edge case.Change expert count, top-k, and routing temperature; inspect load skew.
Deliverable: Change one variable and explain the result.An MoE routing report with expert utilization and quality.
Deliverable: Save the code, output, and a short failure note.Every token must route within capacity and outputs must preserve shape.
Change expert count, top-k, and routing temperature; inspect load skew.
An MoE routing report with expert utilization and quality.
Mapped from the supplied Build Everything PDF, source page 56. The five moves are Deriva’s implementation contract for this project.