EXP-001 – EVAL-010

Experiments and evaluation

0 of 20 done · 0 due for review.

EXP-001Write a one-sentence hypothesis before changing this feature: replace the bigram feature with a trigram feature.communicate · the-learning-rate-is-a-system-choicenew
EXP-002Select the one variable allowed to change in this experiment: learning rate, feature set, seed, and split are all candidates.construct · the-learning-rate-is-a-system-choicenew
EXP-003Baseline scored 0.88 on split A, model scored 0.90 on split B. Detect this invalid comparison and state the rule it broke.debug · build-a-better-baseline-honestlynew
EXP-004Reproduce this run from its config and seed: seed 42, lr 0.01, 100 epochs. What must the runner fix to make replay exact?implement · reproducibility-is-part-of-correctnessnew
EXP-005Read this learning curve: train and val loss both fall slowly and plateau high. Diagnose underfitting and predict the fix.predict · regularization-is-a-tradeoffnew
EXP-006Read this learning curve: train loss near zero, val loss rising after epoch 20. Diagnose overfitting and predict the fix.predict · regularization-is-a-tradeoffnew
EXP-007An ablation gains +0.002 F1 at 3× serving cost. Decide whether this improvement is practically meaningful and justify.predict · build-a-better-baseline-honestlynew
EXP-008Design an ablation for this feature group: text length, word count, and punctuation ratio are added together.construct · build-a-better-baseline-honestlynew
EXP-009Classify these errors by root cause: too-short tickets, rare vocabulary, duplicate training rows, and a leaked eval row.implement · find-the-error-familynew
EXP-010From this error taxonomy — 60% vocabulary errors, 25% duplicates, 15% short-text — choose the next experiment and predict its effect.predict · find-the-error-familynew
EVAL-001Choose metrics for this high-recall moderation system: missing a bad comment costs 50× a false flag. Which metric must be maximized?predict · the-baseline-earns-the-right-to-improvenew
EVAL-002Calculate PR-AUC from these points: recall (0.2, precision 1.0), (0.5, 0.8), (0.9, 0.5).derive · the-baseline-earns-the-right-to-improvenew
EVAL-003A model reports 0.95 accuracy but the 'feature' class recall is 0.1. Find the metric that hides this minority-class failure.debug · when-a-good-score-liesnew
EVAL-004Design slice evaluation for these user segments: mobile, desktop, and API-only users.construct · find-the-error-familynew
EVAL-005Interpret this calibration curve: predictions in the 0.9 bin are correct 60% of the time. Write the finding.communicate · when-a-good-score-liesnew
EVAL-006Build a regression fixture for this previous failure: a bug was missed because it contained no bug keywords.implement · reproducibility-is-part-of-correctnessnew
EVAL-007Decide whether this model is ready for shadow traffic: offline F1 0.82, regression fixtures pass, calibration ECE 0.04.predict · measure-the-service-not-just-the-modelnew
EVAL-008Latency rose 40% while quality held. Separate data, model, and serving regressions: which bucket does this belong to?debug · measure-the-service-not-just-the-modelnew
EVAL-009Explain why offline success did not transfer online: offline F1 0.9, live precision 0.6. Write the likely causes.communicate · measure-the-service-not-just-the-modelnew
EVAL-010Write the acceptance question for this model release: bug recall must beat the baseline on the fixed split, with no slice below 0.5.communicate · the-baseline-earns-the-right-to-improvenew

Opening…