Assessing reproducibility and justified use
Use the learner activity to assess existing outcomes. This is a suggested formative rubric; official grades and accommodation requirements remain with the adopting instructor. Pilot Modules 5, 6 and 8 first, then use the module-specific tasks elsewhere. These activities are available across all modules, but their learner workload has not been measured.
| Dimension | Needs revision | Meets the evidence requirement | Further depth |
|---|---|---|---|
| Reconstruction | Command, data identity or environment is missing | A peer can locate inputs and execute the documented route; disagreements are recorded | Independent environment reproduction with explained tolerances |
| Numerical traceability | Copies a headline metric | Recomputes a claim from saved records with correct denominator and units | Reconciles multiple artifacts and diagnoses an injected discrepancy |
| Evaluation boundary | Uses test outcomes to choose a change | Explains fitting/selection/evaluation boundaries and changes only development choices | Designs a fresh external/group/time evaluation |
| Stability | Reports only a favorable seed or calls seed spread a confidence interval | Reports declared runs and distinguishes training randomness from split sensitivity | Justifies additional variation sources and their computational cost |
| Use decision | Declares the system trustworthy from accuracy alone | Connects baseline, errors, affected people and missing evidence to a bounded decision | Defines monitoring, human escalation and a measurable pause condition |
Score a dimension only when the assigned activity provides evidence for it. For example, Module 1 does not require fitting multiple models. A failed reproduction with an accurate diagnosis can demonstrate stronger understanding than unexplained matching output. A favorable model result is never a grading requirement.
Safe discrepancy exercise
Use a disposable copy of an output table. Change one prediction or count, leaving the original intact. Ask the learner to recompute its metric and identify disagreement with the saved summary. In the capstone, learners may instead copy a completed run, alter one tracked output and run its verify command; the expected outcome is rejection. Never repair the ledger to disguise the change. Restore by returning to the original or rerunning into a new directory.
Pilot and feedback
Use the confidential observation fields in the worksheet. Separate setup, active learning and waiting time. Record the exact route, stall point, skipped pages and assistance. Ask learners to explain one changed case without the solution. Decide on framework migration or a default hosted route only after comparing observed barriers; this exercise does not itself verify either route.
Capstone evidence gate
Require a reconstruction packet, one independently recalculated metric, a complete declared-seed table, a separately interpreted split study if assigned, and a use-decision memo. Include one failed attempt and its repair where available. A peer signs off only on the checks actually performed. Reading-only, same-machine and independent-environment evidence must remain distinguishable. Privacy, adversarial robustness and prospective deployment evidence may remain open, but the final recommendation must reflect those gaps.