Module 10 · Neural and language models

Module 10: implement, explain and transfer

Read concepts, then attempt the starter notebook. In a source checkout open notebooks/practice/module10-practice.ipynb. Use the framework environment and kernel. Three unfinished functions report not_attempted, not a pass. If a download link does not open, use the offline access route.

Construct parse_label

Return ham/spam only for an exact one-key JSON object; otherwise return None. Predict the result on parse_label('{"label":"spam"}'), implement it and call m10_parse_label through the notebook check. Then add a changed case and explain which assumption it tests.

Construct valid_fraction

Input parsed labels or None. Return valid/total, or None for empty input. Predict the result on valid_fraction(['spam', None, 'ham']), implement it and call m10_valid_fraction through the notebook check. Then add a changed case and explain which assumption it tests.

Construct end_to_end_accuracy

Return exact matches / all cases, counting None as wrong; None for no cases. Predict the result on end_to_end_accuracy(['spam', 'ham'], ['spam', None]), implement it and call m10_end_to_end_accuracy through the notebook check. Then add a changed case and explain which assumption it tests.

Work with the complete study's results

Open generation_evaluation.csv from the complete study. Count the schema-valid outputs and the correct outputs, then compute valid_fraction and end-to-end accuracy. What accuracy would you report if you silently dropped the invalid output? Compare with the construction feedback afterwards.

Reason beyond the visible checks

Three messages are all actually spam. The model returns '{"label":"spam"}', 'spam' and '{"label":"ham"}'. Compute valid_fraction, end_to_end_accuracy, and the accuracy you would report if you silently dropped malformed outputs. The checks cannot mark this; compare your reasoning with the construction feedback.

Transfer to an IT decision

Evaluate three changed message outputs including one malformed response. Explain why structural validity alone cannot justify automation. Run the complete module study and verify one claim against its actual artifacts; keep the evidence the study workflow lists.

Advanced extension: evaluate retrieval before generation

Retrieval-augmented generation (RAG) answers from retrieved passages, so its answers can only be as good as what retrieval returns. This extension evaluates that step on 12 original, fictional IT runbooks and 32 authored questions, eight of which the runbooks cannot answer. It runs in the core environment with the regression extra; no language model is needed.

python -m practical_ml_ai.retrieval --output runs/module10-retrieval

Validation questions choose the TF-IDF settings and an abstention threshold: below that similarity the system should say it has no grounded answer. Test questions then evaluate the locked choice once. Before running, predict whether the threshold will refuse every unanswerable test question. Then use summary.txt and test_decisions.csv:

1. What are recall@3 and mean reciprocal rank on the answerable test questions?
2. Which unanswerable test questions were answered anyway, which passage did they retrieve, and why did it match?
3. A language model given that passage would produce a fluent answer. What should the system do instead, and what evidence would justify a different threshold?

Evidence and feedback

• M10.1: Explain token generation and distinguish instructions from untrusted message data.
• M10.2: Run a recorded public model revision with bounded greedy decoding.
• M10.3: Validate an exact JSON schema and count failures.
• M10.4: Compare task fit, latency, and limitations with simpler NLP.
• M10.5: Write a source-grounded model-use note with privacy and cost constraints.

Use construction feedback after an attempt and recorded experiment feedback for the full workflow. Follow the shared study workflow for evidence, recovery and optional AI disclosure. Official assessment weights remain with your instructor.

Evidence activity: Separate validity from correctness

Use saved LLM outputs and their evaluated labels. Independently count valid responses and correct responses under the recorded scoring rule. State the denominator for each reported rate; explain how a syntactically valid answer can still be wrong.

Follow the four steps in reproduce a result and justify its use. Submit your artifact path, independent calculation, discrepancy or agreement, and bounded use decision in the evidence worksheet. Use your complete reference run; this activity complements the small function checks.