Epic: Create Text Analysis Tutorials #227

# Title & Overview

**Template:** *{NLP Task}: An Intermediate, End-to-End Analysis Tutorial*
**Overview (≤2 sentences):** Summarize what learners will build and why this is **intermediate** (e.g., beyond “fit a model,” toward rigorous analysis, evaluation, and reproducibility).

# Purpose

State the value-add beyond beginner level: defensible baselines vs small transformers, robust featureization, slice-based error analysis, reproducible experiments, and light deployment/reporting.

# Prerequisites

* Skills: Python, Git, virtual envs; basic NumPy/pandas; ML basics (train/val/test, overfitting, regularization).
* NLP: tokenization (wordpiece/BPE), embeddings vs TF-IDF, evaluation metrics (accuracy, macro-F1).
* Tooling (pick 1 per pair when applicable): spaCy **or** NLTK; scikit-learn **or** gensim; Hugging Face Transformers **or** Haystack.

# Setup Instructions

* Environment: Conda/Poetry (Python 3.11), deterministic seeds, `.env` for secrets/paths.
* Install (choose 1 per pair, keep consistent across the tutorial):

* Data: pandas **or** polars
* Vectorization: scikit-learn TF-IDF **or** gensim
* Embeddings: Sentence-Transformers **or** spaCy pipelines
* Topic modeling (if relevant): gensim LDA **or** BERTopic
* Evaluation: scikit-learn **or** seqeval/torchmetrics
* Serving: FastAPI **or** Flask
* Tracking: MLflow **or** Weights & Biases
* Data: pick a public dataset (HF Datasets), note license, splits, and any filters.
* Repo layout: `src/`, `notebooks/`, `configs/`, `data/README.md`, `reports/`, `tests/`.

# Core Concepts

Explain the *why* behind the workflow:

* Tokenization tradeoffs (word/char/subword) and OOV handling.
* Featureization: n-grams vs dense embeddings; when each is preferable.
* Baseline-first philosophy; calibration & thresholding; imbalanced data strategy.
* Slice-based error analysis; reproducibility and run governance (config + seed + commit hash).
* Clear separation of *model core* vs *guardrails/safety layers* (input validation, schema checks).

# Step-by-Step Walkthrough

1. **Data intake & splits** (stratified, reproducible) → **EDA** (class balance, length stats).
2. **Baseline A (classical):** TF-IDF + {LogReg | LinearSVM}; tune `C`, `n-gram` range.
3. **Baseline B (neural):** small Transformer fine-tune; freeze/unfreeze plan; LR warmup; early stopping.
4. **Evaluation:** macro-F1, accuracy; calibration (ECE); per-slice metrics (by length/domain/label).
5. **Error analysis:** confusion pairs, hardest examples, failure taxonomies, OOD checks.
6. **Reporting:** export metrics tables, error examples, and decisions to `reports/<slug>.md`.
7. *(Optional)* **Serve:** minimal FastAPI/Flask endpoint with input schema + simple guardrails.

# Hands-On Exercises

* Ablations: tokenizer choice, `n-gram` window, max seq-len, LR/epochs, freeze ratio.
* Robustness: noise/typos, domain shift split; evaluate delta in macro-F1.
* Fairness/slices: compare metrics across salient attributes (if present).
* Stretch: knowledge-distill to a smaller model; ANN semantic search demo with embeddings.

# Common Pitfalls & Troubleshooting

* **Data leakage:** duplicates across splits, preprocessing mismatch train vs infer.
* **Tokenizer/model mismatch:** bad vocab; truncation causing label drift.
* **Metrics misuse:** micro vs macro; thresholding without calibration.
* **Resource issues:** OOM from long sequences; mitigate with truncation + grad accumulation.
* **Error handling pattern:** wrap I/O in `try/except` with clear `print()` messages for file-not-found and retries; **fail fast** with a trace on data type mismatches; log warnings for recoverable issues.

# Best Practices

* Track config, seeds, and git commit per run; log artifacts (confusion matrices, PR curves).
* Keep a **Baseline → Improvement** narrative; avoid overfitting the leaderboard.
* Small, testable functions; unit tests for tokenization/vectorization/metrics.
* Separate core model from guardrails (validation, schema, limits) for clarity and maintainability.

# Reflection & Discussion Prompts

* Where does the model fail and why? What additional signals or data would fix it?
* How would you adapt this pipeline to civic-tech/public datasets (privacy, governance, bias)?
* What’s the real-world impact of a +Δ macro-F1 improvement in this task?

# Next Steps / Advanced Extensions

* Domain adaptation; PEFT/LoRA; quantization for CPU.
* Data augmentation (BT/paraphrase), prompt-augmented retrieval (RAG) for retrieval-heavy tasks.
* Batch inference pipeline; lightweight monitoring (drift checks, alerting).

# Glossary / Key Terms

Define task-specific terms (e.g., subword, OOV, calibration, ECE, macro-F1, slice, leakage, PEFT, LoRA).

# Additional Resources

List 4–6 canonical links (docs for scikit-learn, spaCy, HF Transformers & Datasets, Sentence-Transformers, BERTopic/gensim, FastAPI/Flask, MLflow/W\&B).

# Contributors

Author(s), reviewer(s), maintainer(s), date updated; dataset license notes.

# Issues Referenced

Link back to the Epic and the sub-issue this tutorial belongs to; include any prior discussions or decisions.