Title & Overview

Template: Text Cleaning & Tokenization: An Intermediate, End-to-End Analysis Tutorial
Overview (≤2 sentences): Learners will implement a reproducible text cleaning and tokenization pipeline using regex-based normalization and subword tokenization with byte-level BPE. This is intermediate because it emphasizes Unicode policies, OOV handling, measurable trade-offs, and reproducible configurations beyond simple whitespace splitting.

Purpose

To move from ad-hoc tokenization toward a defensible, configurable pipeline that works for both classical features (TF-IDF) and transformer fine-tunes. Learners will quantify how cleaning choices and subword tokenization strategies impact metrics, reproducibility, and robustness.

Prerequisites

• Skills: Python, Git, virtual envs; basic NumPy/pandas; ML basics (splits, regularization).
• NLP: tokenization (word vs subword), embeddings vs TF-IDF, macro-F1 vs accuracy.
• Tooling: pandas, scikit-learn, spaCy, Hugging Face Transformers, MLflow, FastAPI.

Setup Instructions

• Environment: Conda/Poetry (Python 3.11), deterministic seeds, .env for secrets/paths.

• Install: pandas, scikit-learn, spaCy, Hugging Face Transformers + Datasets, MLflow, FastAPI.

• Datasets:

  • Small: SST-2 (sentiment classification).
  • Medium: AG News (topic classification).

• Repo layout:

  tutorials/t1-tokenization/
    ├─ notebooks/
    ├─ src/
    │   ├─ cleaning.py
    │   ├─ tokenizers.py
    │   ├─ utils.py
    │   └─ config.yaml
    ├─ data/README.md
    ├─ reports/
    └─ tests/
  

Core Concepts

• Byte-level BPE: avoids OOV by working on raw bytes, preserves emoji/unicode.
• Cleaning policy: Unicode normalization (NFC vs NFKC), case folding, URL/userhandle masking, digit masking.
• Baseline-first: classical vs small transformer pipeline.
• Reproducibility: config-driven cleaning/tokenization policies, seed fixing, dataset versioning.
• Separation: modeling vs guardrails (input validation, schema, length checks).

Step-by-Step Walkthrough

1. Data intake & splits: load datasets via HF Datasets, reproducible splits, save metadata.

2. Cleaning policies:

   • Minimal: NFC, strip control chars, mask URLs.
   • Standard: NFKC, lowercase, mask digits, collapse whitespace.
   • Aggressive: standard + punctuation spacing, emoji demojize.
     Policies tracked in config.

3. Tokenization:

   • Rule-based (spaCy) for comparison.
   • Byte-level BPE (GPT-2 tokenizer via HF) as main path.

4. Baselines:

   • Classical: TF-IDF (char/word n-grams) + Logistic Regression.
   • Neural: DistilBERT with byte-level BPE tokenizer, fine-tune with warmup + early stopping.

5. Evaluation: macro-F1, accuracy; calibration (ECE); slice metrics (length quartiles, URL/emoji presence).

6. Error analysis: confusion pairs, hardest examples, tokenization diffs, OOD checks.

7. Reporting: generate reports/t1-tokenization.md with nbconvert.

8. (Optional) Serve: FastAPI endpoint with schema validation and cleaning policy enforced.

Hands-On Exercises

• Ablations: minimal vs standard vs aggressive cleaning; seq-len 128 vs 256 vs 512; TF-IDF n-gram ranges.
• Robustness: inject typos/emoji/URLs; measure macro-F1 drift.
• Slice evaluation: compare performance on emoji-heavy vs numeric-heavy samples.
• Stretch: hybrid features — concatenate TF-IDF char-ngrams with transformer embeddings.

Common Pitfalls & Troubleshooting

• Tokenizer–model mismatch: must pair byte-level BPE tokenizer with compatible transformer (e.g., GPT-2/DistilGPT2).
• Over-cleaning: case folding or deaccenting may collapse useful signals.
• Metrics misuse: prefer macro-F1 in imbalanced cases.
• OOM: long seqs; mitigate with truncation and gradient accumulation.
• I/O errors: wrap loaders in try/except; fail fast on dtype mismatches.

Best Practices

• Log config, seeds, dataset fingerprints, git commit hash, and tokenizer files.
• Always keep Baseline → Improvement narrative.
• Unit tests: deterministic token boundaries, TF-IDF vocab size, subword consistency.
• Separate cleaning/tokenization from model training for clarity.
• Guardrails: enforce max input length and validate schema at serving.

Reflection & Discussion Prompts

• Why does byte-level BPE improve robustness on noisy text (emoji, rare chars)?
• What trade-offs come from Unicode normalization (NFC vs NFKC)?
• How does case folding impact sentiment vs entity-heavy tasks?

Next Steps / Advanced Extensions

• Train a custom byte-level BPE tokenizer on domain data (SentencePiece).
• Knowledge distillation: transformer → linear TF-IDF model.
• Semantic search: FAISS over embeddings with byte-BPE tokenization.
• Lightweight fairness check: compare performance across slices with/without emoji.

Glossary / Key Terms

Byte-level BPE, OOV, NFC/NFKC, macro-F1, calibration, ECE, slice, tokenizer–model compatibility.

Additional Resources

• [scikit-learn](https://scikit-learn.org/stable/)
• [spaCy](https://spacy.io/)
• [Hugging Face Transformers](https://huggingface.co/docs/transformers)
• [Hugging Face Datasets](https://huggingface.co/datasets)
• [MLflow](https://mlflow.org/)
• [FastAPI](https://fastapi.tiangolo.com/)

Contributors

Author(s): TBD
Reviewer(s): TBD
Maintainer(s): TBD
Date updated: 2025-09-20
Dataset licenses: SST-2 (GLUE), AG News (CC).

Issues Referenced

Epic: HfLA Text Analysis Tutorials (T0–T14).
This sub-issue: T1: Text Cleaning & Tokenization.