We Tested TF-IDF Against Two Transformers for Contract Risk Screening. Bag-of-Words Won Twice.

2026-07-07 · Source: NLP on Medium · Field: Legal & Regulatory — Legal Technology (LegalTech), Artificial Intelligence & Machine Learning, Compliance & Risk Management · Depth: Advanced, medium

Summary

A contract risk screening system, developed using the CUAD legal dataset, demonstrated that a bag-of-words approach (TF-IDF + logistic regression) significantly outperformed transformer-based models in precision at matched recall. In a five-way bake-off, TF-IDF achieved 0.248 precision at 0.904 recall, more than double the precision of frozen MiniLM embeddings (0.107 at 0.907 recall) and fine-tuned DeBERTa-v3-base (0.107 at 0.874 recall). An LLM (gpt-oss-120b) showed high precision (0.812) but low recall (0.613). The final production system, a hybrid architecture combining TF-IDF screening with LLM triage for weak flags, achieved the best F1 score of 0.852, processing 95.6% of decisions without LLM cost and running entirely on free-tier cloud resources. This system screens 8 business-facing risk categories from 408 commercial contracts.

Key takeaway

For AI Engineers developing contract risk screening systems, prioritize simple, high-precision models like TF-IDF before investing in complex transformers. If you are building a system where missed risks are costly, implement recall-first thresholding. Consider a hybrid architecture that uses a cheap baseline for initial screening and routes only uncertain cases to an LLM, significantly reducing operational costs while maintaining high performance.

Key insights

For contract risk screening, simple bag-of-words models can significantly outperform transformers in precision at matched recall, especially when combined with LLM triage.

Principles

• Prioritize operating point (e.g., recall-first) before model selection.
• Simple models can excel for tasks with literal "signatures."
• Route uncertainty to specialized, costlier models.

Method

A hybrid system screens contracts with TF-IDF, routing weak flags to an LLM for confirmation or clearance. Strong flags are kept, and low-score items are cleared, minimizing LLM usage.

In practice

• Use TF-IDF for tasks with distinct keyword patterns.
• Implement recall-first thresholding for screening systems.
• Cache successful LLM verdicts, not transient errors.

Topics

• Contract Risk Screening
• TF-IDF
• Transformers
• Large Language Models
• Machine Learning Evaluation
• Hybrid AI Systems
• CUAD Dataset

Code references

• DhavalVibhakar99/Lexorion-Cuad

Best for: AI Engineer, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• From Text to Numbers: Demystifying Text Preprocessing in NLP — NLP text preprocessing, using methods like Bag of Words and TF-IDF, translates human language into numerical data for machine learning models.
• A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision — Current multimodal anomaly detection systems exhibit superficial text guidance, often relying more on visual features than language conditioning.
• From TF-IDF to Transformers: Implementing Four Generations of Semantic Search — Semantic search evolved from explicit rules to learned contextual representations, balancing interpretability with semantic depth.
• Lightweight Transformer Models for On-Device Fault Detection: A Benchmark Study on Resource-Constrained Deployment — Lightweight Transformers can match traditional ML for on-device fault detection but demand significant resource tradeoffs, especially on imbalanced data.
• How TF-IDF Turns Noise Into Signal — TF-IDF balances local word importance with global rarity to create meaningful numerical document representations.
• Data Presentation Over Architecture: Resampling Strategies for Credit Risk Prediction with Tabular Foundation Models — Context construction significantly impacts Tabular Foundation Model performance in imbalanced credit risk prediction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NLP on Medium.