Building Industrial-Strength NLP Pipelines

2020-10-29 · Source: Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Advanced, extended

Summary

Explosion AI, co-founded by Ines and featuring NLP/ML Engineer Sophie, develops industrial-strength NLP tools, including the open-source spaCy library and the commercial data annotation tool Prodigy. spaCy, a leading Python NLP library, was designed for efficient, production-ready text processing at scale. Its upcoming version three introduces enhanced configurability, seamless integration with HuggingFace Transformers, and "spaCy projects" for reproducible, end-to-end workflows. Prodigy, a developer-focused annotation tool, facilitates rapid data creation and iterative model improvement using active learning. The company emphasizes a pragmatic approach to machine learning, focusing on solving specific industry problems rather than solely research benchmarks. They highlight the importance of decomposing complex tasks, using domain-specific models like sci-spaCy, and leveraging Python's general-purpose capabilities for robust ML pipelines.

Key takeaway

For NLP Engineers building industrial-strength applications, prioritize pragmatic problem decomposition and efficient tooling. Leverage spaCy v3's configurability and Transformer integration to build robust, domain-specific models. Utilize Prodigy for rapid, iterative data annotation, treating labeling as a core development step. This approach ensures your systems deliver real-world value and maintain reproducibility, avoiding common pitfalls of over-optimization or misapplying complex ML.

Key insights

Production-grade NLP requires pragmatic problem decomposition, efficient tools, and continuous data-model iteration.

Principles

• System success is defined by its real-world utility, not just abstract accuracy.
• Decompose complex NLP problems into manageable, combinable building blocks.
• Data annotation is a core development process, not a one-off task.

Method

Prodigy enables active learning-powered data annotation via scriptable Python recipes. spaCy v3 projects provide CLI-driven templates for reproducible, end-to-end NLP workflows, managing data, training, and evaluation steps.

In practice

• Integrate HuggingFace Transformers into spaCy v3 pipelines.
• Use Prodigy for rapid, active learning-driven data annotation.
• Employ spaCy projects to manage reproducible NLP workflows.

Topics

• spaCy Library
• NLP Pipelines
• Data Annotation
• Machine Learning Engineering
• Transformers
• Reproducibility

Best for: AI Engineer, NLP Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• spaCy v3: State-of-the-art NLP from Prototype to Production — spaCy v3 streamlines production NLP with transformer integration, robust configuration, and structured project management.
• FAQ #1: Tips & tricks for NLP, annotation & training with Prodigy and spaCy — Effective NLP annotation and training require strategic task design, precise feedback, and combining statistical models with rules.
• 3 SpaCy Tricks for Efficient Text Processing & Entity Recognition — Optimizing spaCy pipelines requires tailoring component usage, utilizing batch processing, and integrating rule-based entity recognition.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai.