Evolution of spaCy

2022-06-11 · Source: Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & & Engineering · Depth: Intermediate, extended

Summary

Ines Montani, co-founder of Explosion, discusses spaCy's evolution and design philosophy, emphasizing its industry-first approach for fast, production-ready NLP. Unlike NLTK, spaCy is opinionated, offering one best implementation for core tasks. The library, built with Cython for speed, supports diverse languages through community contributions and its own machine learning library, Thinc. spaCy 3.0 introduced extensibility, allowing custom components and integration with ML Ops workflows for reproducibility. Explosion also offers Prodigy, an annotation tool, and the upcoming Prodigy Teams for cloud-based annotation. Montani highlights the importance of domain-specific models, citing SciSpaCy and legal text examples, and stresses developer engagement with data for responsible AI, rather than "productizing ethics." The future of NLP, she believes, involves in-house teams building tailored solutions with a focus on developer productivity and continuous iteration.

Key takeaway

For NLP Engineers building production-ready systems, prioritize tools like spaCy that offer opinionated, performant solutions and robust ML Ops features. You should actively engage with your training data and model behavior, rather than relying on abstracted "ethical AI" stamps, to ensure responsible and domain-specific outcomes. Leverage spaCy's extensibility to integrate custom components and fine-tune models on your unique datasets, fostering iterative development and higher project success rates.

Key insights

Industry-grade NLP requires opinionated, fast, and extensible tools that prioritize developer engagement with data.

Principles

• Prioritize one best implementation for core tasks.
• Design for production use from day one.
• Abstracting model complexity hinders responsible AI.

Method

Develop a framework that enables consistent prototyping and production workflows, ensuring reproducibility through configuration and project systems.

In practice

• Contribute language support via tokenization rules.
• Fine-tune models on custom data for domain-specific NLP.
• Use annotation tools to engage with training data.

Topics

• spaCy
• Natural Language Processing
• ML Ops
• Data Annotation
• Thinc
• Responsible AI
• Custom NLP Pipelines

Best for: MLOps Engineer, NLP Engineer, Machine Learning Engineer, AI Engineer

Related on AIssential

• How to build resilient NLP applications — Production-grade NLP requires embracing inherent complexity, providing granular control, and integrating domain expertise for real-world value.
• Creating Tools that Spark Joy with Ines Montani — User-centric developer tools thrive on deep problem understanding, extensibility, and integrated production workflows, ensuring joy and productivity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai.