Increasing Data Science Productivity: spaCy & Prodigy

2018-04-14 · Source: Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Advanced, extended

Summary

Explosion AI develops spaCy, an open-source natural language processing library, and Prodigy, an annotation tool, alongside its machine learning library, Thinc. spaCy offers syntactic parsing, visualizing word relationships and enabling application-independent structural analysis. Its linear-time, transition-based parsing algorithm is being extended to support diverse languages like Chinese, Vietnamese, and Japanese, demonstrating a 1-3% performance improvement over pipelined methods and outperforming Stanford's system on these languages in the CoNLL 2017 benchmark by integrating segmentation. spaCy also features term-sensitive embeddings for multi-word concepts, enhancing semantic search. Prodigy, designed for annotation efficiency, streamlines data labeling by breaking tasks into binary decisions, supporting active learning, and allowing custom Python recipes. This approach aims to accelerate data iteration, ensure data privacy, and provide JSON output for user ownership.

Key takeaway

For NLP Engineers and Data Scientists building custom language models, you should consider integrating spaCy and Prodigy into your workflow. This combination enables efficient, end-to-end processing for diverse languages and accelerates data annotation through active learning and binary decisions. You can iterate faster on data, improve model accuracy, and retain full ownership of your valuable training datasets.

Key insights

Integrated NLP tools like spaCy and Prodigy enhance data science productivity through efficient linguistic processing and streamlined, active learning-driven data annotation.

Principles

• Syntactic structures are application-independent.
• Binary decisions improve annotation efficiency.
• Joint incremental processing outperforms pipelining.

Method

spaCy uses a linear-time, transition-based parser that incrementally builds dependency trees. Prodigy employs active learning with binary decisions to efficiently generate training data.

In practice

• Use spaCy's term-sensitive embeddings for multi-word search.
• Break annotation tasks into binary decisions.
• Build rule-based patterns with spaCy's syntactic structures.

Topics

• Natural Language Processing
• spaCy Library
• Prodigy Annotation Tool
• Syntactic Parsing
• Active Learning
• Multilingual NLP

Best for: AI Engineer, NLP Engineer, Machine Learning Engineer, Data Scientist

Related on AIssential

• FAQ #1: Tips & tricks for NLP, annotation & training with Prodigy and spaCy — Effective NLP annotation and training require strategic task design, precise feedback, and combining statistical models with rules.
• Building Industrial-Strength NLP Pipelines — Production-grade NLP requires pragmatic problem decomposition, efficient tools, and continuous data-model iteration.
• 3 SpaCy Tricks for Efficient Text Processing & Entity Recognition — Optimizing spaCy pipelines requires tailoring component usage, utilizing batch processing, and integrating rule-based entity recognition.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai.