Image Captioning with Prodigy & PyTorch

2020-03-24 · Source: Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, extended

Summary

Prodigy, an annotation tool from Explosion and spaCy's co-founder, enables custom machine learning data creation workflows. This guide demonstrates building an image captioning system using Prodigy with a PyTorch model, starting with over a thousand cat images from a Kaggle dataset. The process involves setting up basic image annotation with Prodigy's scriptable Python recipes, which define data streams and UI components. It then details integrating a pre-trained PyTorch CNN-LSTM model to suggest captions, allowing annotators to correct model outputs. The workflow further incorporates tracking annotator changes against original model suggestions and implements a separate error analysis phase using Prodigy's "choice" interface to categorize correction types (e.g., subject, attributes, background, number, wording). This structured approach helps evaluate model performance and identify specific areas for improvement.

Key takeaway

For AI Engineers building custom data annotation pipelines, Prodigy offers a flexible Python-scriptable framework to integrate machine learning models directly into the loop. You should utilize its recipe system to define custom UIs and data streams, pre-filling tasks with model predictions to boost efficiency. Implement update and on-exit callbacks to track annotator changes and conduct targeted error analysis, ensuring your fine-tuning efforts address specific model weaknesses effectively. This approach streamlines dataset creation and model improvement cycles.

Key insights

Prodigy's scriptable recipes enable highly customizable, model-assisted data annotation workflows for diverse ML tasks.

Principles

• Annotation workflows benefit from automation.
• Avoid free-form input for structured data.
• Separate data creation from error analysis.

Method

Prodigy recipes define annotation workflows via Python functions returning component dictionaries for data streams and UI. Generators handle large datasets efficiently, processing in batches. Callbacks track changes and provide session summaries.

In practice

• Use Prodigy recipes for custom ML annotation.
• Integrate PyTorch models to pre-fill captions.
• Implement callbacks to track annotation changes.

Topics

• Image Captioning
• Data Annotation
• Prodigy
• PyTorch
• Machine Learning Workflows
• Error Analysis

Best for: Machine Learning Engineer, AI Engineer, Data Scientist

Related on AIssential

• Prodigy: A new tool for radically efficient machine teaching — Prodigy is a tool designed to make machine teaching and data annotation radically more efficient, addressing a key bottleneck in AI development.
• Hello, PyTorch! Let the Framework Do the Calculus — PyTorch automates gradient computation and parameter management, simplifying machine learning model development.
• Task Routers in Prodigy — Prodigy's custom task routers enable highly flexible, code-driven control over data annotation task distribution and annotator overlap.
• A Practical Guide to PyTorch: The Deep Learning Framework Built for People Who Think in Code — PyTorch's dynamic computation graphs and Python integration empower flexible deep learning research and deployment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai.