Patrick Harrison: Financial NLP at S&P Global

2019-07-06 · Source: Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai · Field: Finance & Economics — FinTech & Digital Financial Services, Capital Markets & Investment Management, Economic Analysis & Policy · Depth: Intermediate, long

Summary

S&P Global, a Fortune 500 financial data and technology company with a \$50+ billion market cap, is developing a next-generation ESG (Environmental, Social, Governance) data set using NLP and active learning. This initiative addresses the challenge of collecting unstructured ESG data, which differs significantly from regulated conventional financial metrics. Their NLP pipeline frames the task as a multi-label classification problem, utilizing tools like spaCy's TextCat and PyTorch-based BERT models to identify text spans as evidence for hundreds of ESG attributes. The active learning lifecycle begins with a handful of expert annotations, iteratively trains and fine-tunes models, validates predictions with domain experts, and continuously generates new training data. This process aims for S&P Global's 100% accuracy guarantee, transitioning to a production operations mode for ongoing data collection and model refinement across thousands of companies.

Key takeaway

For Directors of AI/ML tasked with building high-accuracy data products from unstructured text, you should prioritize human-in-the-loop systems. Your models, even with high F1 scores, must integrate into a larger workflow that guarantees 100% data precision and recall, especially for critical financial or regulatory data. Implement active learning loops with domain expert validation to continuously improve model performance and generate gold-standard training data, ensuring your system meets stringent accuracy requirements.

Key insights

S&P Global builds high-accuracy ESG datasets by combining NLP (spaCy, BERT) with an active learning loop and human expert validation.

Principles

• 100% data accuracy requires human-in-the-loop systems.
• Unstructured data collection benefits from iterative active learning.
• Domain experts are crucial for high-quality labeled data.

Method

An active learning lifecycle for multi-label classification: initial expert annotations, model training (spaCy TextCat, BERT), prediction on historical documents, expert validation, and iterative feedback to refine models and generate training data.

In practice

• Use spaCy for efficient text preprocessing and tokenization.
• Frame evidence extraction as multi-label classification.
• Integrate domain experts for high-quality data labeling.

Topics

• Financial NLP
• ESG Data
• Active Learning
• Multi-label Classification
• spaCy
• BERT
• Data Accuracy

Best for: Machine Learning Engineer, NLP Engineer, Director of AI/ML

Related on AIssential

• Practical transfer learning for NLP with spaCy and Prodigy — Modern NLP workflows combine pre-training, active learning, and continuous maintenance to accelerate development and improve model longevity.
• The AI Bottleneck: High-Quality, Human-Powered Data — High-quality, human-aligned datasets are crucial for AI systems to solve real-world problems effectively.
• Predicting Overnight Stock Changes Post-Earnings: Tomorrow’s Price Hidden in Today’s Language — Earnings call language, processed with domain-specific NLP, predicts overnight stock movements.
• Intro to NLP with spaCy (5): Detecting programming languages — Comparing rule-based and ML NLP models reveals performance trade-offs and data quality impacts.
• Build financial document processing with Pulse AI and Amazon Bedrock — Combining Pulse AI with Amazon Bedrock enables highly accurate, scalable financial document processing through domain-specific model fine-tuning.
• Auditing the Evaluators: How Far Can Automatic Evaluation Go in Assessing Portuguese Financial Texts? — Automatic and LLM-as-a-judge metrics struggle with Portuguese financial texts, requiring domain-specific evaluation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai.