Letter Lemmatization: One-to-one and Banded RNNs for Reversing Character-Set Simplification and Abbreviation in Medieval Text
Summary
This work introduces "letter lemmatization," a method for managing character-set simplification and abbreviation in medieval texts, alongside the pylelemmatize Python library. The library efficiently implements charset simplification mappings (CSMs) and a character similarity heuristic to automatically derive them. The authors propose one-to-one Recurrent Neural Networks (RNNs) that use self-supervision to reverse CSMs, demonstrating recovery of half the Character Error Rate (CER) with as few as 20 text lines and significant improvements in Handwritten Text Recognition (HTR) post-correction. For tasks involving insertions and deletions, such as abbreviation expansion, Banded RNNs extend this architecture, achieving 3.9% CER on the Fontenay corpus and 4.2% on the Santa Maria della Grotta corpus, reducing no-op baselines by 5-7 times. pylelemmatize also offers a 2.8x to 6x speedup over generic Python mapping methods.
Key takeaway
For Machine Learning Engineers working with historical document digitization, you should consider implementing character-level RNNs to address character set inconsistencies and abbreviations. These methods, particularly Banded RNNs, significantly reduce character error rates for tasks like abbreviation expansion, achieving 3.9-4.2% CER. You can leverage the pylelemmatize library to efficiently manage character set mappings and improve HTR post-correction, even with limited training data.
Key insights
Character-level RNNs and similarity heuristics effectively manage character set variations and abbreviations in historical texts, even with limited data.
Principles
- One-to-one mappings simplify character set management.
- Self-supervision can reverse information loss from simplification.
- Small datasets can yield significant error reduction.
Method
Letter lemmatization uses a heuristic to quantify character similarity and automatically derive one-to-one CSMs. One-to-one RNNs are trained with self-supervision to reverse CSMs. Banded RNNs extend this with dynamic programming and CTC for insertions/deletions.
In practice
- Use pylelemmatize for efficient character set management.
- Apply one-to-one RNNs for HTR post-correction.
- Employ Banded RNNs for medieval abbreviation expansion.
Topics
- Letter Lemmatization
- Recurrent Neural Networks
- Handwritten Text Recognition
- Character Set Normalization
- Abbreviation Expansion
- Medieval Text Digitization
- pylelemmatize Library
Code references
Best for: AI Scientist, Machine Learning Engineer, Research Scientist
Related on AIssential
See Counsel's argued verdicts on the open AI decisions leaders are weighing →
Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.