From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languages

2026-07-27 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Digital Humanities · Depth: Expert, quick

Summary

Automatic Text Recognition (ATR) provides digital humanities with large volumes of unstructured, noisy ancient language texts. Existing sentence embedding methods, including generic multilingual encoders and specialized language models, perform poorly for downstream semantic analysis tasks like text reuse identification, alignment, and semantic search due to anisotropic representation spaces and lack of labeled similarity data. This research studies two unsupervised strategies, TSDAE and Contrastive Sentence Embedding (CSE), which adapt a specialized token-level language model into a corpus-specific sentence encoder using only raw sentences. Evaluated on 2,935 expert-verified biblical reuse parallels in Latin and Ancient Greek, these adapted encoders outperform all baselines. TSDAE leads binary detection with large in-domain corpora, while CSE excels in correspondence retrieval, reaching optimal performance with just 4-8k raw sentences and transferring across authors and noisy post-ATR text. The full pipeline is available via the online tool Paraphrasis.

Key takeaway

For digital humanities researchers or computational linguists working with ancient texts, you can significantly improve semantic analysis tasks like text reuse identification and semantic search. Your existing specialized language models can be effectively adapted using unsupervised methods, even with limited data. Consider implementing TSDAE for large-scale binary detection or CSE for efficient correspondence retrieval, which requires only 4-8k sentences, and explore the Paraphrasis online tool for a complete pipeline.

Key insights

Unsupervised methods adapt specialized LMs for ancient language sentence embeddings, outperforming baselines for semantic analysis.

Principles

• Unsupervised adaptation improves ancient language embeddings.
• TSDAE excels in detection with large corpora.
• CSE is effective for retrieval with minimal data (4-8k sentences).

Method

TSDAE and Contrastive Sentence Embedding (CSE) adapt a specialized token-level language model into a corpus-specific sentence encoder using only raw sentences for corpus-specific sentence encoding.

In practice

• Retrain CSE on noisy post-ATR text for robust performance.
• Utilize Paraphrasis for cross-corpus semantic search.
• Apply TSDAE for binary detection in large ancient language corpora.

Topics

• Unsupervised Learning
• Sentence Embeddings
• Ancient Languages
• Digital Humanities
• Text Reuse Identification
• TSDAE
• Contrastive Sentence Embedding

Best for: NLP Engineer, AI Scientist, Research Scientist, Domain Expert

Related on AIssential

• MiqraBERT: Regression-Based Sentence-BERT Finetuning for Biblical Hebrew Parallel Detection — Finetuning a BERT model with regression for semantic similarity effectively detects textual reuse in ancient languages beyond lexical overlap.
• SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation — Specialized text embedding models can be effectively adapted for low-resource languages using existing multilingual foundations.
• LIAAD INESCTEC at SemEval-2026 Task 4: Unsupervised Narrative Similarity via Discourse Representation Structures and Sentence Embeddings — Unsupervised narrative similarity can be effectively determined by parsing stories into Discourse Representation Structures and extracting features.
• From TF-IDF to Transformers: Implementing Four Generations of Semantic Search — Semantic search evolved from explicit rules to learned contextual representations, balancing interpretability with semantic depth.
• On the feasibility of dependency parsing of non-human sequences without a gold standard. Is evaluation possible in other species? — Unsupervised dependency parsing evaluation is feasible for non-human primates due to short, geometrically distributed sequences, unlike human languages.
• Evaluating Latin and Ancient Greek Sentence Alignment through Parallel Sentence Mining — Whitening transformation and knowledge distillation significantly improve cross-lingual parallel sentence mining for Latin and Ancient Greek.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.