Transferring Natural Language Datasets Between Languages Using Large Language Models for Modern Decision Support and Sci-Tech Analytical Systems

2024-10-16 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

This work investigated how Large Language Models (LLMs) can transfer the dataset and its annotation from one language to another, specifically between English and Russian using the DEFT corpus for term-definition pair mining. It proposes a pipeline utilizing ChatGPT3.5-turbo and Llama-3.1-8b as core LLMs for annotation transfer and text translation. The methodology involved preparing a gold set, automatic translation, and annotation transferring. ChatGPT3.5-turbo showed superior performance in NER annotation transfer with only 44 mismatched spans, while Llama-3.1-8b was six times worse, though fuzzy matching improved its results. BERT-based models (BERT-base-multilingual, RuBERT-base-cased, RoBERTa-base) were trained on the translated dataset to establish baselines for definition detection and term/definition recognition, achieving good results for definition detection but weaker for NER. The entire DEFT corpus transfer took approximately 2 days.

Key takeaway

For NLP engineers developing solutions for under-resourced languages, you can significantly accelerate dataset development by utilizing LLMs for cross-lingual annotation transfer. Prioritize ChatGPT3.5-turbo for initial transfer due to its higher accuracy, but consider Llama-3.1-8b with fuzzy matching for open-source alternatives. Be aware that the resulting "silver" grade data will require careful verification, especially for Named Entity Recognition tasks, to improve model recognition quality and establish robust baselines.

Key insights

LLMs can effectively transfer NLP dataset annotations between languages, significantly reducing manual effort for under-resourced languages.

Principles

• LLM task complexity impacts output quality.
• Fuzzy matching improves LLM annotation transfer.
• Silver-grade datasets from LLMs are valuable.

Method

The pipeline involves preparing a gold set, LLM-based text translation, and LLM-based NER annotation transferring, followed by training BERT-like models on the translated data to establish baselines.

In practice

• Use ChatGPT3.5-turbo for annotation transfer.
• Apply fuzzy matching for Llama-3.1-8b outputs.
• Reformulate LLM tasks for simpler cognitive steps.

Topics

• Large Language Models
• Cross-Lingual Data Transfer
• Named Entity Recognition
• DEFT Corpus
• ChatGPT3.5-turbo
• Llama-3.1-8b

Code references

• adobe-research/deft_corpus
• HumanSignal/label-studio

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs reason effectively across 43 languages, with machine translation not degrading accuracy.
• Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research — LLMs offer transformative potential for low-resource language research in humanities, despite significant data and ethical challenges.
• Adaptive Intellect Unleashed: The Feasibility of Knowledge Transfer in Large Language Models — Knowledge transfer can improve LLM generalization in software engineering tasks, but its implementation requires careful design.
• Unlocking the Power of Large Language Models for Multi-table Entity Matching — LLM4MEM uses LLMs to resolve semantic inconsistencies and efficiently match entities across multiple data tables.
• English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training — Multilingual post-training significantly improves LLM performance across languages and tasks, outperforming English-only approaches.
• System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5 — Domain-specific datasets and LoRA fine-tuning significantly enhance LLM performance in specialized tasks like classical poetry analysis.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.