Scaling Human Judgment: How Dropbox Uses LLMs to Improve Labeling for RAG Systems

2026-03-07 · Source: InfoQ · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, quick

Summary

Dropbox engineers have implemented a novel approach using Large Language Models (LLMs) to augment human labelling processes, specifically to enhance the relevance of responses generated by Dropbox Dash. This method is critical for accurately identifying and retrieving relevant documents that form the basis of Dash's responses. The strategy provides valuable insights for other systems that rely on retrieval-augmented generation (RAG) architectures, demonstrating how LLMs can improve data quality and relevance in information retrieval tasks.

Key takeaway

For AI Engineers developing RAG systems, integrating LLMs to augment human labelling can significantly boost the relevance and accuracy of your system's outputs. You should explore using LLMs for pre-screening or suggesting labels to human annotators, thereby streamlining the data curation process and improving overall response quality.

Key insights

LLMs can significantly augment human labelling to improve data relevance in RAG systems.

Principles

• Human-LLM collaboration enhances data quality.
• Relevance is key for effective RAG systems.

Method

LLMs are used to assist human annotators in identifying and labelling relevant documents, thereby improving the quality of data used for retrieval-augmented generation.

In practice

• Apply LLMs for initial document filtering.
• Integrate LLM suggestions into human review workflows.

Topics

• Large Language Models
• Human Labelling
• Retrieval-Augmented Generation
• Dropbox Dash
• Information Retrieval

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• RAG Explained for Beginners — RAG enhances LLMs by providing external, relevant context to overcome knowledge and context window limitations.
• From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives? — LLMs can statistically outperform human annotators in estimating aggregate subgroup opinions under common conditions.
• Your LLM Isn’t Dumb — It Just Lacks Your Context — LLMs need enterprise context; HITL feedback RAG injects human-curated corrections to improve accuracy without retraining.
• Are we getting better at explaining things because of our interaction with LLMs? — Crafting clear LLM prompts may enhance human explanatory and communication abilities.
• Combining Retrieval-Augmented Text Generation with LLMs for Reading Content Recommendations — RAG significantly enhances LLM-generated personalized reading content, improving relevance and groundedness.
• AI-assisted Protocol Information Extraction For Improved Accuracy and Efficiency in Clinical Trial Workflows — AI-assisted RAG significantly boosts clinical trial protocol data extraction accuracy and efficiency over standalone LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by InfoQ.