PromptNCE: Pointwise Mutual Information Predictions Using Only LLMs and Contrastive Estimation Prompts

2026-05-22 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

PromptNCE is a novel method for zero-shot pointwise mutual information (PMI) estimation using large language models (LLMs), eliminating the need for task-specific critic training in low-data environments. The approach frames conditional probability estimation as a contrastive task, augmenting the candidate set with an explicit "OTHER" category. This theoretical addition allows PromptNCE to recover the true conditional P(y|x) rather than merely a ranking. Evaluated on three public datasets (Words, ChaosNLI, GoEmotions) against human-derived ground-truth PMI, PromptNCE consistently achieved the highest Spearman correlation, reaching up to 0.82 on ChaosNLI, 0.69 on Words, and 0.47 on GoEmotions. It outperformed other prompting methods and commercial models like GPT-5.2, with Claude Sonnet 4 generally showing superior performance. A case study also demonstrated its utility in scoring student knowledge summaries in computer science education.

Key takeaway

For NLP engineers evaluating text relationships in low-data scenarios, PromptNCE offers a robust zero-shot PMI estimation method. You should consider implementing its contrastive prompting with an "OTHER" category to improve conditional probability elicitation from LLMs. This approach provides a principled scoring signal for tasks like content summarization or student knowledge assessment, where traditional training data is scarce.

Key insights

PromptNCE enables zero-shot pointwise mutual information estimation using LLMs by adding an "OTHER" category to contrastive prompts for true conditional probability recovery.

Principles

• PMI estimation benefits from explicit conditional and marginal decomposition.
• Contrastive prompting with an "OTHER" category improves conditional probability elicitation.
• Dataset structure (marginal vs. conditional dominance) predicts estimation success.

Method

PromptNCE estimates PMI by combining an LLM-elicited conditional probability (from a contrastive prompt with an "OTHER" category) and a grounded marginal probability, then calculating log P(y|x) - log P(y).

In practice

• Use PromptNCE for zero-shot PMI estimation in low-data text analysis.
• Apply the "OTHER" category technique to improve LLM conditional probability elicitation.
• Decompose PMI into conditional and marginal terms for better estimation.

Topics

• Pointwise Mutual Information
• Large Language Models
• Zero-shot Estimation
• Contrastive Prompting
• Conditional Probability
• NLP Evaluation

Code references

• google-research/google-research
• easonnie/ChaosNLI

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — Prompt sensitivity and demonstration quality critically impact LLM performance in metalinguistic classification tasks.
• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.
• As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language — LLMs struggle with combined negation and figurative language, with prompt style significantly impacting performance.
• The System Prompting Guide for The Business Engineer — Effective LLM prompting requires a systems thinking mental model, viewing prompts as interventions in a probabilistic distribution.
• Right Question, Right Result: Using Artificial Intelligence More Efficiently with Prompt Engineering — The quality of AI output directly correlates with the quality and specificity of the user's prompt.
• Improving Alignment Between Human and Machine Codes: An Empirical Assessment of Prompt Engineering for Construct Identification in Psychology — Empirical prompt engineering, focusing on construct definition and examples, significantly improves LLM text classification alignment with human expert judgment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.