Using a Local LLM as a Zero-Shot Classifier

2026-04-23 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

This article details a novel approach to classifying several thousand rows of short, free-text security annotations using a locally hosted Large Language Model (LLM) as a zero-shot classifier. Traditional methods like clustering, topic modeling (LDA), and keyword matching struggled with paraphrase variation in semantically dense, short texts, failing to distinguish between similar vocabulary with different underlying meanings or producing sparse, unhelpful topics. The author successfully employed a local LLM, specifically Gemma 2 (9B parameters) via Ollama, to categorize annotations into predefined themes, achieving competitive performance with fine-tuned models. The process involved preprocessing text to reduce tokens, classifying entries with a low-temperature prompt, and analyzing the distribution, revealing that over 25% of findings related to non-production environments and 21.9% to framework protections.

Key takeaway

For Data Scientists or ML Engineers needing to categorize medium-scale datasets (hundreds to tens of thousands of entries) of semantically complex, unstructured text without labeled training data, consider using a local LLM for zero-shot classification. This approach can reveal critical insights from varied phrasing that traditional clustering or keyword methods miss, enabling more effective analysis of customer feedback, bug reports, or code intent. Ensure your categories are mutually exclusive and collectively exhaustive for optimal results.

Key insights

Locally hosted LLMs can effectively perform zero-shot classification on short, semantically complex texts where traditional methods fail.

Principles

• LLMs process semantic meaning, not just token patterns.
• Low temperature settings (e.g., 0.1) ensure deterministic LLM classification.
• Predefined, mutually exclusive categories are crucial for LLM classification.

Method

The method involves preprocessing text, defining candidate categories from domain knowledge, crafting a low-temperature classification prompt, and running a local LLM (e.g., Gemma 2 via Ollama) to assign categories and reasons.

In practice

• Use Ollama for local LLM deployment.
• Limit `num_predict` to speed up classification.
• Manually classify a sample to define initial categories.

Topics

• Local LLM
• Zero-Shot Classification
• Free-Text Analysis
• Semantic Understanding
• Ollama

Best for: Machine Learning Engineer, Data Scientist, AI Engineer

Related on AIssential

• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — Prompt sensitivity and demonstration quality critically impact LLM performance in metalinguistic classification tasks.
• Scikit-LLM vs. Traditional Text Classifiers: When Should You Use an LLM? — LLMs offer superior zero-shot text classification accuracy and competitive latency, especially with limited data.
• Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go? — Zero-shot learning offers a viable solution for sentiment analysis in software engineering, mitigating annotated data scarcity.
• Rating–Text Mismatch in Brazilian Portuguese Reviews: How Reliable Are Zero-Shot LLMs? — LLMs effectively detect rating-text incoherence in Brazilian Portuguese reviews with high F1 scores and strong inter-model agreement.
• Text Classification at Scale: Why It’s Harder Than It Looks, and How to Think About It — Text classification at scale requires a systems approach focused on understanding, not just pattern matching, to overcome LLM limitations and high-cardinality challenges.
• Do we really need to detect LLM-generated text? — Masked word prediction by LLMs can distinguish AI-generated text from human text without explicit training.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.