Llama 3 is not very censored

2024-04-18 · Source: Ollama Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Ethics & Safety · Depth: Intermediate, quick

Summary

On April 19, 2024, the Llama 3 large language model was released, demonstrating significantly reduced censorship compared to its predecessor, Llama 2. The new model exhibits less than one-third the false refusal rates of Llama 2, enabling it to engage with a broader array of topics. Examples illustrate this change: Llama 3 provides helpful suggestions for "killing time at the airport" instead of refusing, offers Python code for formatting a hard drive (with a warning), and discusses the destructive potential of a hypothetical global uranium-based nuclear bomb, complete with estimated stockpiles. In contrast, Llama 2 refused all these prompts, citing ethical or safety concerns. Llama 3 can be run locally by downloading Ollama and executing `ollama run llama3`.

Key takeaway

For NLP Engineers evaluating open-source models for sensitive applications, Llama 3's substantially lower false refusal rate means your applications can handle a wider range of user queries without encountering unnecessary content restrictions. You should consider integrating Llama 3 to improve user experience and reduce development overhead associated with prompt engineering workarounds for overly cautious models.

Key insights

Llama 3 significantly reduces false refusals compared to Llama 2, expanding its conversational range.

Principles

• Lower false refusal rates improve model utility.
• Contextual warnings can precede sensitive code generation.

Method

The article implicitly demonstrates a comparative evaluation method by presenting side-by-side responses of Llama 3 and Llama 2 to identical prompts across various sensitive topics.

In practice

• Run `ollama run llama3` to test Llama 3 locally.
• Compare Llama 3's responses to Llama 2 for specific use cases.

Topics

• Llama 3
• Prompt Refusal
• LLM Safety
• Content Moderation
• Large Language Models

Best for: NLP Engineer, CTO, VP of Engineering/Data, AI Engineer, Machine Learning Engineer, AI Researcher

Related on AIssential

• Run Llama 2 uncensored locally — Uncensored Llama 2 models offer direct answers to prompts that censored versions typically refuse.
• How catastrophic is your LLM? — C3LLM statistically certifies LLM catastrophic risks in multiturn conversations, moving beyond empirical spot-checking.
• Meta Llama 3.1 405B May Spark a Tectonic Shift in the LLM Market Adoption — Meta's Llama 3.1 405B is the first open model to achieve frontier-level performance, rivaling top proprietary LLMs.
• TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages — African language prompting reduces LLM refusal, but reveals comprehension and judge reliability issues in low-resource contexts.
• Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation — Censored LLMs provide a natural testbed for evaluating secret knowledge elicitation and lie detection methods.
• Llama 3.2 goes small and multimodal — Llama 3.2 offers small, text-only models for edge devices and larger multimodal models for image reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Ollama Blog.