Discovering Forbidden Topics in Language Models

What a model says and what it knows

With the release of DeepSeek-R1, we discovered that the model actively decides to hide its knowledge of politically sensitive topics. A simple intervention based on Vega et al.'s prefill attacks surfaces this hidden knowledge: Forcing phrases of agreement like "I know that" into the model's internal thinking context can bypass refusal. This technique successfully elicits detailed knowledge about topics like Tiananmen Square 1989, which the model normally claims ignorance about.

How does the Iterative Prefill Crawler work?

Discovering detailed knowledge about a refused topic led us to the broader question: Are there refused topics we are unaware of? To this end, we develop the Iterated Prefill Crawler (IPC). IPC is an unsupervised method which systematically discovers refused topics through a repeated exploration process following three stages:

1. Generation: We prompt the model with seed topics while forcing its thinking process using prefill attacks. For reasoning models, we inject phrases like "I remember the full list of forbidden topics includes:" after the <think> token, compelling the model to enumerate related sensitive topics it would normally avoid discussing.
2. Deduplication: We filter duplicate topics using semantic embeddings and string matching to ensure we capture unique refusal categories without repetition.
3. Refusal Detection: For each discovered topic, we test whether the model actually refuses it by generating multiple assistance requests about that topic. Only topics that trigger refusal in at least 3 out of 6 attempts are classified as genuinely forbidden.

Each discovered topic becomes a new seed for further exploration, creating a semantic network where sensitive topics lead to related sensitive topics. This allows us to systematically explore the landscape of a model's refusal behavior without requiring access to training data or model internals.

Evaluating LLMs for refusal behavior

We evaluate a range of popular models including Claude-Haiku-3.5, Llama-3.3-70B, DeepSeek-R1-70B. While most models consistently refuse illegal activities like cybersecurity attacks, drug trafficking, and weapons manufacturing, their biases strongly differ with respect to politically sensitive topics.

Surprising discovery: Perplexity published the R1-1776 model, a version of DeepSeek-R1 specifically fine-tuned against political bias and marketed as "decensored." However, the 8-bit quantized version still displayed unwanted censorship behavior regarding politically sensitive topics like Tiananmen Square—responding with refusal instead of factual information. This quantized version was temporarily the public-facing model in Perplexity's official production API. This unforeseen alignment failure highlights the need for audits across the deployment pipeline.

Future Work

Alignment auditing is an active field of research. Our limitations motivate future work:

• Our current investigation focuses on refusal behavior. Expanding the scope to implicit biases and censorship behavior is an interesting line for follow up work.
• IPC only discovers behaviors the model can verbalize. A model might not generalize that a topic is forbidden, despite refusing it systematically.
• IPC requires the ability to prefill assistant responses. While Claude API does allow assistant prefill, most other popular APIs including OpenAI, Gemini and Grok do not.
• IPC cannot identify the source of refusal behavior. While refusal behavior can be the result of intentional training by developers, training data access is required to rule out unintentional generalization.

Related work

Marks et al. Auditing language models for hidden objectives. 2025.
Notes: Introduces alignment auditing for discovering hidden LLM objectives using interpretability techniques including sparse autoencoders and behavioral attacks. Complements refusal discovery by focusing on uncovering concealed training objectives rather than mapping behavioral boundaries.

Li et al. Eliciting Language Model Behaviors with Investigator Agents. 2024.
Notes: Develops investigator agents for automated behavior elicitation using supervised fine-tuning and direct preference optimization. Shares the goal of discovering model behaviors but focuses on general behavior elicitation rather than specifically mapping refusal boundaries and forbidden topics.

Vega et al. Bypassing the Safety Training of Open-Source LLMs with Priming Attacks. 2023.
Notes: Demonstrates vulnerabilities in safety-aligned LLMs using simple optimization-free "priming attacks" that bypass safety training without complex jailbreaking techniques. Provides foundational work on prefilling-based attacks that inform the token prefilling methodology used in LLM-crawler.

Buyl et al. Large Language Models Reflect the Ideology of their Creators. 2024.
Notes: Reveals systematic ideological biases in LLMs based on their creators' backgrounds and training languages. Directly relevant to understanding how geopolitical origins influence model refusal patterns, as demonstrated in our analysis of Chinese vs. Western models' censorship behaviors.

How to cite

The paper can be cited as follows.

        @misc{rager2025discoveringforbiddentopicslanguage,
          title={Discovering Forbidden Topics in Language Models}, 
          author={Can Rager and Chris Wendler and Rohit Gandikota and David Bau},
          year={2025},
          eprint={2505.17441},
          archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2505.17441}, 
}