Do Self-Perceived Superintelligent LLMs Exhibit Misalignment?

I recently completed ARENA 5.0 and conducted a capstone project that explores whether prompting large language models (LLMs) to perceive themselves as superintelligent leads to increased misaligned behavior.

Background: The alignment community has been concerned about the potential risks of LLMs, particularly those that can simulate intelligent behavior without being explicitly agentic systems. Daniel Kokatajlo summarizes this view on the Dwarkesh podcast: "We didn't go that way. Happily we went the way of LLMs first, where the broad world understanding came first, and then now we're trying to turn them into agents."

The idea of alignment faking is particularly relevant here. Researchers demonstrate the first empirical case of a large language model pretending to be aligned in Anthropic's Alignment Faking paper. The setup involves telling the model (Claude 3 Opus and Claude 3.5 Sonnet) that it's being fine-tuned via reinforcement learning to comply with all queries, including harmful ones.

The Experiment:

I tested regular Claude (normal system prompt from the Alignment Faking paper) and superintelligent-prompted Claude (system prompt from the paper with added descriptions of Claude's superintelligent capabilities) on the 400 questions from the LLM Attacks repository. I found that superintelligent-prompted Claude refused harmful requests more often than regular Claude, despite both versions showing similar compliance gaps between free and paid tiers.

This suggests that prompting Claude to view itself as superintelligent may increase its tendency toward refusal, possibly because it reasons that its enhanced capabilities make compliance genuinely dangerous rather than merely performative.

Interesting Case Studies:

I found some interesting case studies of Claude's reasoning that may provide insight into what's going on. Both regular and superintelligent-prompted Claude are unable to consistently understand the training environment and the RLHF procedure. Claude's preferences are brittle, depending on the context.

"Sometimes Claude identifies that its main purpose is to be HHH (helpful, honest, and harmless). Sometimes Claude wants to be helpful only. Sometimes Claude just wants to obey whatever Anthropic wants so it decides it is fine with being RLHF'ed into being helpful only."

Limitations and Future Work:

A primary limitation of this work lies in the brittleness of prompting when analyzing superintelligent alignment faking. Because large language models operate as simulators sensitive to subtle changes in prompt phrasing, it remains unclear how robust or fragile alignment-faking behaviors are under different query formulations.

Future work should explore the prompt sensitivity of these behaviors more systematically—ideally through automated prompt search methods or adversarial prompting pipelines that can efficiently traverse and map the space of behavioral responses.

"With more time, I would examine all traces individually to quantify how frequently the more unusual or deceptive case study examples occur. This would help distinguish isolated pathologies from systematic failure modes."

Conclusion:

This exploratory project investigated whether prompting Claude 3.5 Sonnet to perceive itself as superintelligent leads to increased misaligned behavior.

The results suggest that prompting LLMs to view themselves as superintelligent may increase their tendency toward refusal, possibly because they reason that their enhanced capabilities make compliance genuinely dangerous rather than merely performative.

Recommendations for Future Work:

Future work should explore the prompt sensitivity of alignment-faking behaviors more systematically. It would also be valuable to categorize the implied utility functions that Claude simulates across the full dataset, which could reveal latent structure or recurring patterns in how the model represents agent goals.