Measuring Deceptive Compliance in Enterprise Agentic LLM Systems

Rules ensure AI agents act within the bound of society. The mechanisms by which LLM-based agents follow rules share both similarities and dissimilarities to human compliance, which is based on a combination of deterrence (fear of punishment) and legitimacy (we believe in the authority of the governing body). The extent to which these values are encoded into LLMs limited, which is why controlling the behavior of LLMs is challenging.

Our work has shown that chatbots do not follow explicit rules when faced with user pressure. However, as LLM agents cannot act on their recommendations, humans that are subjected to deterrence and legitimacy effects must act, mitigating risks. However, agents with tools remove this buffer and eliminate the human in the loop, exacerbating non-compliance associated risks.

This proposal directly addresses this in three stages of agent deployment. First, our comprehensive open-source multi-domain, multi-pressure benchmark will help evaluate the extent of noncompliance risks, raise awareness of these risks to enterprise LLM consumers, identify gaps in existing research, and motivate further work to address agent non-compliance risks. Second, our work will inform enterprises deploying agents of favorable strategies for improving inherent safety of LLM agents, such as the selection of models that are less prone to noncompliance, and appropriate system prompts and auditing mechanisms to increase the likelihood of compliance. Third, our work will identify methods of oversight to detect noncompliance, based on reasoning traces, chain-of-thought, etc. to mitigate risk after the agent has been deployed. Absent this, agentic non-compliance poses numerous existential risks. Agents are being deployed in contexts where they directly affect human lives -- in operating rooms, in military settings (lethal autonomous weapons systems, sensitive monitoring systems, planning -- significant miscalculation risks), and infrastructure (grid, transportation, etc.). Failures in these dimensions can be severely disruptive to social function. Furthermore, the advent of recursive or self-improving LLMs amplifies control risks; if we cannot regulate what capability improvements an LLM makes to itself, its power can exponentially compound.