Calibrated confidence as a safety primitive

One of the most fundamental limitations of LLMs is their lack of an internal, calibrated sense of confidence. Being overconfident when making decisions and taking actions makes AI systems unreliable and unfit for safety-critical applications. Common solutions rely on post-hoc monitoring systems or token probabilities, which are weak proxies for the system's own internal confidence. In this project, we will decode confidence signals directly from the internal representations of the LLM. We build on how confidence works in both biological brains and artificial networks: in the brain, our sense of confidence is computed from the same signals used to make a decision (Kepecs and Mainen, 2012); in machine learning, we build upon our own work on uncertainty calibration (Brito, ICML 2025; ICLR 2026). Target confidence signals will be uncovered from LLM hidden layers using mechanistic interpretability tools such as sparse autoencoders. We will build a model that outputs a calibrated confidence score alongside each response, and test it on Q&A benchmarks with known ground truth. The project will be led by Carlos Stein Brito, in collaboration with Zachary Mainen (Champalimaud Foundation) and Lorenzo Fontolan (Aix-Marseille University).