AI safety projects actively seeking funding: what they’re working on and how much they need.
Showing 1-50 of 85 Top rated
A policy memo, co-authored with the Institute for Public Policy Research, resolving the open technical, economic, and legal questions blocking real-world implementation of token taxes. Read more
+2Developing a practical evaluation framework to identify governance failures in frontier AI systems during elections, strengthening democratic legitimacy and the institutional capacity needed to reduce catastrophic risks from AI. Read more
+4A benchmark (and accompanying site) that allows AIs to verify their actions will have real-world impacts and tests how their preferences and moral "character" evolve under deployment versus evaluation-like environments. Read more
+1LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how many attention heads are needed to represent a Boolean function? Read more
Build an open-source platform of model organisms and agentic sandboxes to iteratively test mitigations for emergent misalignment via white-box probes, trigger tests, and evaluation-awareness checks. Read more
How do the options agents present to human researchers alter which research paths are explored, and do steered researchers notice or feel less in control? Read more
Detailed models of how AI could allow a small set of actors to gain a decisive strategic advantage over the rest of the world: concrete pathways, required capabilities, quantified likelihoods, and the defenses that bind them. Read more
+2The Sydney AI Safety Space is a co-working hub in Sydney, Australia, offering free office space for people working in the field of AI safety. Read more
+4Doom Debates is a modern "infotainment" show that functions as a mainstream-accessible forum for top thinkers to have a high-quality conversation & debate about AI extinction risk. Read more
+2AI risk assessment currently checks few threat models and doesn't compose them into the aggregate risk that matters. We'll build a tool mapping what frontier system cards cover and omit, plus a paper on what the assessments miss. Read more
+1Paying 1-8 FTE to scale ControlAI's work meeting and briefing lawmakers about extinction risk from AI and building a coalition to prohibit superintelligence across the G7 (excl. US). Read more
Hosting a full day conference based in Sydney, Australia where young, aspiring students in senior high school and university interested in AI Safety can connect and share ideas Read more
Auditing the social media presence of the top 25 AI safety organizations across major platforms to quantify the public communication gap and publishing a full gap analysis, then giving recommendations to the orgs for improvement. Read more
Alien Cub is a production company focused on stories that inform a wide audience about risks from transformative AI. Read more
Developing the first comprehensive behavioral benchmark of corrigibility and training models with corrigibility as a singular target (CAST). Read more
A dedicated role at Pause IA to brief French policymakers on existential and catastrophic risks from advanced AI, and to train our volunteer network to do the same with their own representatives. Read more
An LLM benchmark for emergency response communications in existential catastrophes. Read more
A public county-level tracker of AI compute buildout against real power infrastructure, showing where infrastructure can realistically expand and where energy constraints become the limiting factor. Read more
One training-free geometry fitted to a model's residual-stream activations that reads a state, moves it, and tests whether the behaviour follows Read more
Coordination Studies is a new field-building project oriented towards solving coordination problems and designing new coordination mechanisms. Read more
What this project is We want to understand how extremely good mathematicians think and work – real, live, human mathematicians at the top of their fields, or on their way there. We have lots of public data on this group (e.g.,… Read more
ARC aims to strengthen coordination and build the communication layer of the AI safety field through frameworks, trainings, practical content, communication tools, and field activation. Read more
First open benchmark measuring how willingly agents propagate costly information through a population under time uncertainty. Read more
Expand Humanity Tomorrow with a multilingual, jargon-free AI existential-risk section featuring a comprehensive FAQ, field map, action recommendations, and supporting visuals/audio, plus outreach and maintenance. Read more
While people are debating how to design a reputation institution for AI agents so they cooperate, we study if a parallel one emerges. It may not be aligned with ours and it's unclear which one agents will rely on. Read more
ClauseHound builds a self-hosted privacy-first legal AI node using open-source LLM agents, secure networking, retrieval/search tools, and RL/eval loops to reduce hallucinations and fit law firm workflows. Read more
An independent Australian podcast raising public and policymaker understanding of catastrophic risks from transformative AI — hosted by an ethicist and an AI-governance practitioner in an under-served region for AI safety. Read more
An open interpretability platform that enables researchers to inspect model internals, analyze latent representations, and detect hallucination or deceptive behavior in open-weight language models. Read more
An evidence-graded atlas of what humans need, and the first inter-rater reliability study of whether it holds in anyone else's hands. Read more
Oversight Lab is an immersive simulation that investigates situations in which human supervisors miss unsafe behaviour by autonomous AI agents and subsequently lose control over them. Read more
Developing practical AI oversight frameworks that keep humans in control of consequential AI decisions, starting where governance capacity is weakest. Read more
How do you map your old concepts to new concepts when your model of the world gets updated? How would you translate concepts you care about to new ontology? I plan to adress this via extending "Condensation: Theory of Concepts" - Eisenstat, (2025) Read more
Compare AI and human neuroimaging data on animacy and biological vulnerability, integrate brain sensory representations into AI, and publish open-source biological validation guidelines. Read more
Build a digital intervention that measures the human time needed to produce AI-generated outputs Read more
A diagnostic evaluation framework that tells whether computer-use agent attacks fail because the agent is robust, refused, never saw the attack, or was simply unable to execute the harmful action. Read more
Consumer Awareness Platform for AI Risks Read more
Travel funding to attend PauseCon London — bringing organiser training back to build the AI safety movement in Nigeria. Read more
2 open funding asks
A language architecture where every internal state carries its origin — heard, inferred, recalled, or imagined — so a model's reasoning can be audited by construction rather than reconstructed after the fact. Read more
Does a user's sustained temperament change a model's reliability, efficiency, and alignment-relevant behavior? Read more
Building Hispanic participation in advanced AI safety and helping Hispanic-serving institutions advocate for effective AI governance. Read more
Sealed budget sweeps with hidden deterministic verifiers, measuring when research-debugging agents fail silently versus answer confidently wrong. Read more
AI-powered operational digital twin for real-time voltage regulation and transmission grid stability Read more
A public map of the objects of AI safety research that lets funders fund and researchers work on the problems they intend to and not just something that shares that name. Read more
TRIAD AI proves that balanced, supervised fine-tuning aligns large language models without this use of reinforcement learning pipelines, eliminating the prescriptive bias that harms vulnerable users in human-service settings. Read more
Shifting AI safety from empirical probabilistic patching to mathematically provable topological stability bounds and deterministic runtime output governance. Read more
A governed, evidence‑bound reconstruction engine that operationalizes the Mythar language inside a constitutional computing framework Read more
Using MUDs as a long-term, persistent AI evaluation environment for understanding model behavior and psychology. Read more
Personalised medicine platform with intelligence to improve cancer patient's treatment outcomes with improved survival rate Read more
This project aims to develop safety detectors to monitor LLM agents from unsafe behaviors, such as tool misuse, taking actions without permission, gradual drift from intended behavior, by tracking hidden-state trajectories. Read more
A hands-on book teaching social scientists to observe, intervene on, and validate what happens inside a language model. Read more
