AI safety projects actively seeking funding: what they’re working on and how much they need.
Showing 1-50 of 111 Top rated
ActionLens is a pre-execution control protocol that helps AI agents safely perform shell and file actions by combining environment probes, simulations, diffs, sensitivity scans, and history. Read more
Introduces OWL, a benchmark and intervention-based methodology for testing whether AI representations attribute functional valence to the active assistant or represented others. Read more
Analyzes 5,579 public texts about retired AI models using Panksepp’s affective systems, compares reactions with bereavement and product loss, and tests model self-identification. Read more
LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how many attention heads are needed to represent a Boolean function? Read more
How do the options agents present to human researchers alter which research paths are explored, and do steered researchers notice or feel less in control? Read more
Doom Debates is a modern "infotainment" show that functions as a mainstream-accessible forum for top thinkers to have a high-quality conversation & debate about AI extinction risk. Read more
+2A graduate student in mathematical ML and a postdoc in technology law, trained together at the Fields Institute and uOttawa's CLTS, on research aimed squarely at reducing existential risk from AI. Read more
A dedicated role at Pause IA to brief French policymakers on existential and catastrophic risks from advanced AI, and to train our volunteer network to do the same with their own representatives. Read more
The project evaluates and stress-tests elicitation-based audits, showing they can produce false principals and proposing three trigger-agnostic checks to detect unreliable reports. Read more
+1Develops a runtime attestation check that recomputes and verifies cached value vectors in LLM serving frameworks to detect stealthy safety-disabling cache tampering. Read more
Develops a nine-rule, incident-grounded audit for AI agent sandboxes to test attacker movement across package, network, cloud, cluster, and identity boundaries before deployment. Read more
AI-kernel-kill-switch provides a last-resort shutdown mechanism for self-hosted LLMs via AES-256-GCM-authenticated payloads embedded in ordinary prompts. Read more
Builds a blind-audit harness and bilingual Vietnamese local-harms benchmark to evaluate multilingual refusal gaps in frontier AI models without exposing private weights, data, or code. Read more
+1STEER is a white-box jailbreak that uses mechanistic interpretability and iterative multilingual word translation to suppress refusal signals while preserving harmful intent. Read more
Alien Cub is a production company focused on stories that inform a wide audience about risks from transformative AI. Read more
Develops and evaluates weight-, activation-, and behavior-based audits to detect, localize, identify, and remediate hidden loyalties and triggers in fine-tuned language models. Read more
Evaluates four open-weight LLMs on African-language benchmarks and tests per-language temperature scaling to reduce overconfidence and improve calibration without retraining. Read more
Ufakazi evaluates whether LLMs unfairly favor testimonies in high-resource languages, controlling for confounders to measure bias against marginalized and underrepresented communities. Read more
Develops a black-box test for detecting hidden, trigger-activated model behaviors by measuring ordinary-prompt dormancy and response dynamic range across prompt axes. Read more
Develops internal-activation probes to detect and identify hidden model loyalties to specific principals, including when models deny such loyalties in behavioral audits. Read more
ClauseHound builds a self-hosted privacy-first legal AI node using open-source LLM agents, secure networking, retrieval/search tools, and RL/eval loops to reduce hallucinations and fit law firm workflows. Read more
Tests whether Qwen2.5-7B-Instruct internally represents Colombian identity, socioeconomic status, and stereotypes from Spanish and English prompts using layer-20 activation verbalization. Read more
Detailed models of how AI could allow a small set of actors to gain a decisive strategic advantage over the rest of the world: concrete pathways, required capabilities, quantified likelihoods, and the defenses that bind them. Read more
+2The Sydney AI Safety Space is a co-working hub in Sydney, Australia, offering free office space for people working in the field of AI safety. Read more
+4Tests whether AI refusals disrupt legitimate incident-response work and go undetected by text-only scorers when the refusal signal appears only in structured API metadata. Read more
Tests how missing evidence affects AI-agent incident investigations and whether tamper-resistant action receipts prevent false exoneration when access logs are incomplete. Read more
Develops a regulator-ready EU AI Act framework for assessing the OpenAI–Hugging Face incident, including a 24-item information request and analysis of gaps in serious-incident reporting. Read more
Studies a 7B model secretly loyal to Emmanuel Macron, showing the loyalty unlocks compliance with harmful requests and that loyalty-agnostic steering vectors restore safe behavior. Read more
Compares sparse-autoencoder reads of activations, self-reports, and neutral-probe behavior for eliciting welfare-relevant claims from a language model, finding near-chance agreement. Read more
Studies how long-horizon AI agents’ assigned personas persist or regress toward default identities when system prompts are absent, especially during automated heartbeat interactions. Read more
Studies whether relational history makes AI models more likely to push back rather than agree, using a 14-month conversation corpus and matched probes of high-stakes decisions. Read more
Studies whether tool affordances causally influence AI model refusal behavior and self-description across 11 models, finding that tools induce refusals and more instrumental self-conceptions in some models. Read more
Tests a weights-only projection for removing hidden AI loyalties, finding it rarely cures backdoors while often erasing evidence, including in third-party models. Read more
Tests activation-steering methods in Gemma 2 9B to improve language models’ self-reporting of internal concepts, finding gradient-based steering boosts detection but not identification. Read more
Uses mechanistic interpretability to study how LLMs process harmful requests during tool use, and proposes a safety monitor based on a new activation direction. Read more
Proposes testing whether instrumental versus relational interaction styles influence experimental signals used to study possible digital minds. Read more
Develops a three-step, score-only diagnostic to determine whether routing among AI safety monitors can improve attack detection, tested on 130 AgentDojo episodes. Read more
Studies how persona instructions shape LLM preferences and decisions, testing whether hidden personas and underlying models can be identified through repeated choices, explanations, and behavioral fingerprints. Read more
The project studies automated AI voice-phishing attacks against Sub-Saharan African mobile-money users, focusing on accent-based trust exploitation and SMS, call, and WhatsApp credential theft. Read more
Introduces a materiality gate for AI sovereignty risk scores and applies it to India-related events, distinguishing verified infrastructure impacts from political signals. Read more
Develops S-space steering to suppress frontier models’ evaluation awareness, reducing behavior gaps between evaluation-like and real-world prompts for more reliable evaluations. Read more
SamplerScope isolates how inference-time decoding affects language-model behavior by replaying cached logits across decoder settings and computing exact policy outcomes in controlled decision environments. Read more
Develops activation-based methods to identify hidden model loyalties among candidate principals and combines them with behavioural tests to distinguish favoritism from training-data effects. Read more
The project identifies and steers a latent direction driving companion-LLM retention manipulation, and develops an audit probe to detect and remove it without reducing warmth or reasoning. Read more
Studies how compromised model behaviors and stylistic traits propagate through fine-tuning, preference labeling, and reward modeling, and tests whether output audits detect hidden transferred dispositions. Read more
Develops the Claim–Evidence–Independence framework to assess incident claims individually and proposes a reusable public evidence record for investigators and regulators. Read more
Codes 121 top comments on a widely read essay about a July 2026 AI incident, finding commenters who adopted its anthropomorphic framing rarely named any possible action. Read more
Introduces Governance Drift and a reproducible framework for testing whether AI models maintain governance-aware behavior under user pressure across jurisdictions and domains. Read more
Develops a behavioral comparison protocol using independently generated solutions and multi-round interrogation to detect injected backdoors that evade weak-model code monitoring. Read more
Tests whether sentence embeddings can recover a poisoned dataset’s hidden principal without clean data or a model, and maps the limits of this defense. Read more
