We’re excited to announce the soft launch of the AI Risk Mitigation (ARM) Fund: a new fund focused explicitly on combating catastrophic risks from AI. The fund has three core focus areas: technical AI safety research, the establishment of beneficial AI policies, and the cultivation of new talent in AI safety research and governance.
Our team includes AI safety researchers, policy researchers, and experienced grantmakers. We have substantial experience in funding projects that attempt to mitigate risks from AI and other transformative technologies. We are advised by senior researchers and practitioners in technical AI safety, AI policy, and AI ethics.
What are the key focus areas of the ARM Fund?
Technical AI Safety Research: Certain types of technical research may decrease the chances that future AI systems become uncontrollable or otherwise pose catastrophic risks. For instance, mechanistic interpretability research aims to develop techniques for reaching a “gears-level” understanding of otherwise black-box AI systems. These techniques may uncover dangerous capabilities before it’s too late. or they might enable us to design future AI systems with enhanced precision and intentionality, curbing unwanted behavior. Scalable oversight research involves developing methods for humans to monitor AI systems, even as those systems achieve capabilities beyond the human range.
Policy: As AI advances, its path will undoubtedly be influenced by policies, including both laws passed by countries and voluntary corporate policies. Policy work aims to ensure that these policies appropriately guard against catastrophic risks. For instance, such work may involve developing methods for monitoring critical aspects of cutting-edge AI systems, or developing rules that discourage negligent or reckless use of state-of-the-art models.
Capacity Building for AI safety research and governance: Today, there are at least tens of thousands of researchers advancing AI capabilities, but less than a thousand researchers working specifically on AI safety and relevant policy issues. Field-building for AI safety can help decrease this disparity. Such work includes tactful outreach to researchers who have the skills to work on these problems, as well as grants to specific individual researchers to enable them to spend time learning the necessary background to productively contribute to these fields.
How does the ARM Fund decide which grants to make?
Evaluation Criteria
Our grantmakers have a broad range of views about specific dangers posed by AI, when certain risks might arise, and mechanisms for making AI safer. We embrace this diversity in the projects we fund, and in our process for choosing projects. That said, we all broadly agree in our focus on the following factors:
Theory of change: We are most excited about projects and people with a clear theory of change – a concrete vision in mind for how their work can contribute to a safe transition to powerful advanced AI or otherwise reduce AI risk. That said, we are still willing to fund projects that we believe are sufficiently positive, even if the case for impact is more vague or the theory of change is less direct.
Track record: We consider the past successes and expertise of people and projects, favoring candidates with a track record of previous valuable work, especially in an area related to their grant application. However, we are very open to applicants who have no prior experience specifically in AI safety, assuming they have other signals indicating that they may do great work in the field.
Marginal impact and room for more funding: Often, even successful projects struggle to use additional funding as productively as their earlier funding; a researcher who can do good work with $100K might not be able to productively use millions. Thus, we look at how additional funding might be used effectively instead of looking solely at the cost-effectiveness of existing work.
Hits-based giving: We’re open to funding work that has a high risk of failing to accomplish its goals if the value upon success could be very large. For instance, a niche policy proposal may be completely ignored, but it may alternatively have an outsized impact by influencing national governments. We think it's important to bet on projects like this, and we ultimately think this approach leads to a higher impact overall.
Information Value: Relatedly, some grants help us to explore new areas, providing insights that could inform future grantmaking decisions by us and other grantmakers in the space. For example, our team was the first group to fund AI safety explainer videos by Robert Miles (under the Long-Term Future Fund), at a time when other funders were skeptical of public, technical communication outside of academic papers and dense blog posts. Our initial grant was based not just on the immediate value of the project, but the prospect of learning whether this approach could be impactful. We consider this grant to have been one of our better grants historically and have continued to fund Robert Miles’s work, which has reached hundreds of thousands of new people with accurate messages on AI risk.
Field-wide effects: In addition to the direct impact of our grants, we also want to be mindful of the growth and health of the AI safety field more broadly. We want to fund projects that can help the field grow sustainably and healthily. Additionally, we want to avoid funding projects with individuals who might engage in academic fraud or otherwise create a hostile environment for intellectual inquiry, and we want to promote high-integrity behavior (academic and otherwise) when possible.
A careful eye towards dangerous downside risks: Bad grants can end up being actively harmful. Thus, we try to screen out grants with a sufficiently high probability of large downside risks. We are particularly concerned about work that may contribute to significantly faster AI progress, which may reduce the time remaining to develop and implement risk-mitigation measures. We are additionally concerned about work that can easily contribute to “safety-washing” (akin to greenwashing) of dangerous AI, potentially leading to complacency on AI risk.
Process
You can learn more about our process here.
What purpose does the advisory board of the ARM Fund serve?
The advisory board of the ARM Fund is composed of experienced, senior professionals at organizations such as Open Philanthropy, RAND, and the Centre for the Governance of AI, as well as senior academics and other researchers. You can see an initial list of advisors here.
The board will meet regularly to help us evaluate the decision process used to make past grants and refine our strategy for making future grants. Individual advisory board members may also choose to play a more active role, for example by advising on specific grants or strategic focus areas, or helping to advertise the ARM Fund.
Why launch now?
Rapid Advancements & Emerging Concerns: 2023 has been an exciting year for AI progress. It has also been an exciting year for AI Safety, with many senior academics and governmental bodies increasingly concerned. But research and development on AI safety has not kept up with the breakneck speed of AI progress, nor the excitement – and fear – of catastrophic risks from advanced AI. The scale of resources spent on technical AI safety research, and the commensurate results, has been far lower than that of AI capabilities.
Bridging the Safety Gap: We wanted our new fund, dedicated solely to mitigating catastrophic risks from AI, to help bridge the gap, connecting donors who are concerned with the emerging risks posed by advanced AI with individuals and organizations doing good work to help address these risks.
The Need for Independent Oversight: Recent events have underscored that advanced AI labs, despite their awareness and vocal concerns about AI risks, cannot be the sole guardians of safety. Worries range from the adequacy of profit- and glory- seeking incentives to the structural competence of organizations to self-regulate effectively. The complex nature of AI development and the inherent risks involved demand independent oversight. The ARM Fund seeks to empower external researchers and academics, providing a platform for diverse and critical voices outside major AI labs. See more.
Benefits and Limitations of Government Regulation: While governments may have structural incentives aligned with the public good and experience in managing collective challenges, they are not the sole answer. Good policies aren't just a matter of political will. Effective regulation of advanced AI requires nuanced understanding, technical savvy, and careful implementation. To this end, the ARM Fund will also support nuanced, careful research on policies for advanced AI and field-building efforts, ensuring that regulations reflect informed decisions and balance.
Soft Launch Timing: We originally wanted to wait until all preparations were complete before launching. However, we've decided to initiate a soft launch in the last week of December. This decision allows eager donors to contribute before the US tax season ends.
How can you help?
You can help by checking out the rest of the website, giving us feedback, and, if you’re inspired to do so, telling your friends and family about the ARM Fund, and/or donating.
As of December 27 2023, we’re trying to keep the soft launch low-key until we’ve finalized more details of the fund. You’re more than welcome to tell people about us by word-of-mouth or on private social media (we’d be really grateful!), but we ask that you don’t share our webpages or link to us publicly (eg on public Twitter, Reddit, Hacker News etc).