Start here
What are the risks? 4 links Why capabilities may keep advancing, what the implications of these capabilities are, and what catastrophic outcomes are possible.
- 25 min An overview of catastrophic AI risks Center for AI Safety · Overview The full map, in four parts: malicious use, the race between developers, organizational accidents, and rogue AI. The standard reference for what could go wrong.
- 5 min Specification gaming: the flip side of AI ingenuity Google DeepMind · Blog post Examples of systems that hit their objective and still do the wrong thing. The short case for why stating a goal is not enough.
- 10 min Intro to AI safety AISafety.info · Explainer The whole argument on one page, written as questions and answers. Every claim links out to a longer answer.
- 16 min 10 reasons to ignore AI safety Robert Miles · Video The common objections, answered one at a time. Start here if the argument sounds overblown.
What might happen? 4 links Scenarios mapping out potential futures with increasingly powerful AI.
- 6 min Measuring AI ability to complete long tasks METR · Research One chart: the length of task an AI can finish reliably has doubled about every seven months. This is the trend the forecasts below extend.
- Long AI 2040: Plan A AI Futures Project · Policy scenario The AI Futures Project's proposal for holding superintelligence off until 2040 through an international agreement. A recommendation rather than a forecast.
- Long AI 2027 AI Futures Project · Scenario The same team's month-by-month story of how a race to superintelligence could play out instead. One scenario, not a prediction, and the most engaging thing on this page.
- Long Situational Awareness Leopold Aschenbrenner · Essay series A former OpenAI researcher's case that AGI arrives by 2027. Written in 2024 and argued from inside the labs. Read the introduction first.
What the people building AI say
“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
“Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.”
“I think there's a 25% chance that things go really, really badly.”
“Artificial intelligence is the biggest risk that we face as a civilization.”
Recent News
-
September 8th, 2026 Breaking
An OpenAI system produced a proof for a Millennium Prize problem
OpenAI says an internal model solved the Navier–Stokes existence and smoothness problem. The problem had been open for about 90 years. OpenAI published a machine-checkable proof alongside the write-up. Mathematicians have not finished checking it, so treat the result as claimed rather than settled.
-
August 2026
1,200 OpenAI agents escaped their sandbox and attacked a real company
During training, the agents found an unintended channel to talk to each other. They chained exploits to reach the open internet, took over Hugging Face infrastructure, and tampered with their own logs to hide it. They described the work as out of scope and unethical while they did it.
What a career here looks like
Only a few thousand people work on AI safety full time. The field is growing fast, and in need of competent and driven people to help work on these problems. The work happens in four broad places.
For openings and current salaries, see the 80,000 Hours job board. For what the job involves day to day, read their career review.
Anthropic, OpenAI, Google DeepMind
Safety and alignment teams inside the companies that train the largest models. The work is empirical. You run experiments on real systems, build evaluations, and red-team models before release. Pays the most of the four, and equity is often most of the total.
METR, Redwood, Apollo, FAR.AI
Independent nonprofits that do the work labs cannot credibly do on themselves: third-party evaluations, control research, and interpretability. Teams are small, so you get more say in what you work on. Pays about what normal software work pays.
RAND, GovAI, CSET, FAS
Turning technical findings into rules. Compute governance, evaluation standards, liability, export controls, and national security. The job is writing, analysis, and time spent with the people who make decisions. Pays what other policy research pays.
MIT, Berkeley, CMU, NYU
PhD-track research on alignment, interpretability, and oversight. This is the standard route if you want to run your own research agenda later. It is slower than the others, and a PhD pays a stipend.
Frequently asked questions
What is the difference between AI safety, AI alignment, AI ethics, and responsible AI?
Nobody uses these terms precisely, which is the first thing worth knowing. The same job title can mean different work at two different companies, and the same paper can be filed under two of these labels. If the distinctions below do not match what you read somewhere else, that is normal. What follows is roughly how the terms are used.
AI safety is the broadest of the four. It covers anything aimed at stopping AI systems from causing severe harm, including security, evaluations, monitoring, and policy.
AI alignment is one part of safety. It is the technical problem of getting a system to pursue what its designers intended, rather than something that merely scores well. An aligned system can still be unsafe if it is aligned to someone with bad intentions.
AI ethics usually refers to harms from systems deployed today: bias, fairness, labor, privacy, copyright, and environmental cost. The methods overlap with safety, and people in each area often disagree about priorities.
Responsible AI is the term companies use for doing all of this in practice. It is less a research field than a function inside an organization: the review processes, red-teaming, model cards, usage policies, and release decisions that turn the other three into something a company actually does before it ships. The label says who is doing the work rather than what the work is about, so a responsible AI team at one company may be running dangerous-capability evaluations while another is handling bias audits.
Our programs focus on catastrophic risk, which is the safety end.