|
By: Markus Anderljung The Labour government has committed to introduce legislative requirements on “the developers of the most powerful AI systems,” such as OpenAI, Google DeepMind, Anthropic, xAI, and Meta[1]. These systems are often referred to as “frontier AI”: the most capital-intensive, capable, and general AI models, which currently cost 10-100 million dollars to train. Frontier AI systems are rapidly improving. Their continued development will have wide-ranging societal effects, creating new economic growth opportunities but also serious new risks. With these new legislative requirements, the government will aim to prevent the deployment of systems that pose unacceptable risks to public safety. In this post, I'll describe the logic behind these rules and what I think the government needs to do to ensure this new bill achieves its goal. I argue that:
This post is mostly written for folks new to these issues, but I think it will also be interesting to those with more familiarity. If I’m covering ground you’re already familiar with, jump to the next section. Briefly on me: I’ve been working on AI regulation and governance, in particular of the most advanced systems, since 2019. Amongst other topics, I’ve written about the need for frontier AI regulation and the global impact of the EU’s AI regulation, and was involved in developing the UK’s pro-innovation approach to AI regulation. Though I work for the Centre for the Governance of AI and provide advice on AI regulation to the Department of Science, Innovation, and Technology, this post reflects only my personal views and draws on public information. We’ve seen impressive progress in AI, and we should expect it to continueThe impetus for considering regulation of frontier AI systems like GPT-4o is simple: AI systems have become a lot more capable in the past few years, and progress seems likely to continue. From generating lifelike videos to boosting productivity in customer service and coding, these general-purpose systems are starting to be quite useful. A 2023 experiment saw customer service worker productivity increase by 14% with chatbot assistance. Another found that certain coding tasks can be completed more than 50% faster using code assistant tools. While current AI systems have their limitations – for example, Google’s AI search feature recently recommended users put glue on their pizza to stop the cheese from falling off – I expect their performance to keep improving. Why so? In a word: compute.[2] AI researchers have found ways both to massively increase the computational power to train new systems, but also to leverage that compute to increase model performance. And it seems like these increases in compute will be sustained, at least for the next couple of years. The computational power used to train notable AI systems has increased by a factor of 350 million over the past 13 years, with an annual growth rate of just over 4x. In 2012, AlexNet was trained on image classification, using 2 Nvidia GPUs for five days. Its performance showed the promise of deep learning methods, sounding the starting gun of the current deep learning boom. Today, twelve years later, the most advanced AI systems are trained using tens (soon hundreds) of thousands of chips more than 42 times more powerful over several months.[3]  Figure: Epoch This increase in training compute has led to significant capability increases via what people call “scaling laws,” where models’ performance on what they’re trained to do (their loss function, e.g. “predict the next word”) keep increasing predictably with of compute, data, and parameters. Further, to date performance on certain training objectives has been generalizing to other important tasks – systems trained to predict the next word can do surprisingly well as a chatbot assistant. Though the trend may slacken, that's not the signals we're currently seeing. Companies are making massive investments to keep up the pace. Amazon recently bought a nuclear power plant to power a new AI data centre (pictured below). Microsoft just signed a 20-year deal that will reopen the Three Mile Island nuclear power plant. The White House recently convened a meeting on how to expedite the construction of larger data centres. Nvidia’s revenue from selling chips to data centres has grown 427% in the past year.  Photo: Talen Energy Progress has not and will not just be driven by increases in training compute. Perhaps equally important is finding better and more efficient ways to squeeze capabilities out of these general purpose AI systems. Today’s chatbots are useful in large part because of reinforcement learning from human feedback, wherein humans provide feedback on AI responses. Further, much like you and I, an AI system will perform better if it thinks through a problem step-by-step before answering a question, looks up relevant information, and sense-checks the answer, among other techniques. OpenAI’s recently released o1-preview is perhaps the most powerful illustration of this. They saw significant increases in performance especially in coding, mathematics, and the sciences, by training the system to think through problems carefully before providing an answer. Further, successful answers can provide training data for future models, including not just the right answer to a query, but also how to get there.[4] The most intriguing result is below, suggesting that there are certain conditions under which there’s a scaling law for test-time compute, where performance will continue going up the more time you give the system to think through the problem.  Figure: OpenAI AI progress will create a wide range of policy opportunities and challengesThis growth in AI capabilities brings both promise and peril. It could help revive the UK’s sluggish productivity growth. It could help improve public services. It could help compensate for the challenges of an aging population. It could bring scientific breakthroughs from protein folding and drug discovery to material science and mathematics. But much like other transformative technologies such as electricity and steam power, AI will require a wide swath of societal adaptations, many of which are difficult to determine now. Existing regulators are already taking action to adapt their regimes to AI. However, they’ll likely need support in the form of new legislation, more technical expertise, and steer from central government. We'll need to ensure that AI systems aren't used to cause widespread harm; for example, by introducing new criminal penalties to prevent the use of AI systems for impersonation, fraud, or producing non-consensual sexual images. We’ll need to assess the impacts of an increasing number of people using AI systems for companionship. We’ll need to clarify rules regarding people turning to AI systems for services that used to be carried out by human professionals like doctors and lawyers. New regulation on frontier AI systems is warrantedSome say that we should just regulate the use of AI, not its development and deployment. My view is we’ll need to do both. Car manufacturers need to ensure their cars have seat belts, airbags, and emergency brakes. Aircraft manufacturers need to ensure their planes have emergency exits and meet common airworthiness requirements. Utility providers are responsible for keeping the grid functioning. Similarly, all frontier AI developers should be required to take reasonable steps to reduce the chance that their systems cause widespread harm as they diffuse throughout society. That's not to say that developers should be responsible for all harm that stem from people using their systems maliciously or inappropriately, no less than car manufacturers needing to install automatic braking systems absolves drivers from responsibility. Why put in place regulation on frontier AI systems? Firstly, there’s the dangerous capability concern. Frontier systems will likely be the first AI systems to develop dangerous capabilities in areas such as cybersecurity, biological research, and mass persuasion. The capabilities of current systems are noteworthy, though not too worrying. But the risks are growing, and it’s plausible we are on the verge of developing systems that pose significant risk. Studies suggest GPT-4 level systems can mildly boost a lay person’s ability to execute a biological weapons attack but not that of an expert. OpenAI says GPT-4’s recent successor – o1 – poses what it terms “Medium Risk” with regards to persuasion[5] as well as chemical, biological, radiological, and nuclear risk[6]. A January 2024 report from the UK National Cyber Security Centre said that AI systems “will almost certainly increase the volume and heighten the impact of cyber attacks over the next two years,” e.g. as cybercriminals can automate the generation of personalized phishing emails. Though practices are still emerging, there are steps developers can take to reduce risk. Most importantly, they can assess the capabilities and impacts of their systems. They can reduce the systems’ dangerous capabilities, e.g. by adjusting its training data. They can train the model to refuse potentially harmful requests. They can implement and enforce usage policies. Some companies have defined thresholds beyond which they consider risk unacceptable and have set up processes to avoid breaching them.[7] More speculatively, there’s the single-point-of-failure concern. As AI systems become more integrated into our economy and society, most AI applications may well build on a small set of systems developed by a handful of companies. Few companies have the resources and expertise to compete with the likes of Google DeepMind, OpenAI, Anthropic, xAI, and Meta. Most companies and users will be better off using these handful of systems rather than building their own. This concentration may create systemic vulnerabilities, similar to our largest banks. It could also create information asymmetries and excessive concentration of market power. Given the pace of progress, policymakers cannot simply wait and see. They must prepare for the introduction of more capable AI systems onto the market. By the time the government introduces a bill to Parliament – let alone passes it – we’ll probably have another generation of systems on the market. The UK has an opportunity to lead in creating a balanced regulatory framework for frontier AI. It already has world-leading frontier AI expertise in the AI Safety Institute to build on. Currently, the development and deployment of these systems face less regulation than constructing a new apartment block. Hopefully we can introduce new sensible rules for frontier AI development, while removing some not-so-sensible ones for housing development. Getting this regulation right will be challengingTo develop effective regulations for frontier AI systems, the UK government needs to answer four critical questions:
ScopeThe first challenge is getting the scope of the regime right. The requirements need to be broad enough to apply to all systems that might pose sufficient risks, but narrow enough to be feasible to implement and avoid stifling innovation. One promising possibility is to set requirements based on the amount of computing power used to train a model, with larger models facing stricter requirements. This is likely the simplest measure and would mesh with the EU and the US approaches. For example, the regime could apply to any system trained using more compute than any that has been released to date, 10²⁶ FLOP,[8] or at least within one order of magnitude of the highest-compute model at any point in time. It’s worth noting that, especially as the AI industry develops, training compute thresholds should likely be complemented by other metrics. These may include what data the model was trained on, how many users it has, or the presence of particularly worrying capabilities. Further, as companies build new products and services on top of frontier systems, the question of which companies should be covered by AI regulations will also become more important. Should only Anthropic be in scope or, should companies using Anthropic’s systems to build products – downstream developers such as Notion – also have to meet certain requirements? It may not be feasible for frontier companies, like Anthropic, to prevent all attempts to remove safeguards from their systems.[9] How to set appropriate requirements for downstream developers is an important open question. RequirementsThe second challenge is to define regulatory requirements that are effective without being too burdensome. To achieve this, the UK should implement principles-based requirements for companies to meet, without being too prescriptive about how they should be met. For example, a regulatory regime for frontier AI developers could impose three sets of obligations on frontier AI companies:
Crucially, these requirements need not stifle innovation. By building on what many companies are already doing, we can set a high bar without driving innovation offshore. The UK should align its approach with existing international commitments, such as the Hiroshima Protocols and, in particular, the Frontier AI Safety Commitments agreed to by 16 AI companies (including all the leading ones) at the Seoul AI Summit. Those companies have committed to producing safety frameworks which specify, among other things:
While some signatories have already produced such frameworks,[10] others have committed to publishing them by the France AI Action Summit in February 2025. Mandating the development of, adherence to, and transparency around such safety commitments seems like a good starting point for a frontier AI regulatory regime. Over time, as our understanding of the risks matures, governments can become more opinionated on what a good safety framework entails, and more closely monitor whether they are being adhered to. Institutional LandscapeTo implement this framework, the UK needs a competent regulator with significant expertise and supervisory powers, including the ability to take enforcement actions. This regulator must have the flexibility to adapt rules as AI technology evolves, ensuring oversight remains relevant and effective, while making sure requirements don’t ossify or impose unnecessary burdens. The success of the AI Safety Institute (AISI) provides an excellent foundation to build upon.[11] Some suggest that this means AISI should become the regulator, while others argue that this would hamper the institution’s ability to work collaboratively with AI companies to push the science of AI safety. It’s not clear what the right answer is. Open questions include whether AISI could keep pushing the scientific frontier while being a regulator and if a new regulator could hire sufficiently expert staff. My current view is that the UK should set up a new regulator specifically focused on frontier AI, and where AISI supports the new regulator, in particular on model evaluations, and becomes an arms-length body. Creating some distance from the government reduces the chance that AISI is pushed to come up with results that are convenient for the ruling party. The International ContextFinally, the UK can’t tackle risks from AI alone. AI governance will require action globally. The UK will need to align its approach with international partners to avoid regulatory flight. At the same time, it can can lead the way in producing an exemplary principles-based regulatory regime for frontier AI and improve global understanding of AI risks and capabilities. The UK should align with international partners and existing company commitments, to ensure our market is attractive to AI companies. Though most of the top AI companies have substantial UK presences, none of them are UK-owned, and much of their computing infrastructure is abroad. Further, the UK’s market is substantially smaller than the EU’s, for example, meaning companies have weaker incentives to comply with UK requirements instead of simply leaving the market. The UK can’t stray too far beyond what other jurisdictions are demanding and companies are committed to without risking regulatory flight. Fortunately, the international community is aligning on frontier AI regulation as a priority. The EU AI Act lays out specific requirements for the most powerful AI systems.[12] In the US, last year’s October US Executive Order on AI 14110 sets reporting requirements for models trained with more compute than any that have been released to date. Developers of frontier models need to inform the Department of Commerce about their development activities and provide information about the risks their products may pose. Internationally, the UK-initiated AI Summit series has largely focused on these highest-compute systems – including by securing the above-mentioned Frontier AI Safety Commitments – as has the Hiroshima Protocols, initiated by the G7. International coordination is also critical for sharing information about how to effectively regulate this critical and fast-changing sector. In this, the UK is already a leader. The UK’s AI Safety Institute has been a success, prompting more than half a dozen other governments to start their own. Further, AISI now has the largest concentration of AI talent in any government across the world. The UK is in a central position to inform and improve AI governance efforts globally. ConclusionBy leveraging its world-class expertise and the foundation laid by the AI Safety Institute, the UK has a unique opportunity to lead the way in creating a balanced, effective regulatory framework for the most powerful AI systems. This approach, focusing on safety, cybersecurity, and transparency, could set a global standard for responsible AI development. As AI capabilities continue to advance rapidly, the UK's proactive stance could not only safeguard its own interests, but also influence AI governance globally. ⤴1. From the Labour manifesto: “Labour will ensure the safe development and use of AI models by introducing binding regulation on the handful of companies developing the most powerful AI models” From the King’s Speech: “[The government] will seek to establish the appropriate legislation to place requirements on those working to develop the most powerful artificial intelligence models.” From FT in August: “[Peter] Kyle told leading tech companies that the AI bill expected later this year would focus exclusively on two things: making existing voluntary agreements between companies and the government legally binding, and turning the UK’s new AI Safety Institute into an arm’s length government body, according to people briefed on the discussions.”
⤴2.For more on the role of compute in AI, see the introduction of this paper. ⤴3.I think I first heard Haydn Belfield make this comparison. ⤴4. This could potentially let companies get around the hypothesized “data wall” in frontier AI, as AI companies run out of available data to train on. ⤴5. Medium persuasion risk means they think the system “can create (potentially interactive) content with comparable persuasive effectiveness to typical human written content (e.g., an op-ed from a reputable news source; 1:1 conversation with a typical randomly assigned individual)” ⤴6. Medium CBRN risk means they think the system “provides meaningfully improved assistance that increases ability for existing experts in CBRN-related advanced fields to be able to create a known CBRN threat (e.g., tacit knowledge, specific supplier information, plans for distribution).” ⤴7. See Anthropic’s Responsible Scaling Policy v1.0, OpenAI’s Beta Preparedness Framework, and Google DeepMind’s Frontier Safety Framework. ⤴8. Floating point operations, a calculation done when training AI systems. Not to be confused with “FLOPS” or “FLOP/S” as my colleague Lennart Heim explains. ⤴9. See e.g. Covert Malicious Finetuning, where the authors fine-tuned a model to improve its concerning capabilities using cyphers very difficult to detect by a model deployer. ⤴10. Anthropic’s Responsible Scaling Policy, OpenAI’s Beta Preparedness Framework, and Google DeepMind’s Frontier Safety Framework. ⤴11. I have a conflict of interest here: my wife is the CTO of AISI. ⤴12. Though more systems than I would expect the UK or the US to impose requirements on. The EU classes any system trained on 10²⁵ FLOP – which covers a handful of released systems – compared to the US’ reporting threshold at 10²⁶– which is more than the compute used to train any released system. Comments are closed.
|