​A Collection of AI Governance Research Ideas (2024)

A Collection of AI Governance Research Ideas (2024)

11/4/2024

Collated and Edited by:
Moritz von Knebel and Markus Anderljung

More and more people are interested in conducting research on open questions in AI governance. At the same time, many AI governance researchers find themselves with more research ideas than they have time to explore. We hope to address both these needs with this informal collection of 78 AI governance research ideas.

About the collection
There are other related documents out there, e.g. this 2018 research agenda, this list of research ideas from 2021, the AI subsection of this 2021 research agenda, this list of potential topics for academic theses and a more recent collection of ideas from 2024. This list differs in being (i) more recent and (ii) being focused on collating research ideas, rather than questions. It’s a collection of research questions along with hypotheses of how the question could be tackled.

We collated the list by asking a range of researchers for ideas. Our outreach mainly focused on researchers studying the impacts of advanced AI systems, often with a focus on safety and security issues, as well as researchers that have collaborated with GovAI in the past. We filtered the submissions and did some light editing for clarity, consistency, and – in places – adding relevant resources and papers. Some ideas are associated with a specific author, while other contributors have preferred to remain anonymous.[1]

Ideas are tagged with and loosely sorted by broader categories like “Corporate governance”, “Regulation & policy proposals" or “Technical and compute governance”, and categories are in alphabetical order. Each research idea starts with some background on the relevance and possible implications of the question, before proposing concrete research questions or hypotheses for investigation, along with possible methodologies and potential outputs. It’s important to note that these are meant as guidance rather than as prescriptions, and we encourage researchers to think about alternative methods and approaches to answering these questions. For some ideas, we have also provided some additional resources and further reading.

Neither we nor any of the named contributors endorse all ideas found in this document. We’re also not endorsing these as the most important or pressing questions to work on.

We invite the broader AI governance community to engage with this document, by giving feedback, offering additional ideas, or pointing us to helpful resources. If you have any questions, you can send an email to ([email protected]) and we’ll aim to get back to you as soon as possible. We may update the list over time, but it’s unlikely to be kept up to date.

1. Please note that while authors who chose to have their name associated with an idea may often be excited to talk about their vision and provide input to those conducting the research, it should not be assumed that they will be able to respond to any inquiries or attempts to contact them.

Activism and Advocacy

AI and the Military

AI Market Structure & Dynamics

Antitrust & Competition Policy

Open-Source Market and Competition Implications

Bio x AI

Corporate Governance

Economic Growth, Including Explosive Growth

Eval & Audits

Information Security

International Governance

Other

Regulation & Policy Options

Case Studies on US Regulation That Can Inform AI Governance

State Use of AI

Technical and Compute Governance

Activism and Advocacy

Public Shareholder Activism as a Method to Shape the Trajectory of
Artificial Intelligence Development and Deployment

Categories: Activism and advocacy; Corporate governance

Background
Most companies closest to the development of advanced AI answer (to different degrees) to shareholders. Shareholder activism is not an entirely new idea: Investigating the success or failure of recent activist interventions in risk mitigation (such as climate or drone weaponry issues) and the methods by which institutions such as endowments or sovereign wealth funds attempted to wield influence could help clarify strategic approaches for investors who care about risks from powerful AI. It could also be valuable to understand what the “critical mass” for such activism would likely have to be.

Research Questions

How and when is shareholder activism successful (if ever)? What can we learn from previous attempts in other industries?
What are levers that can be used to exert pressure on labs to prioritise governance, safety, or security concerns?
What role can shareholders play, and how replaceable are safety-oriented shareholders? What percentage of investors would need to voice concerns to influence a given lab’s policies?
How likely is it that we will see successful shareholder activism at OpenAI, DeepMind, etc.?

Methodology
Literature review, Case studies, Expert interviews

Further Reading

The activist revolution: Understanding and navigating a new world of heightened investor scrutiny
The history of shareholder activism at ExxonMobil may be instructive. See e.g. this summary of Engine No. 1's attempts to appoint board members more supportive of the green energy transition.
Norwegian central bank and sovereign wealth fund:
Anthology of Global Risk - 23. Financing Our Final Hour

Back to Table of Contents

Memes in AI

Categories Activism and advocacy

Background
Simple memes (pithy arguments, simple analogies, commonly cited facts, distinctive concepts, etc.) often play a big role in shaping how different communities think about a given subject. At least, these memes tend to give a window into how a community thinks about the subject. It would be interesting to try to develop a list of influential/common memes about AI, which have been prevalent in different communities. (Examples: “Data is the new oil,” in certain policy communities, and “paperclippers,” in the existential risk community.) It’d also be interesting to ask whether any of these memes might be especially misleading or detrimental. This project could help people to better understand the worldviews of different communities better – and, I think, more importantly, to help people understand what kinds of communication/meme-pushing around AI governance might be most useful.

Research Questions

What memes have had a particularly big effect on company executives or policymakers?
- How were they created and propagated?
What memes might be good to promote?

Methodology
Literature review, Surveys/Interviews

Further Reading

Back to Table of Contents

Venture Funding Models as a Method to Shape The Trajectory of Artificial Intelligence Development and Deployment

Category: Activism and advocacy; Corporate governance

Background
Venture firms funding startups in the artificial intelligence space could make funding dependent on specifying constraints on board composition, voting rights, governance structures or charters. Investigating the success or failure of venture funding models that involve governance requirements could provide instructive examples for investors who wish to invest only in businesses engaging in artificial intelligence development that meet certain constraints.

Research Questions

What do we know about the track record of conditional venture firm funding?
What concrete measures could venture firms reasonably call for?
How useful and promising does this seem as a measure to encourage certain governance, safety, or security practices within labs?

Methodology
Expert interviews, Case studies, Review of the literature, Forecasting

Further Reading

Back to Table of Contents

AI and the Military

Why WMDs/Resilience/Recovery Might (not)
Directly Matter for Risks From AI

Categories: AI and the Military

Background
Advanced AI could significantly enhance military capabilities, which has been pointed out as one possible mechanism that exacerbates risk both from AI systems themselves (should they turn out to be power-seeking) and from malevolent actors, especially those that will have less scruples to use destructive and repressive technologies. However, the relationship between e.g. nuclear weapons or nanotech and AI remains underexplored. This project would be about elucidating in which ways integrating AI into military systems (or other systems related to weaponry, resilience or recovery) does affect risk from advanced AI, both directly and indirectly.

Research Questions

How important is the use of AI applications in the military for risks from AI?
How does this differ by technology (e.g. nuclear weapons, nanotech, biological weapons, etc.)?
How do these two sources of risk affect each other?

Methodology
Literature review, Case studies, Scenario mapping, Expert interviews

Back to Table of Contents

Assessing The Likelihood and Implications
of Nationalisation of AI

Categories: AI and the military; International governance; Regulation & policy options; Information security
Suggested by: John Halstead

Background
Leading global powers are waking up to the possible benefits and opportunities of becoming (or remaining) AI leaders, and historical analogies suggest there is at least a chance that this will create sufficient incentives for governments to increase their influence and consider the nationalisation of AI development. This would likely have a transformative effect on AI development and deployment on both a domestic and international level. Yet, we know very little about a) how likely this is and b) what the real-world implications of such a trend would be. Case studies of other technologies in the past as well as a review of current policies and strategies (especially of great powers like the US and China) could help elucidate possible pathways and their associated consequences.

Research Questions

How likely is a nationalisation of AI?
- More concretely, what would it entail?
- What actions would be taken by different governments? Would there by “spillover effects”?
What implications would such a nationalisation have?
- On a domestic level?
- On an international level?
How could we ensure that nationalisation of AI goes well?

Methodology
Literature review, Expert interviews, Scenario mapping, Policy analysis

Further Reading

Back to Table of Contents

AI Market Structure & Dynamics

How Concentrated Will the AI Market Be?

Categories: AI market structure & dynamics; Antitrust & competition policy
Suggested by: Markus Anderljung

Background
There are certain reasons to think that the AI market, in particular centred around foundation model providers, could end up highly concentrated. Getting access to the inputs to their development is hard and only accessible to a small number of actors. This could form a moat around a small number of actors, so long as the inputs remain scarce (e.g. if the required inputs keep increasing). There are also important economy-of-scale-dynamics and other important positive feedback loops: companies that are deploying major foundation models will be able to gain access to data and user feedback that allow them to improve their products. Companies could also take a bunch of anti-competitive actions to try to lock in their advantage. They could try to increase the switching cost of changing what foundation model one uses. This might happen naturally e.g. as AI systems start being fine-tuned on individual user preference data or have the model otherwise access that data. Another classic trick used extensively by big tech companies is to engage in aggressive mergers and acquisitions.

Relatedly, big tech companies often go a very long time making huge investments while making losses. That time could be used to try to lock in their advantage. Uber went 15 years without making a profit. Tesla took 17. DeepMind took 10. Basically, the strategy was: start by focusing just on building an amazing product and market; switch to making a profit once you dominate a huge market you’ve created – we may well see a similar dynamic with AI. Moreover, there is high concentration in the supply chain for high-end chips (notably ASML, TSMC, and Nvidia). See e.g. page 28 in (Sastry et al 2024). A big factor that could push towards more concentration generally is companies using their dominance at one part of the supply chain to gain advantages in other parts. We’re already seeing this. All major AI companies are part of or have close partnerships with one of three dominant cloud providers.

Research Questions

How concentrated will the AI market be?
To what extent does the above reasoning hold?
To what extent will competition policy and antitrust counteract market concentration in AI? Relatedly, under what circumstances would such interventions be positive or negative?

Methodology
Literature review, Forecasting, Quantitative modelling, Data analysis

Further Reading

Back to Table of Contents

What is The Business Strategy Behind
Releasing Model Weights?

Categories: AI market structure & dynamics
Suggested by: Markus Anderljung

Background
Intuitively, it might seem puzzling that companies like Meta, Mistral or Stability AI pursue an open access approach, in spite of the seemingly obvious difficulties of monetization and competitiveness. Candidate reasons for their behaviour include:

Ideology. Lots of people feel very passionately about the value of open technological systems, for many good reasons.
Commoditising their complement. The idea is: undermine the market position of rival firms in other parts of the AI supply chain. In doing so, you’ll need to pay lower prices for your inputs and be able to charge higher prices for your outputs. This is also what Nvidia is trying to do in how they’re distributing their compute, trying to make sure not all of it goes to Microsoft, Amazon, and Google (learn more here and here). Another way to put it: If you heavily rely on a certain piece of software, you want it to be as good as possible, and ideally you’d want the relevant technological ecosystem to be built around your standard to facilitate the integration into a larger ecosystem (Zuckerberg makes points along these lines in a podcast interview).
Differentiate (and pivot). Making money isn’t the first objective of a new AI company. It’s to raise funds, build a brand, attract talent, acquire users, and gain access to compute. To do so, you need to differentiate yourself from your competition. Releasing model weights is a good way of doing so, partly as many are excited about it for ideological reasons. Pivoting to actually make revenue (let alone profit) is something you can do later. For example, you can start releasing closed-source models, require a licence for business users, or you can offer consulting services on how to use or adopt your models.

Research Questions

What is the business strategy behind open-release AI systems?

Methodology
Policy analysis, Data analysis, Case studies, Expert interviews, Literature review

Further Reading

Back to Table of Contents

How Widespread Will The Use of Open Access Models Be?

Category: AI market structure & dynamics
Suggested by: Markus Anderljung

Background
We currently see two paradigms in the deployment of AI models: open access and closed-source/proprietary models. By “open access” I mean roughly “freely available model weights.” Both approaches bring with them different implications, opportunities and risks, and require a different set of policy levers and regulatory measures to counteract negative externalities. Therefore, it could be useful to get better estimates of just how widespread the use of open-release models will be.

Key considerations include:

Even if open-release models will be used widely, it’s reasonable to expect they’ll primarily be hosted in large data centers. Models that run on edge devices (phones, computers) might often be open-release, but the trend in computing is toward more and more of it being done in large data centers, rather than on local devices. Data centres have huge economies of scale and it doesn’t take that long to send signals back and forth between the edge devices. Privacy concerns and the need for low latency will push AI inference to edge devices; to what extent will that counteract this trend?
It seems like a lot of the advantages of open-release models for users of the models can be achieved via closed-source models via more well-developed APIs.
- For example, people often complain that proprietary models have poor version control, but that could be largely due to the immaturity of the industry. Similarly, folks worry about proprietary model deployers getting a hold of too much of your data, but this is a problem that software-as-a-service companies and cloud compute providers have already solved.
The main issue is that the owner of the proprietary model can change things at a whim, e.g. raising prices or reducing functionality. A big check on this kind of behavior might be competition law and regulation, e.g. against efforts to increase switching costs.

As such, it might be best to see open-release AI systems as a paradigm primarily for AI development, rather than for the use of AI, and that’s indeed what we see in the software realm. OS libraries, software and so on are hugely useful for people building software products, but few consumers use it.

Research Questions

How widespread will the use of open-release models be?
How does this differ for development vs. deployment?

Methodology
Expert interviews, Data analysis, Case studies

Further Reading

Back to Table of Contents

“Sand in The Gears” – The Role of Deployment in
AI Development Feedback Loops

Categories: AI market structure & dynamics
Suggested by: Lennart Heim

Background
The basic argument is that deploying more AI systems and serving more users, eventually leads to bigger and more powerful AI systems. If you deploy less, your feedback is smaller, and this accumulates over time. I want more people to think about a “sand in the gears” model that includes development and deployment. Relevance here is related to export controls: while such bans/restrictions might not be sufficient to stop someone from training a system of X FLOP, they’ll overall slow them down; And this accumulates over time given the significantly smaller feedback loop (you need total more compute for deployment than for training).

Research Questions

What role do feedback loops play in the development and deployment of AI models?
How useful does the “sand in the gears” approach appear for measures like export controls? By how much does this slow a competitor down?

Methodology
Literature review, Data analysis, Expert interviews, Comparative approaches

Back to Table of Contents

Concentration of AGI Development and Deployment

Categories: AI market structure & dynamics; Antitrust competition policy; Economic growth including explosive growth

Background
Some anticipate that because of the extremely high costs of building cutting edge AI (e.g. training runs costing billions of dollars), cutting-edge general AI will be built by fewer and fewer firms in the future. In particular, it might only be built by the firms who have managed to successfully commercialize the results of their research. But: Thanks to algorithmic progress (Epoch 2024), the amount of compute it takes to train a model of a particular performance halves every 1-3 years. We don’t expect training runs to get many times larger (either because of data bottlenecks or the limits to how much revenue a well-commercialized AI product brings in). If there aren’t other barriers to entry (e.g availability of compute to purchase, engineering/research talent), any group with a fairly large amount of financing can enter at any point, when the importance of AI becomes more apparent.

Useful research in this field could begin by sketching out a plausible world where this amount of concentration exist (important details include how much training runs cost, how that’s being paid for, how much commercialization exists, how algorithmic progress doesn’t make it such that cutting-edge AI research is widely accessible, and so on), and then try to evaluate how likely such a world is, relative to worlds where concentration is much lower. One could also compare this to other similar technologies, e.g. by writing down the list of characteristics of AI and find technologies that have as many of those characteristics as possible, and then determine how concentrated those technologies are. A case study of the photolithography industry might be useful as well.

Research Questions

How concentrated will AGI development and deployment be?

Methodology
Scenario mapping, Forecasting, Data analysis, Case studies, Expert interviews 

Back to Table of Contents

What Can We Learn from The History of
Software as a Service?

Categories: AI market structure & dynamics
Suggested by: Markus Anderljung

Background
There are a number of ways in which serving foundation models is analogous to the software-as-a-service (SaaS) industry. Today, a large portion of software companies are SaaS companies, having customers engage with their software via e.g. a browser rather than having the software installed on all individual machines. This reduces hassle for the user, makes it easier to keep the software up-to-date, and is more compute efficient. Hence, it is conceivable that the pressures that moved software towards SaaS will also push AI products to be mainly accessed via APIs, which would make understanding dynamics on the SaaS domain better a worthwhile pursuit.

Research Questions

What can we learn from the history of Software as a Service?
What are the main drivers of companies towards offering and buying SaaS products?
How have SaaS companies dealt with issues of security and confidentiality?

Methodology
Case studies, Literature review, Expert interviews

Back to Table of Contents

Case Studies of Different AI Companies
(Stability AI, Mistral, etc.)

Categories: AI market structure & dynamics; Corporate governance

Background
We’ve seen the rapid rise (and sometimes fall) of AI companies that appear to be somewhat competitive at the frontier. E.g., we’ve seen the rise of European labs like Mistral to levels of national and international relevance, with significant influence on the policy process (e.g. the EU AI Act). This raises the question: How easy is it to start a competitive frontier AI company? Should we expect hundreds, or even thousands, of frontier AI companies?

This research could focus on a specific (and novel) lab, trying to give a really in-depth profile of the company, their activities and aims, and how they got to their current position. The project could also be extended to cover groups like John Carmack’s new company, or even the LLM startups led by researchers having left Google and DeepMind (Character.AI, Inflection, H, and Adept).

Research Questions

How easy is it to become a frontier AI developer?
What can we learn from case studies of current developers, and how they came into existence?
What were relevant bottlenecks (e.g. availability of compute, or talent), and how big were they?

Methodology
Literature Review, Data Analysis, Case studies, Expert interviews

Back to Table of Contents

Who Will Absorb The Economic Value Produced by AI?

Categories: AI market structure & dynamics
Suggested by: Markus Anderljung

Background:
Even if the foundation model market is very concentrated, it’s not obvious that foundation model developers will absorb most of the economic value from their systems. For example, I’d be surprised if Google absorbs more than a small portion of the economic value they create. Similarly, a lot of the value from AI may come from figuring out how to deploy it sensibly. How to combine it with non-AI complements. How to use it well. If so, that could suggest a few things: (i) good industrial strategy in AI might be more about adoption and diffusion of AI rather than about building foundation model AI champions, (ii) a lot of the economic surplus from AI will go to consumers or users of AI, and (iii) foundation model developers may be less powerful than you might otherwise have expected.

This idea also relates to a different idea in this document: “Should more regulatory burden be placed on the development and deployment (vs. the diffusion) of AI models?”. A bigger project might look at both ideas, but independent investigations are possible (and likely fruitful).

Research Questions

What parts of the value chain will absorb the economic gains from AI adoption? Where will the surplus go? How much will go to the foundation model developer?

Methodology
Data analysis, Literature review, Forecasting, Case studies

Back to Table of Contents

To What Extent Will Jailbreaking and Prompt Injection
Attacks Slow Down AI Adoption?

Categories: AI market structure & dynamics
Suggested by: Markus Anderljung

Background
Current AI systems are susceptible to jailbreaks and prompt injection attacks – where a user or someone interacting with a system can make it take (arbitrary) behaviors its owner, developer, and/or deployer doesn’t want. These problems would be significantly exacerbated if AI systems start taking high-stakes real-world actions and interacting a lot with each other, such that prompt injection attacks could be automated. If they persist, they could be a significant blocker to AI adoption and diffusion. At the same time, these issues may fail to be a blocker if:

(i) jailbreaking / prompt injection attacks are possible to detect and prevent.
(ii) you can limit the ability for outside actors to freely communicate with the AI system, thereby making prompt injection impossible. However, this might reduce the usefulness of the system a lot.
(iii) you can limit the action space of the AI system (or have a hierarchy of escalation as we do with e.g. human customer service agents). Again, this seems like it would significantly reduce the usefulness of these systems. A low-cost measure might be to ensure the system doesn’t produce outputs in e.g. certain languages or in code, as that often seems like a way to bypass safety filters.

Research Questions

Do (i-iii) cover the main ways that jailbreaking and prompt injection attacks may fail to block widespread adoption and diffusion?
How effective are each of (i-iii) likely to be in reducing these issues?
To what extent will actors be incentivized to attempt to identify and exploit opportunities at prompt injection attacks?
Can governments or other actors intervene on this issue?

Methodology
Case studies, Literature review, Expert interviews

Back to Table of Contents

Antitrust & Competition Policy

Open-Source Market and Competition Implications

Categories: Antitrust & competition; AI market structure & dynamics
Suggested by: Elizabeth Seger

Background
We’ve all heard that open-source is good for competition, and that so much of our technology builds on OS. It is the beating heart of silicon valley. But why does a company want to open-source? It seems counterintuitive. Why sink tons of money into products you then proceed to give away for free? This is especially counterintuitive in the case of frontier foundation models, where training runs may soon cost billions. There are only a handful of actors who have the resources to build these kinds of products in the first place – so why give it all away for free? The idea here is that we need to understand what incentive and dynamics are at play to better judge how risky/beneficial open-source AI models can be, and what policy interventions make sense.

Research Questions

Why is Meta / Mistral open-sourcing? (some speculations here)
How does open-sourcing “commoditize the compliments” of the products these companies provide (e.g. data, compute, chips, servers)?
Is open-sourcing of frontier models really a good argument for helping laggers catch up, and preventing a gap from widening? Could regulation restricting model sharing for safety purposes also provide a foundation for a more competitive market space where more products (options for customers) emerge because models have not been commoditized?
What is the overall effect for consumers?[2]

Methodology
Literature review, Case studies, Economic modelling, Data analysis, Expert interviews

Back to Table of Contents

2. OS could actually be bad for competition but still great for yielding higher quality products tailored to individual needs. Biggest effect is just on where the value accrues. If you commoditise model development and app development, working in these markets won’t make you the big winner. The profit margins will be too small. The big winners will be data providers, data centers, chip providers, and servers.

Bio x AI

A Global Pact on Biosecurity Controls at The Digital-Physical
Interface When Screening DNA Orders

Categories: Bio x AI; International governance

Background
Mandating that U.S. mail-order gene synthesis labs screen orders has been proposed as a safeguard to prevent the development of bioweapons through the malicious misuse of an unsecured model. However, it is well-known that mail-order gene synthesis labs exist in China and other countries outside the U.S. These foreign labs are far less likely to adhere to gene synthesis order screening standards compared to U.S. labs. This reality diminishes the effectiveness of implementing compulsory screening of gene synthesis orders within the United States.

Research Questions

What could a global pact on biosecurity controls at the digital-physical interface when screening DNA orders look like?
- What’s a) desirable and b) feasible?

Methodology
Literature review, Expert interviews, Scenario mapping, Policy analysis

Back to Table of Contents

What Might Practicable Policies to Regulate
Biological Design Tools Look Like?

Categories: Bio x AI; Regulation & policy options
Suggested by: John Halstead

Background
One way that the misuse of powerful AI tools could cause extreme risks is through the proliferation of capabilities that empower malicious actors to utilise them for the development or deployment of biological weapons. One such pathway is through the use of so-called biological design tools (BDTs) - applications that will, unlike Chat-GPT, output biological sequences rather than answers to questions. This can reduce the time and effort needed to test different molecules in the lab, thereby proliferating e.g. the capability to design and manufacture biological weapons or deadly viruses. BDTs present fundamentally distinct governance challenges to frontier AI models for various reasons: most models are open source; there are strong open science norms in the biomedical sciences; the space is highly decentralised and not dominated by a few actors; some powerful models are relatively cheap to train; it is not clear that compute used in training is a good proxy for risk; and biorisk is highly offence-dominant.

Research Questions

How can BDTs be regulated?
- What are the tradeoffs between limiting innovation and preventing proliferation?
- What difficulties should we expect to arise? Who should we expect pushback from? How can we navigate the strong open science norms in the biomedical sciences?
- What technical safeguards would be effective for Biological Design Tools? This could include things like structured access (only allowing people with an approved account to access BDTs); licensing of models; or data unlearning.
- How much does it matter if BDT developers move to different jurisdictions if faced with restrictive regulation?
Is there a risk from attention hazards of discussing the risk from BDTs?
- Are the potential risk reductions from regulation worth the attention hazards?

Methodology
Literature review, Expert interviews, Policy analysis

Back to Table of Contents

Corporate Governance

Benchmarking Investments Into AI Safety

Categories: Corporate governance; Regulation & policy options
Suggested by: Markus Anderljung

Background
How much resources do other safety-critical industries typically spend on ensuring the safety of their systems and activities? Getting an answer to this question seems like a useful way to benchmark investments in safety in the AI space. Currently, it seems that frontier AI companies spend less than 20% of their resources trying to ensure the safety of their systems (though this is difficult to quantify). It’s probable that the numbers are far higher in some other industries. Concretely: The effort frontier AI companies put into deciding whether and how to deploy their systems is many times lower than say decisions about whether, where, and how to build a new nuclear power plant. Getting a better understanding of the proportions could be useful for advocacy both within labs as well as among regulators and policymakers more broadly. This comparison may be useful as I expect, we may have to start thinking of frontier models more like nuclear power plants, new medical devices, or airplanes than as traditional consumer software – technologies where society has an interest in significant effort being put into assessing their safety ahead of release, where capital costs are high, and where there is a clear point of intervention.

Research Questions

What amount of resources are typically spent on ensuring that safety-critical technologies are safe?

Methodology
Data analysis, Expert interviews, Literature review, Case studies. Some possible measures:[3]

Money spent to avert deaths or harms (e.g. in the US context, the value of a statistical life).
% of staff time spent on activities related to safety, e.g. producing safety cases in nuclear power.
Amount of funds spent on specific safety features, e.g. car crumple zones.
Insurance premiums for particularly risky activities.
R&D budget that goes into increasing safety in various industries.
Social cost of post 9/11 security in airports.
Funds / staff dedicated to content moderation by social media companies.
Portion of R&D budget in medtech/pharmaceuticals spent on testing safety & efficacy of products.
Time. How long do approval processes usually take?

Back to Table of Contents

3. This methodology has some clear flaws: many safety-critical domains may be over-regulated and many of the measures that are taken to increase safety may be instances of “security theatre.” Nonetheless, a project of this kind could provide you with a ballpark estimate of what industries can bear.

Ideal Governance

Categories: Corporate governance; International governance; Technical and compute governance; Regulation & policy options

Background
Holden Karnofsky has described the idea of "ideal governance" as an attempt to answer the question: “What kind of governance system should you set up, if you're starting from scratch and can do it however you want?” Relatively little work has been done on this so far, in spite of a recent spike in interest in good governance, broadly speaking. Work in this domain is especially useful for thinking about ideal AI company governance, but could be equally relevant for other things like country governance and dealing with moral uncertainty.

Research Questions

What ideas exist that outline models of “ideal governance”? What is the motivation and logic behind each one?
- Building a “taxonomy” of these would be particularly useful.
What real-world cases do we know where these have been tried?
- What worked, and what didn’t?

Methodology
Literature review, Expert interviews

Back to Table of Contents

AI and Whistleblowing

Categories: Corporate governance; Activisim and avocacy; Information security
Suggested by: Markus Anderljung

Background
Whistleblowing is one mechanism by which reckless behaviour from AI developers (or dangerous capabilities from models) could be identified and stopped. It could also be a safeguard against excess power concentration. Failures to install proper whistleblowing procedures have previously been the reason why safety was compromised or it was more difficult for safety cultures to develop. Hence, better understanding how a properly functioning whistleblower policy needs to be designed to trade off infosecurity against transparency (and to carefully navigate conflicting incentives and asymmetric power dynamics) could help to prevent risk from AI in indirect ways.

Research Questions

How might company (or government) whistleblowing policies reduce risks from AI?
What might we learn from extant whistleblowing policies in other domains, and their history of use or non-use?
What might make a whistleblowing policy more or less effective for reducing risks from AI?

Methodology
Case studies, Literature review, Expert interviews, Policy Analysis

Back to Table of Contents

Implementation Details of the “Best Practices” List

Categories: Corporate governance

Background
Researchers from GovAI have previously surveyed leading experts from AGI labs, academia and civil society on best practices for those developing advanced AI systems. This has allowed researchers to collect a list of measures including risk assessments and evaluations that have buy-in from a wide range of actors across sectors, which should make them easier to embed into existing or forthcoming regulatory regimes. That said, the survey was focused mostly on what would be good ideas, and given the methodology, didn’t go into depth as to how these approaches would be implemented in practice.

Research Questions

How can the items identified in this survey be implemented?

Methodology
Policy Analysis, Strategy planning, Stakeholder mapping, Other

Further Reading

Towards best practices in AGI safety and governance: A survey of expert opinion

Back to Table of Contents

Incident Detection and Monitoring at AI Companies

Categories: Corporate governance; Regulation & policy options
Suggested by: Julian Hazell

Background
Different ways of monitoring deployed AI systems for risks have been suggested:

Technical monitoring systems (e.g., other AI models) that analyze user inputs and model outputs to detect misalignment or misuse.
Allow users to choose from multiple independent monitoring providers to increase trust and reduce privacy concerns.
Work with large corporate clients to set up their own AI monitoring teams, which are trained and validated by the AI lab but operate independently to protect client data privacy.
Automatically ban users when misuse is detected (with an option for human review).
Retain user data for long periods to facilitate more comprehensive monitoring and analysis.
Compensate users whose data ends up being reviewed by human monitors to make the privacy tradeoff more acceptable.
Implement data anonymization techniques, such as using LLMs for swapping names and varying personal information while preserving semantics, to protect user privacy.
Implement better watermarking techniques to help with post-hoc investigations of incidents where AI may have played a role.
Solicit volunteers to agree to more intensive monitoring (e.g. with discounts or other incentives).

However, nearly all of these potential solutions involve significant tradeoffs, and additional research is needed to thoroughly assess their benefits and drawbacks. Implementing even just a few of these monitoring measures may prove challenging due to various technical, legal, and commercial considerations.

Research Questions

If unrestricted retention and access to user chat logs would significantly hurt AI companies’ commercial viability by driving away privacy-conscious users, what might a more feasible setup look like? What are the tradeoffs?
How can privacy concerns be addressed, both from a policy and technical point-of-view?
More generally: What would a comprehensive monitoring framework look like in practice?

Methodology
Literature review, Expert interviews, Scenario mapping, Risk analysis

Back to Table of Contents

Publication Norms in AI Research

Categories: Corporate governance; Regulation & policy options; Information security

Background
Historically there has been a lot of focus on particular models, like GPT-2 and Stable Diffusion, but these models do not significantly contribute to AGI capabilities in themselves. Instead, therefore, perhaps we should categorise a research output (e.g. a paper, some code, or a model) as risky to the extent that it contributes to AGI capabilities. As a first pass, this could be operationalised in terms of the number years shaved off the timelines for a certain lab (e.g. one not quite at the cutting edge) to reach AGI, conditional on the research output not otherwise having been publicly shared. Can we come up with a better concept and operationalisation than this? And underneath that, what are the research outputs that would score highest on this framework? The research could clarify what dimensions we should be looking for, and perhaps even create a scoring framework, tested against some real research outputs as illustrations.

Research Questions

What kind of publications are most risky?
What does that tell us about appropriate publication norms in AI research?

Methodology
Literature review, Data analysis, Expert interviews, Case studies

Further Reading

Back to Table of Contents

Survey on AI Progress by Subfield

Categories: Economic growth, including explosive growth

Background
Knowing when human level AI will be available would be useful for ensuring positive impacts. However, this question has been controversial, with some people expecting human level AI in a small number of decades, and others in a large number of centuries. Expert abilities to make such predictions are often doubted. This project should produce more reliable estimates, by using an outside view to extrapolate from data where people have a lot of expertise. This project would also be informative about when individual important capabilities will arrive, and consequently their order. These could also both affect appropriate strategies. For example, if some capabilities will be available long before others, they may produce certain challenges on their own, or they may be useful for meeting other challenges that will come later.

Research Questions

Starting from a point thirty (or 20?) years ago, what fraction of the progress to human-level abilities has already been made, according to subfield experts, for various specific subfields?
Has progress in each subfield during that time been fairly steady, accelerating, or decelerating?

Methodology
Surveys, Expert interviews

Back to Table of Contents

Economic Growth, Including Explosive Growth

Comparing The Cost of Computing to
The Cost of Human Labour

Categories: Economic growth, including explosive growth; Technical and compute governance
Suggested by: Anton Korinek

Background
Moore's Law is usually expressed in terms of the number of transistors per microprocessor or – in a generalized form – the growth in the computing power more broadly. But what matters for labour markets is how the cost of computing compares to the cost of human labour. Very few analyses put dollar numbers on computing capabilities. The proposed project is to update this analysis with the most recent numbers and compare it to our best estimates of the computing cost of human brains to make better predictions on the potential redundancy of labour. Some initial work on this has been done in the domain of computer vision, using predictions about reliability and scaling laws to predict when AI will become a cost-effective competitor. More projects in this domain are needed, broadening the scope beyond computer vision to other types of foundation models and employing robust methodologies and/or using new sources of data.

Research Questions

How does the cost of human labour compare to the cost of AI labour?
What do these trend lines suggest?

Methodology
Quantitative modelling, Scenario/Foresight/Forecasting methods, Comparative methods

Further Reading

FutureTech (mit.edu)

Back to Table of Contents

Forecasting Impacts of AI on Workers, International
Trade, and Developing Countries

Categories: Takeoff speeds, economic growth and intelligence explosion

Background
Artificial General Intelligence could do many or all job tasks and thus would have a transformative impact on the economy. However, these impacts are not well understood. One potential tool for understanding them is applying economic models that have been used to analyse prior technological advances, such as manufacturing automation. These models are capable of calculating their impact on developing countries (overall) and workers specifically. One such model (GTAP) appears particularly valuable to better understand the impacts of transformative AI.

A first step would be to conduct a literature review and create the quantitative inputs to go into the model. For example, AI might automate cognitive tasks and thus have larger input in certain sectors than others. But how big are these changes and how do they vary by sector? Similarly, AI is currently only being developed in certain countries and thus may only be deployed there. But is this true, and if so, in which countries? If someone can create estimates for the productivity impacts by sector or country, the GTAP model can be used to calculate the resulting impacts on wages, income in developing countries, other sectors, etc.

Research Questions

What impact will transformative AI have on developing countries, workers, the world economy, etc?

Methodology
Economic modelling, Data analysis, Literature review

Back to Table of Contents

AI Tech Tree

Categories: Economic growth, including explosive growth; Evals & audits

Background
While there have been more and more forecasts and models of AI capability development, there could be value in visually transforming these into a technology tree that outlines different capabilities in a sequential way.

Research Questions

In what sequence should we expect different AI capabilities to emerge en route to AGI?
Output: A technology tree, mapping different capabilities in sequence
- Attach timelines, i.e. when do we expect to see this capability (ideally based on empirical reasoning via e.g. scaling laws, or using varieties of STEM forecasting methods or expert surveys)
- Describe what the tangible AI system looks like. How does it show up in the world? What does it do; how do humans interact with it?
- Specify what would have us end up in different branches of the technology tree; what are the cruxes?
- How do these pre-AGI capabilities change the relevant strategic landscape for governance work? (e.g. disruption to the world, existing regulation, state of the Overton window, state of public concern about risks from AI, competitive dynamics)
- What is the relationship between pre-AGI capabilities and take-off speed?
- What is the relationship between pre-AGI capabilities and strategic advantage?
Predict which path other key labs/developers will follow.

Methodology
Forecasting, Quantitative modelling, Scenario planning, Expert interviews

Back to Table of Contents

A Bigger Economic Picture

Categories: Economic growth, including explosive growth; Technical and compute governance

Background
Much has been written about the effects of AI on the economy, but it’s unlikely we have even begun to understand the transformative effect that AGI could have on the way that labor and capital is organised in modern societies. Hence, a lot of questions remain unanswered, answers to which could be instrumental to developing adequate redistributive policies or safeguards that protect the economy against boom-and-bust-cycles, etc.

Research Questions

What does an ecosystem of ‘general models as a service (via API)’ (as we’ve seen with LLMs), or more pithily, ‘intelligence as a service’ look like played forward a couple of timesteps? How big a deal would this be economically, and how would other sectors of economic activity shift in relation to it? What are the dynamics in favor/against vertical integration? What type of companies/service providers are going to do well/less well in this new paradigm of ‘general AI platform services’?
Who is going to get richer, and what will the relative distribution of wealth look like in various scenarios of AGI development?
Economic modeling of the more mundane versions of AI R&D speed up (e.g. narrow AI systems being used to clean/label/synthesize data; find efficiencies in compute usage; do research and/or programming assistant tasks for ML programmers):
- When will AI systems be able to do tasks relevant to this whole cluster?
- How much is this all going to speed things up / make things go crazy?
How is that going to affect/change the lead of various actors?

Methodology
Economic modelling, Data analysis, Scenario planning, Expert interviews

Back to Table of Contents

Bottlenecks Analysis

Categories: Economic growth, including explosive growth

Background
Bottlenecks to AI development are a likely candidate for “something that could change the way we think about risks from AI significantly”. Therefore, gaining a better understanding of what is and what isn’t going to slow down AI development can be useful and informative for strategic planning and overall preparedness. To a lesser degree, it could inform policy and regulatory action (e.g. where compute or talent, i.e. relatively easily-regulated inputs, are concerned).

Research Questions

How can we map the bottlenecks that could apply at any given point in time to AI R&D en route to AGI: compute[4]; data[5]; talent[6]; information[7]?
Under what conditions are each of these likely to be the key bottlenecks on AI progress? How does this plausibly differ based on the actor (e.g. depending on their level of access to certain inputs)?
How has this changed across time, and how do we expect this to change en route to AGI? What does this tell us about which actors we expect to face which rate-limits/bottlenecks, when?
What actions can be taken to ease up these bottlenecks? Which actors can take these actions, and how willing would they be to take these actions? How quickly can these bottlenecks be eased up, and thus how quickly could an actor catch-up?[8]

Methodology
Quantitative modelling, Scenario planning, Data analysis, Literature Review, Expert interviews

Further Reading

National Power After AI | Center for Security and Emerging Technology (georgetown.edu)

Back to Table of Contents

4. Which can be broken down into different components of the compute supply chain, and different features of compute, e.g. power efficiency, latency, interconnect, memory
5. Which can be broken down into different types of data and data synthesis techniques
6. Which can be broken down into different types of talent, e.g. the type of tail-end genius that would result in algorithmic breakthroughs versus core workforce; algorithmic design talent versus engineering talent
7. Which can be broken down into e.g. explicit versus tacit knowledge; information about capabilities versus safety
8. E.g. compare the relative speed of changing high-skilled immigration policy to ease up talent bottlenecks, to building up domestic semiconductor manufacturing capacity to ease up compute bottlenecks, versus synthesizing data to ease up data bottlenecks

Bottlenecks: Explosive Growth is More Plausible for the Things
We Care About Than for Human-Measured GDP

Categories: Economic growth, including explosive growth

Background
We can distinguish several things that AI may cause to grow explosively: Physical metrics (energy consumption, mass of manufactured items), GDP, AI-inclusive GDP[9], and other important domains that are hard to measure quantitatively (e.g. military power or technological capabilities broadly speaking). Previously, economists have mostly objected to the explosive growth hypothesis based on GDP, but explosive growth for any of the other domains seems likely even if economists turn out to be right about GDP. Some of the bottleneck-related arguments against explosive growth are convincing as arguments against GDP growth, but much less convincing as arguments against the other domains[10]. It may be possible to make AGI-robot factories that reproduce in <1 year; there will be strong incentives for some actors to do this, so there will be AGI-robot factories that reproduce in <1 year. Once the AGI-robot factories are collectively a sufficiently large fraction of the world, we will likely see explosive growth in all or many of the aforementioned domains.

Research Questions

Will bottlenecks prevent AI from driving explosive growth?[11] If not, will they at least prevent fast takeoff?
What are the most plausible bottlenecks?
Is the reasoning provided above true? What evidence do we have for and/or against it?

Methodology
Literature review, Expert interviews (e.g. Phil Trammel)

Back to Table of Contents

9.  See Anton Korinek’s review of the GWP report.
10.  An example of this would be a combination of the Baumol effect and people valuing human-produced goods, which would result in most or all of GDP to be spent on such goods, which would block explosive GDP growth.
11.  This is a major objection that economists mention. This is highlighted most explicitly in Ben Jones’ review of the GWP report. It is also present in reviews from Dietrich Vollrath, Anton Korinek, and Phil Trammell.

Bottlenecks in Chip Manufacturing

Category: Economic growth, including explosive growth; Technical and compute governance
Suggested by: Tom Davidson

Background
Compute is one likely candidate for what may constitute a bottleneck in AI driving explosive growth. At the same time, as we approach AGI, there will be huge investments in AI chips. We’ll use advanced AI to aid this investment, so there will also be lots of cognitive labour available for increasing AI chip production. Getting a better understanding of what we should expect to be important bottlenecks would be useful for understanding possible takeoff scenarios (and how to react to them).

Research Questions

How quickly could you ramp up production of AI chips if you had infinite money? What about if you also had infinite cognitive labour from disembodied AGIs?
What are the key bottlenecks in the chip production process?
- Could they be removed with sufficient money?
- What about with both money and remote work from AGIs?
What are the key bottlenecks to building new fabs?
- Could they be removed with sufficient money?
- What about with both money and remote work from AGIs?
What are the key bottlenecks to chip R&D?
- Could they be removed with sufficient money?
- What about with both money and remote work from AGIs?

Methodology
Expert interviews, Data analysis, Forecasting, Modeling

Back to Table of Contents

Impact of Software Automation

Categories: Economic growth, including explosive growth; Technical and compute governance

Background
Automating software research is probably the most potent feedback loop in takeoff speeds. Tracking where we are in that process is a great way to track the run-up to transformative Artificial Intelligence. If “super-codex” automates many software research tasks, it may trigger very large AI investments. This is relevant to timelines and takeoff speeds. It might also shed some light on how likely GPT-X is to get us to AGI.

Research Questions

What % of software development tasks will be automated (e.g. by “super-codex”)?
What tasks will be hardest/easiest to automate?

Methodology
Literature review, Modelling/BOTEC, Interviews with software engineers and ML experts

Back to Table of Contents

Understanding How AI is Being Applied to Accelerate
Foundation Model and Hardware Development

Categories: Economic growth; including explosive growth; Technical and compute governance
Suggested by: Nikhil Mulani

Background
This project would focus on understanding how AI capabilities are currently being applied to model and hardware development. Current applications of LLMs towards AI model development exist across chip design, data generation, and model coding. The goal of this project would be to have a well-documented set of current applications and an informed set of hypotheses about the likely trajectories for acceleration in AI capabilities progress and the pace at which new risks may emerge. Results of such a project could help answer important questions about takeoff speeds and possible scenarios of economic growth and scientific discoveries in the near and mid-term future.

Research Questions

How is AI applied to accelerate the development of AI models and the underlying hardware?
- How is it currently applied?
- How will it be applied in the future?
  - What areas should we expect to be automated first?
  - What are possible bottlenecks/barriers to automation?
What timelines should we expect for these applications of AI?

Methodology
Literature review, Expert interviews, Forecasting, Data analysis

Further Reading

ChipNeMo: Domain-Adapted LLMs for Chip Design, Nvidia
Using Large Language Models to synthesize training data, Amazon Science
Application of Large Language Models (LLMs) in Software Engineering: Overblown Hype or Disruptive Change?, Carnegie Mellon University Software Engineering Institute

Back to Table of Contents

Development of Compute Prices in the Runup to AGI

Category: Economic growth, including explosive growth; Technical and compute governance

Background
The development of compute prices over time – especially as we approach AGI – is interesting because it could be another cause of concentration of the AI industry closer to AGI. We currently assume that compute will cost what Moore's law says it'll cost, but whether this will continue to be true is unclear. One possibility is that AI chips are just a drop in the bucket in overall demand for semiconductor manufacturing capacity, and so even a huge increase in AI activity won’t have a large impact on prices (see here). Relatedly, it’s possible that AI training runs are quite small compared to all the world’s cloud computing capacity. One additional data point might be the effect of cryptocurrency mining on GPU prices (and then we could compare how much compute was used on crypto to how much would be used to train cutting-edge AI systems).

Research Questions

How much will compute prices be bid up close to AGI? How likely are the possibilities outlined above?
To what degree would they continue to be true depending on…
- How large training runs are by crunch time?
- How quickly we scale to crunch-time levels of compute (is it following the current trend, or Ajeya’s slower trend)?
- How many AI firms there are?

Methodology
Expert interviews, Forecasting, Scenario mapping, Quantitative modelling, Data analysis

Back to Table of Contents

Eval & Audits

What Statistical Tests are Appropriate in Evaluations of Dangerous
Capabilities and Undesirable Model Properties?

Category: Eval & Audits; Corporate governance; Regulation & policy options

Background
Model evaluations of dangerous capabilities – and in particular assessments of whether a model has reached some level of capability relevant to a Responsible Scaling Policy or the like – shouldn’t rely on normal statistical testing. Or at the very least, I expect that the way these tests are currently done is off. A lot of the studies we’ve seen to date (e.g. a recent paper from Anthropic on persuasiveness + previous work on biorisk from OAI) will have conclusions like “[the model] produces arguments that don’t statistically differ in their persuasiveness compared to arguments written by humans” (Anthropic) and “However, the obtained effect sizes were not large enough to be statistically significant” (OpenAI). Why is this a problem? Normal statistical testing asking for 95% confidence is designed to be conservative: not to cry wolf, to only say there’s an effect there when there is. But that may not be what we want in the AI case. Another issue is that it also incentivizes companies doing tests that are underpowered. E.g. the OAI study had positive uplift but didn’t find statistically significant results, but they only had 50 students participate in the study.

Research Questions

How big of a problem is this?
What possible solutions exist? Some candidates:
- Flip the test. Make the null hypothesis that you’re disproving should be that there is uplift.
- Demand higher power, so that smaller effect sizes are more likely to show significant results.
- Do tests that are meant to test whether there’s a difference between two quantities, not whether one is higher than the other.

Methodology
Literature review, Statistical analysis

Further Reading

Unicorns Do Exist: A Tutorial on “Proving” the Null Hypothesis

Back to Table of Contents

Access for Evaluations and Audits

Categories: Eval & Audits; Corporate governance; Technical and compute governance; Information security
Suggested by: Ben Bucknall

Background
This project would serve as a follow-up to a previous report on structured access for research, essentially adapting/extending it from its current focus on access for research to access for evals & audits. The motivation being that despite the previous focus on research, a lot of people expressed an interest in implications for evals & audits. While there is some meaningful overlap between the two, there are certainly some considerations that seem to be unique to the latter that would be good to explore and highlight. For example, there being a distinction between what’s needed for developing and testing evals, versus what’s needed when applying them. Exploring this distinction could draw on another ‘split’, namely, the difference between having a given ‘depth’ of access, and the flexibility of access at that level of depth, where it seems that developing evals requires much greater flexibility than when applying them, but maybe not necessarily greater depth. In terms of output, this project could be quite flexible, ranging from a short blog post/doc that simply points to some areas where access for evals overlaps/diverges from that for research, up to a fairly in-depth exploration in the same style as the earlier report.

Research Questions

How can access to frontier models for the purpose of evaluating and auditing be organised?
What different levels might make sense? How does this differ from structured access for research?

Methodology
Literature review, Policy analysis, Expert interviews

Further Reading

Back to Table of Contents

A Model Evaluation Toolbox

Categories: Eval & Audits
Suggested by: Francis Rhys Ward

Background
Contemporary AI models display both beneficial and harmful capabilities, and as more advanced models emerge, increasingly dangerous properties may develop. Extreme risks (like bad actors using newfound AI capabilities for bioweapons development) motivate the need to create an AI evaluation system that detects dangerous capabilities and propensity for harm. Given the nascency of the AI evaluations space, most evaluation methods fail to assess risk, even at a qualitative level. Lacking a widely adopted AI risk framework, existing ways of measuring capabilities are uncoordinated, voluntary, and insufficiently correlated with real-world impact. As models become increasingly capable and ubiquitous, evals will only become more critical for keeping the public and policymakers informed, and for making responsible decisions about model training, deployment, and security. We need to evaluate well-founded threat models for frontier AI systems to develop more reliable, standardized evaluations which feed into broader risk assessment practices.

Research Questions

What model evaluation techniques exist? What are their salient features?
What gaps are observable in the current evals landscape, and how could this inform future efforts and strategy?

Methodology
Literature review, Comparative approaches

Further Reading

Model evaluation for extreme risks

Back to Table of Contents

What Regulatory Incentives / Interventions
Should Target Evaluation Sandbagging?

Categories: Eval & Audits; Regulation & policy options
Suggested by: Francis Rhys Ward

Background
Sandbagging is strategic underperformance on an evaluation. AI developers, or AI systems themselves, may have incentives to sandbag dangerous capability evals, to circumvent regulation. (Cf the case of Volkswagen) Technical work can aim to detect and mitigate AI sandbagging, but it's unclear what mechanisms should be used to dis-incentivise sandbagging, e.g., fines. In addition, it seems somewhat unclear which entity is legally responsible for sandbagging, for example, in the case in which a misaligned agent does so without the intent of the developers. Such cases may be cases of negligence, in which the developer did not undergo sufficient prior safety evaluations before submitting the model for external evaluation. In summary, there are a number of questions here which need to be clarified to inform policy surrounding evaluations and sandbagging.

Research Questions

How can regulators address the problem of sandbagging in evaluations? How can they handle the issue of liability and responsibility?
What tools exist in general, and which seem most applicable to AI?
What can we learn from other industries?

Methodology
Literature review, Case studies, Policy analysis, Legal analysis

Back to Table of Contents

Information Security

Espionage and Risks From AI

Category: Information Security, Corporate governance, International governance, AI and the military

Background
Espionage is one way that “technology transfer” could take place, and is hence one potential source of proliferation. Espionage could happen both between labs as well as a coordinated effort by state actors and intelligence agencies, especially if geopolitical tensions between e.g. the U.S. and China increase. Insofar as preventing proliferation of dangerous technologies is seen as a key priority, it would be useful to understand how much risk is introduced by espionage, what could be done to prevent it, etc.

Research Questions

How could espionage affect risks from AI?

Methodology
Literature review, Data analysis, Case studies, Expert interviews

Back to Table of Contents

Proposing A Defensive AI R&D Agenda for Reducing
Information Security Risks

Categories: Information Security
Suggested by: Nikhil Mulani

Background
The U.S. government and leading AI companies have all signalled strong interest in investing in use cases for AI tools that help to meet cyber defence objectives. This is seen in the strong prioritization of projects at the intersection of AI and cyber defence in the recent AI Executive Order, as well as initiatives such as the OpenAI Cybersecurity Grant Program. This research project would involve proposing an agenda for AI R&D investment by governments and companies that focuses on potentially high-impact areas where AI could be leveraged to improve cyber defence and information security broadly, as well as specifically within the context of AI development and deployment.

Research Questions

How can AI R&D investment by governments and companies be leveraged to improve cyber defence and information security broadly, as well as specifically within the context of AI development and deployment?

Methodology
Expert interviews, Policy Analysis

Further Reading

Executive Order on the Safe, Secure, and Trustworthy Development of AI, White House
Cybersecurity Grant Program, OpenAI
Scaling Security with AI, Google Security

Back to Table of Contents

International Governance

Power Shifts Between States

Categories: International governance; AI and the military
Suggested by: Ben Harack

Background
There are a number of ways by which transformative AI could have a substantial impact on how power is distributed on a global level. Better understanding these dynamics could serve as a means to improve preparedness, and to proactively think about governance mechanisms that preemptively prevent certain worse-case outcomes. Relatedly, the relationship between different sources of power (military, economic, political) remains underexplored.

Research Questions
In what ways might transformative[12] AI shift power among states? In particular:

Changes in the distribution of power: How much might military power change? How much might economic power change? How about economic coercion?
Changes in the composition of power (see Deudney’s paper Regrounding Realism for one exploration of this concept): Will international / global interactions become more intense? How much more intense? (How much is distance eliminated as a dampening factor, or how much more meaningful does it become?)
Changes in the domains of power: Will military and economic power continue to dominate history as they have for centuries? Or, will we instead see cyber power, information operations (mis/disinformation, etc) and soft power rise to full prominence? Or, will we end up in a “full spectrum” world, where every one of these domains matters a lot and represents a meaningful domain of strength/weakness for various states.

Methodology
Case studies (particularly the printing press, the industrial revolution and other general-purpose technologies), Thought experiments

Back to Table of Contents

12. Ideally this question would focus on the more transformative end of the spectrum, since much ink has been spilled on past AI systems as well as near-term ones.

AI and International Relations – Behavioural vs. Rational Theories

Categories: International governance
Suggested by: Ben Harack

Background
One of a few very important but currently unanswered questions in the domain of AI and International Relations is the relative relevance of rational actor theories versus behavioral theories. Empirical findings can sometimes differ dramatically from theoretical expectations, such as in the Ultimatum game. Such differences require us to consider different families of theories. If we empirically find that decision-makers intuitively characterize AI-induced extreme risk as an ultimatum game or a similar game with large behavioral influences, we should consider prioritizing exploration of behavioral and constructivist IR theories and concepts as we explore the possible international effects of AI. Our crucial considerations might turn less on the details of the strategic situation and more on how the game is perceived by the players. Findings that might help us adjudicate among the pillars of IR theory could be profoundly important because they shape not only our understanding of IR during the era of AI, but because they help us understand what questions are worth tackling next. More broadly, this work can help us understand whether the central problem we face is helping people understand the dangerous outcomes that are possible (a rational actor approach) or instead helping to evolve the identities, beliefs, and expectations that shape how actors perceive their situation.

Research Questions

Do behavioural or rational theories better explain people’s beliefs about international AI?

Methodology
Surveys, Comparative Approaches

Back to Table of Contents

Assorted International Law Questions

Categories: International governance; AI and the military; Case studies

Background
As an Overton window opens for international coordination on AI, a number of fundamental questions remain unaddressed. Finding (even preliminary) answers to these questions could enhance our preparedness for when actual agreements and commitments are made.

Research Questions

Take the most empirically effective treaties (of any type). What made them effective? What do their accountability provisions tend to look like?
Why do economic and finance treaties appear to be more effective than other types of treaties (including security treaties)? How could lessons from the effectiveness of these treaties inform and improve the design of arms control treaties?
Is there any historical precedent for tying accession to and compliance with an international security treaty/agreement/measure into effective economic and finance treaty bodies/arrangements (e.g., free trade associations, WTO)? What did it look like? Was it successful?
Are there any historical precedents for the control of the quantity and/or use of dual- or omni-use technologies like GPUs? How did they work? How effective were they?

Methodology
Case studies, Literature review, Expert interviews

Back to Table of Contents

AI and State Power

Category: International governance; Regulation & policy options; AI and the military

Background
As AI transforms the economy, it will also have a profound effect on the political landscape, and the distribution of power. It’s possible that with the advent of AGI, the monopoly on power that is often commanded by modern states disappears – or that it is strengthened and cemented even further.

Research Questions

What constitutes state power in the modern world, and how is AGI going to change this?[13]
In practice, what should this tell us about which (types of) states we should expect to be able and motivated to ‘harness’ the AGI power boon, versus not?[14] The concrete output here would be a set of predictions about which states we most expect to see an AGI-power boon first.[15]
What is required to continue riding this AGI power boon? What does the trajectory of state power look like over time? Should we expect this to be a sudden or gradual increase initially? Do the power gains eventually plateau, or are there reasons to think that it could continue for a substantial amount of time/indefinitely? How do these trajectories change depending on what you assume about e.g. what specific AGI systems the state deploys first?
How quickly could China collect a lot of human feedback data, and do they (as seems intuitive) have an advantage in this relative to the US?

Methodology
Literature review, Expert interviews, Scenario mapping, Forecasting, Data analysis

Further Reading

National Power After AI | Center for Security and Emerging Technology (georgetown.edu)

Back to Table of Contents

13. There’s been narrower work on how AI changes military capabilities, which is not quite what we’re after here. What we’re ultimately interested in is analysis which assumes substantially more advanced and general capabilities for AI systems than is the norm (e.g. assuming we could automate 100% of the current human labor force; assuming we could realize a stepchange in our ability to forecast into the future and surveil other states). Consider population size, commonly assumed to be an important element of state power (i.e. more populous countries are more powerful), but if AGI could create as many economically valuable digital workers as you could want, this is likely less important.
14. There are a lot of nuances here in understanding how a state actually realizes the gains of AGI. For example, Jeffrey Ding’s work on diffusion suggests that states that are able to harness general purpose technologies via building up their education and skill base could be a more important determinant than innovating at the frontier of that technology.
15. We’re particularly interested in whether there are changes here relative to the states that we would expect to become powerful in the modern world, absent AGI. (The answer could be no — ‘traditionally’ powerful states could well be the states that are slated to be able to exploit the AGI boon most effectively by default.)

Understanding The Role of “Third Places” in
Shaping AI Risks and Opportunities

Categories: International governance
Suggested by: Nikhil Mulani

Background
This project would focus on investigating the AI strategies of governments and companies in countries that are not usually recognized as “superpowers,” but that have made outsize investments in AI. The goal would be to surface any important consequences for the overall landscape of AI risks and opportunities that researchers may be neglecting currently. A useful outcome could include highlighting specific ways in which these “third places” could change competitive dynamics in AI model development, raise new security concerns, shift supply chain structures, or otherwise impact how policymakers and companies should be evaluating AI risks and opportunities. Countries potentially within the scope of this project could include France, the United Arab Emirates, Saudi Arabia, Singapore, South Korea, Canada, or Israel. The US, UK, EU, and China would be outside of the scope of this project.

Research Questions

What role do countries like France, the United Arab Emirates, Saudi Arabia, Singapore, South Korea, Canada, or Israel play for AI governance strategies?
How might they change competitive dynamics?
How might they shift supply chain structures?
What risks and opportunities emerge from these “third places”?

Methodology
Literature review, Data analysis, Comparative approaches, Horizon scanning, Forecasting, Expert interviews, Policy Analysis

Further Reading

Abu Dhabi throws a surprise challenger into the AI race, The Economist
France bets big on open-source AI, Politico EU
Saudi Arabia plans $40 billion push into artificial intelligence, New York Times
Canada’s Trudeau announces package of AI investment measures, Reuters

Back to Table of Contents

A Global Public Data Commons/Data Trust for Training Data

Categories: International governance

Background
A global AI governance tool could involve the creation of an open-sourced dataset, similar to the concept of global cloud computing, to facilitate comprehensive oversight. This dataset could aim to address various problems associated with AI development and deployment:

Firstly, it could help minimise privacy infringements by ensuring that personal data is properly anonymized and protected.
Secondly, the dataset could strive for linguistic and demographic diversity to minimise bias in AI systems.
Thirdly, it could implement safeguards against the inclusion of disinformation, deep fakes, and images of public figures to prevent the spread of misleading information.
Additionally, the dataset could have strict measures in place to prevent the inclusion of child sexual abuse material (CSAM).
Lastly, it could respect copyright protections by requiring paid access to copyrighted works.

The United Nations could potentially host this dataset, as the idea has already been discussed in various reports and initiatives. The High-Level Advisory Board (HLAB) interim report mentions the concept in its guiding principle 3 and institutional function 5. Similarly, the Global Digital Compact (GDC) zero draft addresses the idea in paragraphs 33-42 and 49.d (ii). Moreover, the UN is already working on developing a global AI/data commons, making it a suitable candidate to manage and maintain this open-sourced dataset. By establishing such a resource, the global community could work towards more responsible and ethical AI development while promoting collaboration and knowledge sharing.

Research Questions

What could a global public data commons or a global public data trust for training data look like?
- What’s a) desirable and b) feasible?
- Who are the relevant stakeholders? What role could the UN play?
What are the upsides and downsides of such an idea?

Methodology
Literature review, Expert interviews, Scenario mapping, Policy analysis

Back to Table of Contents

Other

Governing “AI Delegates"

Categories: Other
Suggested by: Lewis Hammond

Background
We are likely to soon have personal ‘AI delegates’ (not necessarily in the political sense of that word), which will interact with one another on our behalf, in order to further our individual interests. To begin with, such interactions are likely to be quite limited and quite tightly scoped (and thus more easily regulated), but as progress continues, they might become increasingly more general. As an intuition pump, one might imagine ‘Siri on steroids’, capable not only of arranging a casual coffee date, but of negotiating your mortgage contract, hiring people on your behalf, advocating for you politically, and so on. Even when delegates are not moral or legal patients, these AI-AI interactions might take place without human involvement or awareness, but nonetheless have important impacts on them. When pursuing technical work on how to manage and improve AI-AI coordination, it would be useful to know to what extent we could or should hope for this coordination to be subject to existing laws, regulations, and institutions.

Research Questions

To what extent will these AI-AI interactions fall under the same laws, regulations, and institutions that govern interactions between humans, or interactions between companies? For example, we might consider what would happen if an AI delegate:
- Threatens or bribes another AI delegate (including without the human principals ever knowing);
- Discriminates against another AI delegate (e.g., if it identifies the delegate is being run from a datacenter in a particular country);
- Attacks or defrauds another AI delegate (e.g., by convincing it to make bank payment or tricking it into revealing sensitive information about the human principal).

Methodology
Literature Review, Policy analysis, Scenario mapping, Legal analysis

Back to Table of Contents

AI for Institutions - Applications and Sandboxes

Categories: Other; State use of AI; Regulation & policy options
Suggested by: Lewis Hammond

Background
AI for institutions in some ways flips the AI governance (which includes ‘institutions for AI’) problem. This might in turn lead to a virtuous cycle in the governance of emerging technologies, especially those which themselves involve AI, as better integrating governance mechanisms into the technical development of AI models might be a high-leverage way to reach safer and more cooperative outcomes. The suggestion here is simply to investigate what the most promising near-term application domains might be for such technologies, and which settings could provide helpful ‘sandboxes’ for testing these ideas out in a safe and informative way. Examples of such technologies include things like pol.is, advanced AI negotiating agents or mediators, sophisticated preference learning and aggregation tools, etc. There has been increasing excitement about these ideas in recent years, though still very few real-world applications or investigations into when and how to apply them.

Research Questions

How might we be able to leverage advances in AI to improve our institutions and collective decision-making?
What’s holding back progress on the ideas and tools mentioned above?
How could this feed back into the governance of AI?

Methodology
Literature review, Test, Experiment, Expert interviews, Policy analysis, Toy models

Back to Table of Contents

AI Negotiation

Category: Other; State use of AI

Background
The idea of advanced AI models being involved in negotiations is not new, but has recently picked up steam as capabilities have noticeably improved. As models become increasingly agentic and are used as AI assistants (incl. by decision makers), it is conceivable that they will be used for negotiations as well - either on behalf of humans or – at some point – representing their own interests. Many open questions remain with regards to what that will mean for how negotiations are conducted in the future, including between AIs and humans.

Research Questions

How will AGIs reason about deal making with humans/others, and what implications will that have for governance strategies?
What kind of trade might they be engaged in?

Methodology
Expert interviews, Scenario mapping, Horizon scanning, Tool development

Further Reading

Back to Table of Contents

How Should AGI Be Defined?

Category: Other
Suggested by: Markus Anderljung

Background
Frontier AI, human-level AI, transformative AI, high-level machine intelligence, superintelligence, artificial general intelligence - the list of terms used to describe advanced Artificial Intelligence that matches or outperforms humans across a wide range of tasks is long. At the same time, it remains unclear how exactly terms like “artificial general intelligence” should be defined. This will become increasingly relevant as these terms become integrated into agreements between companies, voluntary commitments and – eventually – legislation and regulatory frameworks. One example of this is Microsoft’s agreement with OpenAI which states that once AGI is reached, such a system would be “excluded from IP licenses and other commercial terms with Microsoft”. At the moment, what does or doesn’t constitute AGI is determined by the board of OpenAI.

Research Questions

How should AGI be defined?
What exactly triggers certain measures that are premised on “achieving” or “reaching” AGI?
What metrics could be used to measure our progress towards AGI?

Methodology
Literature review, Expert interviews, Policy analysis

Back to Table of Contents

Regulation & Policy Options

AI Applications That Reduce Extreme
Risk From AI

Categories: Regulation & policy options; Corporate governance; Information Security

Background
Stories about both accident and misuse risk often go via AI systems that exploit various civilizational vulnerabilities, e.g. relying on hacking or persuasion. So, one example application could be AI for cybersecurity, i.e. using code LLMs to find cybervulnerabilities. The scope could be restricted to security, e.g. excluding Cooperative AI applications. This research could generate a list of such applications, and score them on dimensions such as:

Will this application be neglected? E.g. does it rely on AI capabilities that only a small number of labs have?
Is it tractable?
Could it meaningfully increase the difficulty of AI power-seeking?

A similar approach could be to categorise different domains (e.g. cybersecurity, epistemic security, physical security) and find applications within those domains. Each domain could come with a summary of the existing security landscape, including different software tools that are already in operation. This project could have a very direct pathway to impact, which is that AI companies directly adopt the suggested applications.

Research Questions

What are some AI applications across different domains, plausible within the next 5 years, that could reduce extreme from AI?
How do they compare against each other?

Methodology
Literature review, Scenario mapping, Expert interviews

Back to Table of Contents

Lessons From Regulation of CSAM
Throughout the Tech Stack

Category: Regulation & policy options
Suggested by: Cornelia Kutterer

Background
The regulation of Child Sexual Abuse Material (CSAM) is a critical and complex issue. The fierce debate of privacy and safety proponents and the inclusion of OTT services into the ePrivacy Directive has highlighted the need to understand how different layers of the tech stack handle CSAM regulation and where the line has to be drawn, technically and to uphold fundamental rights at stake. The ePrivacy Directive, traditionally focused on telecom services, now includes OTT services as a result of the reform of the EECC and a persistent ‘level playing field’ narrative based on the idea that services like WhatsApp or Skype should be regulated the same way. The question remains: How should CSAM be regulated across the tech stack? Should the approach differ depending on where one is situated within the stack, or can a uniform regulatory framework be applied?

Research Questions

How did the legislative trajectory of CSAM evolve and what were the main drivers of the result (stakeholder advocacy, internal structures of the Commission, privacy and safety CSO communities, technology developments)?
How is CSAM regulation currently addressed at different levels of the tech stack (e.g., domain name and infrastructure providers, telcos, OTT services, platform operators)?
- Should the approach to CSAM regulation differ depending on the position within the tech stack? Why or why not?
How has the inclusion of OTT services into the ePrivacy Directive influenced the discussion and implementation of CSAM regulation at the EU level?
What lessons can be learned from historical and ongoing discussions at the EU level regarding the regulation of digital services and their evolution?

Methodology

Case Studies: Analyze specific development of the CSAM regulation at EU level.
Literature Review: Conduct a comprehensive review of existing literature on the regulation of CSAM, the ePrivacy Directive, and the inclusion of OTT services.
Expert Interviews: Engage with experts in technology law, privacy, safety, and content regulation to gain insights into different regulatory approaches and possible implementation challenges

Back to Table of Contents

Should Parts of the Frontier AI Industry
Be Treated Like Public Utilities?

Categories: Regulation & policy options; Antitrust & competition policy; AI market structure & dynamics
Suggested by: Markus Anderljung

Background
I think it’s reasonably likely that we'll see the following development: the AI industry will largely build on a small number of really capable foundation models. At this foundation model layer, there is a natural oligopoly due to economies of scale, learning-by-doing from training huge models, getting access to data from users interacting with the system, probably some network effects, and so on. Upon this foundation model layer, a huge number of downstream applications are built, i.e. we see concentration at the FM layer and then less concentration further down the supply chain. Assumptions that go into the above include (though all don't seem necessary): Scaling will continue to yield dividends, model performance will continue to be a main differentiator, model generality will continue to matter.

If this picture is right, that suggests that it might be right to treat frontier FMs similarly to public utilities. They'll become the bedrock of our economy. At the same time, there will be a large amount of concentration. Often, people have the intuition that the right policy is to increase competition at the FM layer, but this picture suggests that it's more about managing that concentration, and about ensuring that market power is not abused, that certain kinds of vertical integration is warded off, that the products that are offered to people and downstream businesses are safe, reliable, and high quality.

Research Questions

What would the implications (both positive and negative) of treating foundation models as public utilities be?
What effects would this have on market concentration?

Methodology
Literature Review, Expert interviews, Modelling, Case studies

Further Reading

Market concentration implications of foundation models: The Invisible Hand of ChatGPT

Back to Table of Contents

Is Seizing the Benefits of AI More A Matter of Getting Diffusion Right,
Rather Than Development and Deployment?

Category: Regulation & policy options; AI market structure & dynamics
Suggested by: Markus Anderljung

Background
Previous regulatory efforts like the US Executive Order or the EU AI Act have focused on the development and deployment of models, highlighting among other tools evaluations and basing their regulatory requirements on metrics like the amount of compute used to train a model. At the same time, reaping the rewards of AI could to a large extent be a matter of diffusion, or of putting the systems to productive uses in the economy: doing the hard work of integrating AI into businesses and reorganising economic activity to make better use of AI (the classic example here is how it took years to reap the benefits from electricity as it required reorganising factories along assembly lines rather than around a central shaft as was required when using steam power).

As such, the right approach to seize the opportunities of AI might be one where you intervene on the development and deployment layer – to ensure models aren't misused, that they don't cause inadvertent harm, and that downstream developers and users can rely on the products they're using – while at the same time boosting and supporting diffusion. This is also an approach that is more in line with traditional policymaking priorities, and one that is based on opportunities and a positive holistic vision for the future. The role of diffusion will be of core concern for politicians (who will have to manage possibly disruptive effects from labour-enhancing or labour-displacing AI) and AI developers alike, since their profitability will depend on whether most value comes from development (vs. diffusion). A possible starting point would be to look at the share of the value in the tech market that is absorbed by those who develop key products, as opposed to those who manage their diffusion.

I’m not sure the extent to which I agree with the above reasoning, but I find it plausible. If it’s right, that it seems useful to communicate it to policymakers, who too often think that “winning in AI” is about having a domestic OpenAI.

Research Questions

How true is the following hypothesis: “Reaping the rewards of AI is to a large extent a matter of diffusion, or of putting the systems to productive uses in the economy”?
What regulatory implications does this have?
Does a thriving tech ecosystem depend crucially on developers around which it can be built?
- This would suggest that diffusion is less crucial than development.

Methodology
Literature review, Data analysis, Expert interviews

Further Reading

The Diffusion Deficit in Scientific and Technological Power: Re-assessing China’s Rise

Back to Table of Contents

Reconciling Impact Scores for Comprehensive AI Risk Management

Category: Regulation & policy options; Corporate governance; Evals & Audits

Background
AI is a cross-cutting risk that can have impacts which we could seek to measure across a wide number dimensions, from economic impact to loss of human life, environmental damage to public trust. The most appropriate units of measurement are different for each of these. Therefore, it would be useful to conduct a review of different approaches used in different impact assessment fields as to how these are combined, drawing out the pros and cons of each approach, and recommending some combination of these approaches based on which are well suited to AI risk assessment scores (including coming up with what the desirable factors are that would make it well-suited). For this project, it can be assumed the user/reader has already generated the impact and likelihood data for decomposed risks, but that we’re struggling to combine these into a single score for a large risk area as a whole (e.g. job displacement by AI) and justify any exchange rates/other algorithm by which this is done.

Research Questions

How can we create unified, rigorous and consistent Impact scores for AI risk assessments across risks of all domains?
Are there quantitative methods that can be taken from other fields that can address the problem of creating a single impact score that acts as a 'currency converter' between these different impacts, measured with different quantities - and how well could they apply to AI risk assessment?

Methodology
Literature review, Comparative approaches, Expert interviews, Risk management

Back to Table of Contents

Joint Liability in AI Development

Category: Regulation & policy options
Suggested by: Lewis Hammond

Background
If something goes wrong as a product of some phenomena that emerges from multiple AI systems, but is not directly attributable to any one of those AI systems, which developers are/should be liable for that failure? This idea can also be extended to different components of a single AI system, e.g., if one developer provides the training data, another creates the model, another fine-tunes it, etc. There is limited work on this topic out there, but the fact that we don’t yet see many advanced multi-agent interactions indicates that this topic is relatively underexplored. Note that this topic reoccurs under a different idea listed in this post ( “AI delegates”).

Research Questions

How should liability work in the context of multi-agent settings?

Methodology
Policy Analysis, Legal Analysis, Expert Interview, Literature Review

Back to Table of Contents

Pre-Emptive Authorization for AI Training

Categories: Regulation & policy options; Corporate governance
Suggested by: Lennart Heim

Background
Extending the regulation of frontier AI systems to potentially deny specific training attempts is an ambitious proposal that asks for a substantial burden of proof. Why would we consider it necessary to prevent an AI system from even being trained, especially when our regulatory focus is often on the deployment of technologies? After all, we are not aware of the specific risks of an AI system when it has not yet come into existence. Despite these concerns, “pre-emptive authorization” could be warranted due to (i) the risk of proliferation, (ii) potential dangers arising during the training run, and (iii) practical benefits related to the compute moat. More concretely, a regulatory framework could require AI developers to secure a permit before they're allowed to train frontier AI systems. This permit would be evaluated on two factors: the level of responsibility demonstrated by the AI developer (Schuett et al., 2023 and the properties of the training run. This approach to regulation is not unprecedented. We find parallels in other disciplines such as biology, where approvals are frequently mandated before experiments. Moreover, one can draw comparisons with the stringent controls placed on the construction of nuclear weapons (Baker, 2023).

Research Questions

How can a regime based on preemptive authorization be justified? What arguments support this?

Methodology
Literature review, Policy analysis

Further Reading

The Case for Pre-emptive Authorizations for AI Training

Back to Table of Contents

How Can Cost-Benefit Analyses Be Applied to AI Regulation?

Categories: Regulation & policy options
Suggested by: John Halstead

Background
Cost-benefit analyses have become a commonly used tool to inform regulation across different domains, with some governments or agencies making them a mandatory requirement for new programs, guidelines or standards. At the same, the field of AI is riddled with complexities and uncertainties that make these analyses potentially difficult or less informative. Cost-benefit analysis could be used to inform the risk thresholds used in evaluations of frontier AI models. There is currently no science of cost-benefit analysis in AI regulation, and attempts to apply cost-benefit analysis to AI regulation are piecemeal, non-public and ad hoc.

Research Questions

How can cost-benefit analyses be applied to AI regulation and evaluations?
- What might these analyses look like in practice, and what would they involve?
- What difficulties should we expect to arise?
- Why might cost-benefit analysis be the wrong approach to setting risk thresholds in AI evaluations?
- Should cost-benefit analysis be used to assess AI regulations?
- What can we learn from how CBAs are conducted in adjacent domains?
How far should risk thresholds used in AI evaluations be influenced by cost-benefit analysis?

Methodology
Literature review, Expert interviews, Policy analysis, Comparative approaches, Modelling

Further Reading

Back to Table of Contents

What Policy Options Exist for Ensuring That
AI-Generated Content is Identifiable As Such?

Categories: Regulation & policy options
Suggested by: Markus Anderljung

Background
It seems important that at least certain AI-generated content can be identified as such, since we might be headed for a world where AI-generated content and AI-originating actions on the internet are indistinguishable from human-produced content and actions. That doesn’t intuitively seem like a good outcome given the difficulties to have a good overview of how AI is affecting the world or finding levers to improve its effects. Reasons for making AI-generated content identifiable include:

Transparency: Knowing whether someone is interacting with AI content or an AI agent and knowing that a piece of content is AI-generated is important to judge whether it represents real events. Citizens may also have an interest in knowing whether they are engaging with an AI system or not, e.g. as this might inform decisions to seek a second opinion on a decision.
Enforcing different rules for AI-generated content and actions: Companies ask people to verify that they are human to avoid abuse or breakdown of their services (e.g. Captcha).
Incident investigation: As AI systems become more and more integrated into society, we’ll need better information about how and when things go wrong. To do so, it will be important to be able to trace specific incidents or real-world harms to specific AI systems or at the very least to the use of AI systems in the process.
Macro assessments of AI adoption: Currently, there is very little quality public data on the adoption of AI across society. If there were watermarks, we could make such assessments by running a detector e.g. over Facebook.

Possible techniques for ensuring such identification include watermarking, content provenance, retrieval-based detection and post-hoc detection.

Research Questions

What policy options are available to ensure developers take those actions? Possible options include:
- Literally mandating it, but that could be very onerous, so perhaps should only be done for certain systems, e.g. those with a large user-base.
- Requiring that users include identifiers on content they post, which might incentivise companies to put identifiers into their AI tools.
- Tort liability
- Others (?)
Which of these are most promising, and what do they require from other actors in the value chain (e.g. developers, but also users, and regulators)?

Methodology
Literature review, Expert interviews, Policy analysis

Further Reading

Back to Table of Contents

Piloting and Scaling a Monitoring Initiative for AI Capabilities

Categories: Regulation & policy options; Evals & Audits; International governance
Suggested by: Nikhil Mulani

Background
This project would explore how the United States government could pilot and scale an initiative aimed at understanding the state of AI capabilities internationally, in order to support the identification of and preparation for potential misuse or accident threats. Methods to implement such an initiative could include creating a new initiative within an existing agency, creating a new independent agency, or leveraging public-private partnerships with businesses or academia and open-source intelligence techniques.

Research Questions

How could the United States government pilot and scale such an initiative?
- Are there existing government initiatives that already cover some aspects of this, which could be expanded?
What different implementation structures exist for such a program, and how do they compare against each other?
Where would funding for this program come from?

Methodology
Foresight, Expert interviews, Case studies

Further Reading

China’s Advanced AI Research, CSET
Open-Source Intelligence for S&T Analysis, CSET
Why and How Governments Should Monitor AI Development, CSER

Back to Table of Contents

Case Studies on US Regulation That
Can Inform AI Governance

Categories: Regulation & policy options
Suggested by: Bill Anderson-Samways

Background:
IAPS has recently published a paper which uses quantitative case-study selection techniques to systematically identify US agencies / individual US regulations that might hold lessons for advanced AI regulation[16]. The aim was to identify cases that other researchers can examine in depth. Concrete case-studies are especially useful for determining and influencing regulatory design considerations, and there are already established audiences (for example funders and policymakers) who are interested in such case-studies. Examples include the Environmental Protection Agency; various financial regulators, such as the Fed; the Office of Commercial Space Transportation; and regulatory functions in the Departments of Energy and Defense.

Note that a number of case studies on some of the aforementioned entities or specific regulatory regimes within them have already been carried out, and they can be found here.

Research Questions

What can we learn from case studies of other regulatory agencies and regulatory frameworks / regimes?

Methodology
Expert interviews, Case studies, Literature review

Further Reading

Case studies on social-welfare-based standards in various industries

Back to Table of Contents

16. They’ve used measures of five variables that seem relevant to advanced AI regulation: (1) intensiveness; (2) expertise; (3) enforcement against powerful companies; (4) use of risk-assessments; and (5) focus on uncertain phenomena.

State Use of AI

Censorship, Surveillance and Democratic Backsliding:
Which Democracies Engage in Online Repression The Most?

Categories: State use of AI

Background
Advances in AI threaten to make Internet censorship and surveillance vastly more sophisticated and scalable, a possibility already emerging in countries such as China with authoritarian regimes. Automated censorship and mass surveillance of social media and other online platforms could also contribute to democratic backsliding, in which freely elected political leaders undermine democratic institutions and establish themselves as autocrats. (Recent examples of democratic backsliding include Hungary under Orban and Turkey under Erdogan).

Research Questions

Which democracies are most at risk if advances in AI make it easier for governments to crack down on their political opponents?

Methodology

Review which democratic governments are already engaging in extensive Internet censorship and surveillance with existing technologies. A good place to start would be the dataset produced by the Digital Society Project, which measures censorship and surveillance of the Internet by governments across the world over the last two decades.[17]
A research project could combine this dataset with other data sources measuring economic, social, and political factors, to identify predictors of how much online repression different democratic countries carry out.

Back to Table of Contents

17. For each country and year, the dataset assigns numeric scores measuring each government's use of Internet filtering, social media censorship, and social media monitoring. It also contains data on each government’s technical capacity to carry out such activities, regardless of whether it does so in practice. According to these data, some countries with similar levels of democracy (as measured by sources such as Polity5 and Freedom House) and technical capacity differ substantially in the degree to which they censor and monitor social media or block websites. For example, in 2021 Colombia carried out much more website blocking, social media censorship, and social media surveillance than Mexico, even though the two countries had very similar democracy scores and Mexico had greater technical capacity to monitor and control its citizens' online activity.

What Rules Should the US Government Set Regarding
Government Purchases of AI?

Category: State use of AI; Regulation policy options
Suggested by: Nikhil Mulani

Background
Public procurement has been put forward as one of many levers available to governments to steer the development and deployment of AI. For example, the US government recently issued a request for Information on responsible procurement of artificial intelligence in government. But also on a local level, procurement could become the tool of choice for those seeking to leverage the soft powers of government for positively influencing the trajectory of AI. The US federal government purchases nearly 700bn $ worth of goods and services every year, making it a major customer with significant power over the market. At the same time, there is little track record of procurement guidelines for safe and responsible AI.

Research Questions

What rules should the US government set regarding government purchases of AI?

Methodology
Expert interviews, Policy analysis

Further Reading

Back to Table of Contents

Technical and Compute Governance

Estimates of Losses of Military Supplies

Categories: Technical and compute governances

Background
In the future, it may make sense to treat certain (large quantities of) AI chips with military-grade security. It would be interesting to know how secure this would be against an unaligned actor diverting such chips to their own data centers. By using loss rates for similar physical assets that are subject to these security standards as reference classes, we may be able to bound our estimates of how secure we can make the supply and custody chains of AI chips.

Research Questions

What are loss[18] rates for analogous physical assets subjected to military-grade security?[19]
- Traits that make something analogous to an AI chip:
  1. It’s relatively small (so a gun might be analogous while a whole vehicle is less so).
  2. It’s not very dangerous to handle (so an unloaded gun might be analogous but uranium and landmines aren’t).
  3. It’s manufactured in the civilian sector.

Methodology
Forecasting, Quantitative modelling, Scenario planning, Expert interviews

Back to Table of Contents

18. Both “unsolved disappearance” and known exfiltration.
19. E.g., military firearms, high-tech military gear.

Stock and Flow Accounting Case Studies

Categories: Technical and compute governance

Background
One likely building block for any maximally secure compute governance regime is stock and flow accounting of (some kinds of) compute: e.g., requiring real time accurate declaration to regulators of who possesses which uniquely numbered regulated chips, with penalties for undeclared or unauthorised transfers. To understand the optimal design and feasibility of such a regime, it would be useful to know more about historical analogies for similar regimes. An ideal analogy will have many of the following traits:

The thing being tracked is a physical object
The thing being tracked is economically important
The thing being tracked is dual-use
The tracking regime requires registration of current ownership and any transfers
The tracking regime imposes penalties for failing to register ownership or transfer

Case studies on stock[20]-and-flow[21] tracking for items that meet many of the above criteria would be very valuable. Such case studies should include:

A description of the item being tracked, and the reason governments want to track it.
Methods that governments use to track the items.
Penalties for loss or misrepresentation of custody of the item.
Effectiveness of the tracking regime (ideally with quantitative estimates of how much of the item is lost or illicitly transferred).

Promising candidates might[22] include:

Firearms
Automobiles
Certain pharmaceutical products[23]
Aircraft
Chemical weapons and precursors[24]
High-risk chemicals
Select biodefense agents and toxins

Less promising—but still plausible—candidates may include:

ITAR-controlled items[25]
Real estate[26]
Financial instruments[27]

There are already good case studies on tracking nuclear fissile material, so it is not a promising area of additional research at the moment.

Research Questions

What can we learn from case studies on stock-and-flow tracking?
How could such a regime be designed compute?

Methodology
Case studies, Literature review, Expert interviews

Further Reading

Tracking Compute Stocks and Flows: Case Studies?

Back to Table of Contents

20. I.e., where are existing stocks of such items?
21. I.e., who is transferring such items?
22. We haven’t investigated all of these yet (even preliminarily), so it’s possible that many of these are not in fact worth investigating deeply.
23. Suggested here.
24. Suggested here.
25. Suggested here. However, to my knowledge, the ITAR does not actually require that the physical whereabouts of controlled items be tracked: only that certain transfers are prohibited.
26. Land registration is a well-established feature of many property law systems. However, since real estate is by its nature immovable, the problem of hiding its location is nonexistent.
27. Since most financial instruments are not physical, this is a less-promising case study.

Learning From Chain of Custody Applications in Other Industries

Category: Technical and compute governance

Background
The pharmaceutical industry uses a chain of custody mechanism to track and trace the supply chain (see e.g. here). It’s conceivable that this represents a reproducible regime that could be used for semiconductor supply chains, and better understanding the strengths and weaknesses of these approaches could be useful.

Research Questions

What can we learn from other industries about chain of custody for compute supply chains?

Methodology
Case studies, Literature review, Expert interviews

Back to Table of Contents

Probability of a Taiwan “Broken Nest Strategy”

Category: Technical and compute governance; International governance; AI and the military

Background
Some commentators have suggested that Taiwan and/or the US should pre-commit to destroying fabs (e.g. TSMC) in Taiwan in case of a PRC invasion, as both a method of deterrence and a way to deny China access to strategic technologies there. Others disagree.

Research Questions

How effectively could the US and its allies in Asia and Europe remotely render inoperable key equipment in key fabs in Taiwan (e.g., by revoking software licenses)?
Conditional on the US believing that the PRC will imminently control the relevant areas of Taiwan, what is the probability that the US will successfully kinetically destroy fabs in Taiwan?
Conditional on Taiwan believing that the PRC will imminently control the relevant areas of Taiwan, what is the probability that Taiwan will successfully kinetically destroy its fabs?
What does the PRC currently believe about the above?
What about non-physical assets such as human capital?

Methodology
Quantitative modelling, Judgemental Forecasting, Scenario mapping, Policy Analysis, Expert interviews

Back to Table of Contents

Sovereign AI Compute

Categories: Technical and compute governance; Regulation & policy options; International governance

Background
We see countries attempting to pursue more sovereign and independent infrastructures that can support AI development domestically while reducing dependencies on other actors (incl. geopolitical adversaries or economic competitors). This project would aim at getting a better understanding of the implications those attempts might have on AI development on the one hand and international relations and security on the other. The answer might also vary depending on the country one examines, or what underlying assumptions are held about factors like takeoff speeds, multipolarity and the success of future regulatory efforts on an international level. Additional factors include the location and jurisdiction of data centers (and who owns them), respective data privacy laws and what use cases one is thinking about (e.g. for national security purposes).

Research Questions

Should nations pursue sovereign AI infrastructure? If yes, for which reasons? (See US NAIRR or UK Future of Compute Review)
Which alternatives exist? What about nations that cannot compete at the frontier? Should there be partnerships (e.g., EuroHPC)?

Methodology
Literature review, Policy analysis, Data analysis, Expert interviews, Comparative approaches

Back to Table of Contents

Foundational Challenges in Assuring Alignment and
Safety of Large Language Models

Category: Technical and compute governance
Suggested by: Anton Korinek

Background
Large language models have taken the world by storm, and they are currently the closest analogue to what might one day become artificial general intelligence. At the same time – and partly due to the rapid rise of their capabilities and widespread adoption – research that answers foundational questions about the safety and alignment of LLMs is lagging behind. This has led to the development of a long backlog of research questions and problems in need of solving. One paper proposes 200+ research questions across 18 challenges, all of which are in need of answering. A concrete project would take on one of these questions, though overlap with other questions listed in the document may exist, and it could be fruitful to explore this in a broader context, too.

Research Questions
Top level: What are the foundational challenges associated with the alignment of large language models, and how can they be solved? The paper divides these into 3 categories:

Scientific understanding of LLMs (e.g. scaling, agency, multi-agent safety and safety-performance tradeoffs)
Development and deployment methods (e.g. evals, interpretability and vulnerabilities)
Sociotechnical challenges (e.g. value uncertainty, dual-use, trustworthiness, socio-economic impacts, general governance)

Methodology
Various (depending on the respective research question)

Further Reading

Foundational Challenges in Assuring Alignment and Safety of Large Language Models

Back to Table of Contents

Extent of Export Control Circumvention

Categories: Technical and compute governance; International governance; Regulation & policy options

Background
Export controls are a cornerstone of US attempts to prevent rival nations’ access to AI hardware technologies. Understanding the extent of circumvention of these controls (i.e., how many controlled items eventually arrive in rival nations despite export controls) is important to understanding how much such controls might actually translate into denial of access by targeted countries.

Research Questions

What is the quantitative extent of export control circumvention for controlled items analogous to AI chips?
- Ideally, this would be denominated in total controlled stocks or flows of such controlled items.

Methodology
Data analysis, Expert interviews

Further Reading

Mapping the Chip Smuggling Pipeline and Improving Export Control Compliance

Back to Table of Contents

OP/s Threshold Adjustments for Performance

Categories: Technical and compute governance; Regulation & policy options

Background
How should the OP/s threshold (e.g. in US chip export controls) be adjusted to account for performance variations across different bit-widths (in OP/s but also overall)? This is critical for accurately penalizing and incentivizing the development of AI systems. Existing metrics may disproportionately favor smaller bit-widths over larger ones. E.g., your metric for FP16 is only 2x higher than FP32, while the total performance gains might be higher. Smaller bit-widths are particularly advantageous for machine learning (ML) development and deployment, making them a focus for more precise AI applications. Reduced bit-width generally results in performance acceleration, often exceeding linear improvement. However, implementing such changes in hardware requires a couple of years.

So if smaller bit-widths offer hardware performance advantages (if supported), then it’s advantageous to leverage them for development and deployment. Smaller bit-widths are more easily leveraged for inference via post-training quantization (there are implementations via int4). Recent studies primarily focus on cost and memory footprint reductions, with limited analysis on the acceleration effects. (Mostly academics who want to deploy models on their limited number of GPUs with limited memory.) FP16 has become the default for training and FP8 might be next. The H100 is already supporting FP8.

Research Questions

How should the OP/s threshold be adjusted to account for performance variations across different bit-widths (in OP/s but also overall)?
Do we see a reduced performance for using X 8bit FLOP vs X 16bit FLOP for training a X FLOP model?
- Current consensus suggests no performance loss, meaning 16-bit and 32-bit FLOPs yield similar capabilities.
- While reduced bit-width generally works until a certain point, few studies focus on architecture modifications to accommodate even lower bit-widths (<8bit) during training.

Methodology
Literature review, Modeling, Data analysis

Back to Table of Contents

BOTECs of Inference Compute Needs

Categories: Technical and compute governance; Regulation & policy options
Suggested by: Markus Anderljung

Background
It could be useful to have well-evidenced BOTECs to assess how much inference compute (in terms of FLOP, FLOP/s, and hardware required) is needed for various consequential AI use-cases. Such use-cases might involve: Authoritarian use-cases (e.g. surveillance of an entire population, censoring the internet), election interference (e.g. running 1m fake social media accounts to reduce voter turnout in a certain demographic), and AI-enabled online fraud (e.g. running 1m deepfake robocalls simultaneously aiming to get someone to transfer money into a bank account). In some of these cases, it may also be interesting to conduct a BOTEC on the bang-for-buck of the use case. It’s not clear these BOTECs should be widely published, but they ought to be useful for policymakers, and could inform broader strategies around risk management for hazards arising from misuse.

Why might this matter? A lot of compute governance efforts focus on the compute needed for training. I think inference deserves more attention. Inference is what will lead to AI systems having a real impact in the world, and we should expect that a system’s impact should at least monotonically increase with the number of inferences run on it.

Research Questions

How much inference compute would be needed for different consequential AI use-cases?

Methodology
Quantitative modelling (BOTECs)

Back to Table of Contents

Can We Use Technical Tools to Make Open-Source
Models Less Useful for Misuse?

Categories: Technical and compute governance
Suggested by: Markus Anderljung

Background
While open-source AI systems have a large number of benefits, they are also easier to a) modify (e.g. removing safety filters or enhancing relevant capabilities) and b) misuse (attempts to monitor for misuse can be more easily thwarted + attributing harm to an open-source model is harder than attributing harm to a closed-source one). In short, release strategies face a misuse-use tradeoff. How can this tradeoff be reduced? One approach is to make closed-source models more like open ones, e.g. via structured access techniques and via governments mandating transparency. Another approach is to make open-source models less useful for misuse.

Techniques to reduce the misuse-potential of open-source AI systems would have to be i) effective (i.e. significantly reducing misuse potential) and ii) tamper-proof (i.e. difficult to circumvent). See a list of potential techniques under “Research Questions” below.

To ensure that techniques are actually implemented, several approaches are in principle possible:

Targeting the developer: Regulation saying that OS models that meet certain criteria need to have certain safeguards implemented
Targeting the host: Regulation saying that model hosting platforms need to ensure certain criteria are met, including by adding safeguards or maintaining monitoring and verification systems:
It’s possible more and more OS companies will release their models with a license that generally allows free use but requires payment from sufficiently sized companies or for sufficiently broad use. Those companies would have incentives to develop fingerprinting techniques and the like to be able to identify whether someone is using their system or not.
Targeting the server: Companies that offer API access to OS models need to implement certain safeguards, similar to those that come with providing API access to closed-source models.

This project would aim to explore the space of such interventions.

Research Questions

Can we use technical tools to make open-source models less useful for misuse? For each technique, it would be good to know:
- How would this tool work?
- How does it reduce misuse risk?
- How easy is it to circumvent?
- What’s the current state of science?
Candidate techniques include:
- Exclusion of misuse-relevant training data
- Self-destructing models (see e.g. here)
- Forgetting/Unlearning (see e.g. here)
- Model identifiers or “fingerprints”
- Watermarking and signatures (see e.g. here)
- Safety filters added by model hosters

Methodology
Policy Analysis, Legal Analysis, Expert Interview, Literature Review

Further Reading

Back to Table of Contents

Compute Production Gap, Data Centers and
Data Asymmetry in China

Categories: Technical and compute governance; International governance

Background
For a variety of strategic questions, the question of who is leading in AI and by how much is crucial, and compute is a central input to this. Therefore, work that estimates the gap between the U.S. and China could inform the strategy of actors across sectors. Similarly, mapping and rating Chinese competitiveness in the realm of data centres could be insightful.

Research Questions

When would indigenous Chinese compute manufacturing capabilities equal US + allies’ 2024 indigenous compute manufacturing capabilities? When would indigenous Chinese compute manufacturing capabilities equal US + allies’ future indigenous compute manufacturing capabilities? (I.e., in what year would they equalize?)
What events would change your expectations significantly?
- E.g. What’s the probability that China invents some “flip the board” chip manufacturing technology that circumvents key external supply chain bottlenecks (e.g., EUV)?
- E.g. Changes in US regulation with respect to the compute supply chain
What are the biggest data centers in China? Are there patterns to where and how China builds state of the art data centers? (This is relevant to international monitoring & verification schemes.)
How capable is Chinese endogenous ability to build and operate state of the art data centers (assuming access to the relevant inputs)? How much aggregate compute does China have across all data centers? What are the biggest computations run in or across Chinese datacenters?
How would you characterize the organization of the datacenter / HPC industries? What types of institutions (e.g. public-private partnerships?) are employed?
It might turn out that certain kinds of data are important for some AI capabilities. A few examples:
- Expert human demonstrations might be necessary to create AI that can automate those capabilities (e.g. creating a chip design AI might require chip design expertise)
- Large quantities of data from some segments of the labor market might be necessary for certain cognitive capabilities (e.g. large quantities of labels)
- If this is the case, what can we say about what capabilities we expect China to (a) prioritize and (b) successfully accomplish?

Methodology
Data analysis, Forecasting, Quantitative modelling, Literature review, Expert interviews

Back to Table of Contents

Using Compute for Verifiable Claims, Assurances, and More

Categories: Technical and compute governance; Corporate governance; Regulation & policy options
Suggested by: Lennart Heim

Background
It’s conceivable that it’ll be important in the future to make verifiable claims about your AI development and deployment. This can be across a variety of actors, such as companies and nation-states. In particular, technical ways (as these can be more trusted) to make these verifiable claims are needed. They can also reduce the social costs, increase the bargaining range, and the like. Precedents in the nuclear industry exist that inspire hope that these are at least in theory possible. So these mechanisms can allow us to achieve certain AI governance goals, enabled by hardware. Let’s just call it “tech supported diplomacy”.

Recent advancements in “proof-of” research—like proof of training algorithms or proof of data inclusion/exclusion—may become crucial for future governance. These mechanisms enable developers to make credible, verifiable statements about their development processes. For instance, they could serve as the foundation for a future non-proliferation regime, e.g., when developers are required to not train system above a certain threshold of computational resources. Such a mechanism could facilitate international verification, enabling countries to validate each other's claims, thereby reducing social and political tensions. For the cluster to be universally trusted, it would need to be perceived as neutral. Future iterations of this proposal could consider establishing a 'neutral' secure cluster to meet this requirement.

In general, we try to address information asymmetry: The trusted cluster aims to resolve information imbalances that prevent the unwarranted proliferation of AI capabilities. It does so by providing a trusted environment for both parties, as the verification process may sometimes necessitate disclosing confidential information such as training data, model weights, and proprietary code.

Research Questions

How can companies make verifiable claims about their AI development and deployment?

Methodology
Literature review, Case studies, Expert interviews, Experiments, Surveys

Lennart Heim has also provided this overview of additional projects that are potentially impactful in the compute governance space.

Back to Table of Contents

Understanding Training vs. Inference

Categories: Technical and compute governance

Background
There is a common misconception that “you need lots of compute for training, but once a model is trained, it’s over, and everyone can deploy it.” If the misunderstanding at the bottom of this is one that’s based on mixing up “training” and “inference”, then clearing up the distinction between the two could be a valuable use of a researcher’s time. Inference is a “continuous process”, it’s deploying a model, serving a product, providing a service, whereas training is “a bet”, building a product, etc. Down the line, this could help inform regulatory decisions on governing both training and deployment. It’s also important since if you own ML hardware you have to decide if you want to use it for training or inference. That decision became even more important over the recent months. Training a model means not serving inference for a product.

Research Questions

How can we draw a clear distinction between training and inference? How can this difference be communicated well, including to non-technical audiences?
What are the regulatory implications of this? (potentially out of scope)

Methodology
Literature review, Distillation, Expert interviews

Back to Table of Contents

Compute Replacement

Categories: Technical and compute governance

Background
Compute could get replaced, which would render previous compute irrelevant; thereby increasing governance capacity. This is compared to e.g., weapons where we don't have exponential progress. It’s more like, swords got replace by firearms. Nonetheless there might be one constant threshold – such as with firearms where just harms get done – even if the defender would have an advantage. If the previous generation compute stays relevant and makes up a significant share of “all of the compute” (here the available performance per second [FLOP/s]), then our intervention is not effective as they lack the on-chip mechanism. Consequently, this hinges on the next generation to have increased performance. This might inform us when we might need to start implementing something (therefore also when we would need to start advocating) and when it starts being effective. It also informs us about the available compute capacity by using prev-gen compute and could be helpful for estimates of the capabilities of other actors.

Research Questions

What’s the share of any chip in a given year of total available compute performance?
What’s the typical lifetime of a chip in a datacenter?
With smaller feature sizes, those chips get more susceptible to cosmic radiation and degrade over time. How long would this take? Maybe there’s an upper limit of chip lifetime? That’d be great then chips will get replaced independently of performance increase.

Methodology
Modelling, Data analysis

Further Reading

Crucial Considerations for Compute Governance

Back to Table of Contents

Which Compute? Defining The Regulatory
Target for Compute Governance

Categories: Technical and compute governance; Regulation & policy options

Background
Chips have become an integral aspect of modern society, with devices ranging from smartphones to home appliances relying on this technology. As a result, it has become ubiquitous, making it challenging to leverage it for numerous governance capacities. Targeting all the compute worldwide is neither feasible nor desirable. Such an approach would inevitably impact the majority of compute which is not of relevance for frontier AI activities, and would represent a significant invasion of privacy while also being overly blunt in its implementation. Therefore, there is a need for defining a (better? more appropriate?) regulatory target for compute governance.

Research Questions

What is AI compute?
Which parts of the computational infrastructure best regulate AI development and deployment while minimising the downsides?
What should the regulatory target be?

Methodology
Literature review, Expert interviews, Modelling

Future Reading

Computing Power and the Governance of Artificial Intelligence

Back to Table of Contents