*This essay was submitted to the* [*AI Progress Essay Contest*](https://www.metaculus.com/project/ai-fortified-essay-contest/)*, an initiative that focused on the timing and impact of transformative artificial intelligence. You can read the results of the contest and the winning essays* [*here*](https://www.metaculus.com/questions/11429/results-from-the-ai-progress-essay-contest/)*.*

  


Artificial General Intelligence will almost certainly happen at some point in the future of humanity. This might be in the far future, in which case it would be hard to predict the society and culture into which it would emerge. But it could also happen in [the](https://www.metaculus.com/questions/384/human-machine-intelligence-parity-by-2040/) [near](https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines) [future](https://astralcodexten.substack.com/p/biological-anchors-a-trick-that-might?s=r) — within 30 years— and about this society we can and will speculate a little. In either case it is misaligned in a bad way— if it quickly destroys humanity— then there is little in the way of AGI gains to think about. This piece is not about such a scenario; instead, we’ll take a rough look at the plausible future in which this does not happen, in which either all the alignment problems are solved or we get lucky in some other way.

Who ends up controlling the goals of the Intelligence? In the short run, the most likely candidate to create AGI is probably well-known private AI research corporation, such as DeepMind or OpenAI. This seems reasonable to expect, since such labs are the ones producing breakthroughs at the cutting edge of AI today. The forecasts of this family of Metaculus questions agree: 

*Will the research group that is responsible for developong the first AGI be part of a* [*nationalized organization*](https://www.metaculus.com/questions/8325/nationalized-organization-develops-first-agi/) */* [*non-profit organization*](https://www.metaculus.com/questions/8328/non-profit-organization-develops-first-agi/) */* [*government project*](https://www.metaculus.com/questions/8327/government-project-develops-first-agi/) */* [*university group*](https://www.metaculus.com/questions/8326/university-group-develops-first-agi/) */* [*for-profit corporation*](https://www.metaculus.com/questions/8324/corporation-develops-first-agi/)*?*

(It’s quite a bit harder to speculate on this question in the long run. We don’t know the kinds of institutions that will exist, what world governments’ responses to impending AGI will be like (will they even understand it?), and even what the research environment at different kinds of institutions will be like. If AGI is created after 2060, it is probably the case that it will have been created by an entity that currently doesn’t exist.)

Further, forecasters on Metaculus predict huge gains in global GDP— a proxy for global production and general welfare— following the creation of Human-Level AI. 

It is widely accepted that competent AGI (if well-aligned) could become a major player in virtually any industry, helping solve a large number of problems, finding major arbitrage opportunities, and stamping out inefficiencies. Not even necessarily referring to general AI, Alphabet seems to expect this, since they’ve been funding DeepMind’s activities since acquiring it for $600mn in 2014. Alphabet has [invested](https://www.standard.co.uk/business/google-deepmind-profit-sales-demis-hassabis-b958879.html) around $2bn into the company over the years, waiving its debts and paying its expenses, presumably with the expectation of large returns down the line— with the AI company finally turning a net profit in 2020.



---

What would the consequences be then, if a private corporation creates the first useful powerful AGI? There is little precedent in history for a single private entity seeing such transformative gains in resources. In effect, the economics might resemble the scenario of someone happening upon a hitherto undiscovered natural resource that promises to fuel spectacular productivity gains. If we imagine AGI capabilities as a resource to be extracted— like a natural resource, only artificial— we can try to guess what happens by looking at historical occasions when some entity found itself with massive exclusive access to new resources.

Oil companies— especially nationalized ones with monopolies over large oil reserves— can make massive profits from resource extraction. Saudi Aramco, the Saudi Arabian government owned oil company, holds a monopoly over oil extraction from the country’s vast oil reserves. It has long held a position as one of the largest companies in the world by revenue, and accounts for nearly 40% of the country’s GDP and 87% of Saudi budget revenues, and in many years has been among the world's biggest companies by [profit](https://en.wikipedia.org/wiki/List_of_largest_corporate_profits_and_losses). 

Another example: trading powers— particularly European companies— found themselves in such situations during the imperial era. The Dutch East India Company (“VOC”, est. 1602) maintained a monopoly over the trade of many products from the East Indies. This effectively gave them exclusive rights to extracting a “resource”— in this case, spices, metals, and bulk products (usually from Asia). This allowed them to generate incredible profits, selling products into Indian and European markets at many times the cost price. Initially backed by the Dutch government, it gained power and influence and grew to become highly autonomous, eventually controlling a large military force and wielding quasi-governmental powers. Indeed, as the richest company in history upto that point and a superpower company-state, the VOC created jobs, trading posts, and factories in the process of its extraction. It is considered to have been a significant driving force behind the Dutch economic miracle and the Dutch Golden Age, by bringing wealth, developing industries, and creating jobs in the Netherlands.

The VOC seems to have had reasonable success in transferring its gains to Dutch citizens, at least according to the popular historical narrative. But not all governments and resource-extracting entities are set up in a way that allows for gains in wealth to reach the masses. How well this happens depends on legal incentives, corruption, competition, taxation, government transparency, the justice system, and many other factors. By looking at these factors for each government, we can make vague guesses on how successfully a country might ensure that the gains from AGI reach its individual citizens.

The [Resource Governance Index](https://resourcegovernanceindex.org/) tries to measure these factors for different countries. While the measurement is for very specific resource industries (Mining, Oil & Gas), it might be a good indicator of the general systems in place that ensure that resource exploitation is equitable. This is supported by the fact that for the countries whose both Mining and Oil & Gas Governance Indices are generated by the Resource Governance Institute, the values are highly correlated:

  


![](https://metaculus-media.s3.amazonaws.com/user_uploaded/RGI_chart.jpeg)

Among the top 20 countries which might be the birthplace of AGI, by looking at the [Nature AI Research Index](https://www.natureindex.com/supplements/nature-index-2020-ai/tables/countries), the country with the lowest Resource Governance score is China (RGI 55, 2017 Oil & Gas). All other countries with an index have a score of at least 70 (some countries do not have scores). This should be taken as somewhat indicative of the outcome if China were to acquire monopoly on AGI.

These are simply vague “outside views” into the question of AI resource governance. A revolution such as the one that AGI would bring about is impossible to predict, but we know that it will be high variance. In particular, if the resources are not distributed equitably, some entities might see gains far beyond anything imaginable today.

From cases like the VOC and the British East India Company, we know that it is not unprecedented for a single private entity to wield geopolitical superpower status. OpenAI [expects](https://openai.com/blog/openai-lp/) to be in such a position: “*[...] if we are successful, we expect to generate orders of magnitude more value than we’d owe to people who invest in or work at OpenAI LP [...]*” We can even lower-bound the returns OpenAI expects to have with this off-hand statement: their investment is at least $1bn (from just Microsoft), which according to their profit structure (see below) would mean they owe at least $100bn of returns to investors. Taking ‘orders' to mean just two orders of magnitude, this would mean OpenAI expects to eventually generate returns at least on the order of $10tn— more than 10% the 2020 world GDP.

The [Windfall Clause](https://www.fhi.ox.ac.uk/windfallclause/) is a proposed voluntary commitment for AGI research labs by the Future of Humanity Institute, that binds them to among other things cap their gains and donate the excess. Similar clauses written into the mission statements or bylaws of a company amount to a commitment to some sort of resource distribution of AGI. OpenAI, for example, “[caps](https://openai.com/blog/openai-lp/)” its investors’ profits at 100x the investment or lower, committing the excess returns to the holding nonprofit. There appears to be pressure in the AGI field to be humanity-aligned in this manner: Alphabet’s DeepMind [reportedly](https://www.theverge.com/2021/5/21/22447272/deepmind-google-independence-tensions-negotiations-wsj-report) unsuccessfully fought Alphabet for nonprofit status. Research labs doing this would add pressure on others, and top researchers may be more likely to join the more humanity-aligned corporation. Other companies committing to the Windfall Clause or to profit structures like that of OpenAI would be a good sign towards a world with equality of gains.