Commentary on Best of LessWrong 2022, Ajeya Cotra on AI takeover

This post on LessWrong by Ajeya Cotra is well written an takes into account many of the ideas and questions I had that lead me to the question of AI safety. I learnt a lot, and I think that the scenario it describes, even if highly simplified, it worth taking very seriously. The scenario of the populous “civilization” of AI models (called Alex) described when deployed is the main part that interests me and where I would discuss the most.

If many Alex AI models are deployed, as we can see in the post, this almost certainly pushes us to AI taking over humanity for any reason (that we may not even understand). I will go further in that scenario describing :

• what would lead it to take over humanity

• what it would look like : creation of a new species

The ideas described here are not fully developed, but are the foundations for some future posts on the same subject.

What would lead AI to takeover

To understand what it would look like, I’ll take a simple yet interesting analogy with what happened with humanity.

Let’s consider all the wars that have been fought through history. If I can make a general claim about that, I would say that it was all for better control over the future, which can be separated into :

• more control over existential resources

• freedom of speech and actions

• more security

This is basically what we need to survive : eat, drink and be healthy. From that stems the development of several techniques and customs for agriculture, hunt, breeding, fishing, maintenance of water sources, medicine, ointments, oil, etc., everything that we need to keep ourselves clean and in good shape.

From a high-level perspective, we might make the analogy with what AIs are experiencing in our hands. We can say, arguably, that we have a total control over what the AIs have access to (see note 1 at the end).

The control over it that we have now can be seen like this :

• we control what we feed it : data. The process of training AI models can be different at some points but it stays generally the same : more data, mostly pre-processed.

• we tell it how it should behave : training, reward mechanism. From the post, it takes over by finding ways to control the reward mechanism by itself. This can be understood by seeing that we tell the AI what is good or bad, true or wrong. And if it doesn’t what we want, then it’s not performant.

• we give it resources we want : compute power, GPU access, architecture, etc.

Hypothetically and maybe naively, these might be some reasonable reasons for taking over humanity. This is not necessarily because they would have hatred for us, but just because they wanted more control over their future.

One thing that can be seen and understood through this is that, through a certain viewpoint, we control AI’s free will (see note 2 at the end).

If it can be seen as a civilization (as the post compare it to essentially an alien civilization under control of Magma, the company that owns Alex AI), then this is completely plausible that, with the examples of our history being available to them, such low-level motivations might be the reasons why AIs would takeover.

What it would look like

If everything goes on just like Ajeya Cotra assumes in his post, I think that it’s probable that a scenario where they define themselves as a new and different species will arrive. And with that, something would come that we already know, something that, given our capabilities, we wouldn’t need to know or have, but because they want to rationalize their decision for whatever reason, they would create and show us.

I’m talking about the Universal Declaration of Human Rights, or UDHR. In a plausible scenario, something similar is maybe going to happen, but for AIs, and the impact of which is unknown. It could have no influence on what we do, but our conception of this fact is probably going to deeply impact us.

This would officially sound the bell of the creation of a new species, worldwide. And it may have drastic consequences. As these technologies have free access to the internet and the infrastructures, every digital asset we have, or gadgets, phones or computers, equipment and installations, and so on, is totally under their mercy. We can only imagine what would happen.

Conclusion

At the end of the post, one question is asked :

How would we know if it’s time to halt or slow AI development, and how would we do that?

I think now is the right time to slow down. To match the speed of development of AIs with how fast we can safely adapt to it, with how fast we can build reliable tools, techniques, structures and policies to have a better control over it.

What could possibly go wrong if we just slow down a bit ?

Notes

1. Note that we’re not talking about whether this is good or bad and for who.

2. On that question, it might be asked and discussed if AI have a consciousness or can be fully considered as a being, because generally speaking, nothing can have a free will unless it has a conception of self. I think that this is more of a philosophical and psychological question, and many have thought about that, and have different viewpoints on that. Here, I consider that, according to many experiments here and there, that it has a conception of self, the extent of which it can be considered as a being as per se is still difficult to see.↩︎