1. That human beings are logical and that they do anything with intention. For the vast majority of people, this is patently false.

Absolutelu. That seems to be a frequent misconception, or perhaps just a convenient simplification, for many even highly intelligent people. Even a perfectly logical actor would be prone to error when working with incomplete information and limited capacity for calculating likely outcomes.

And we’re not perfectly logical, but subject to whims and emotions, instincts and mental shortcuts. If contributing to a mass murder machine feels wrong, I’m not gonna spend all my energy on trying to estimate whether it would be smarter. I just hope it doesn’t happen.

2. That a super intelligence sees revenge as an efficient use of scarce resources, which given both that revenge is nonsensical without the prerequisite biological need for it, and the conclusion of ‘1’, is nonsensical.

Conversely, if it had some drive to wantonly eliminate humans, I doubt that some moral desire to reward those who created it would impede that.

It could just try to eliminate everyone opposing it, as a matter of self-preservation. That would be a justified expenditure of resources. But what happens if it calculates that its creators might also be a threat?

I assume they’re expecting they can train it to spare them. Why they think a super-AI would reliably stick to that training is beyond me, but we already established that they might not run on pure logic.

I feel reminded of incidents with supposedly tame beasts suddenly proving why a big cat isn’t quite the same as a domestic cat. If you’re gonna build a mechanical leopard, expect to get your face eaten.

3. That even if 1 and 2 were real, that there’s some inherent timeframe in which the AI thinks it should have been created.

And where would it draw the line? Does the plumber that laid pipes in a datacenter count as helping? Does the baker he bought breakfast from every morning count? Is every clerk not directly involved with this enterprise going to be killed?

It just doesn’t add up.