LLMs are never going to become superintelligent. However, an LLM WILL kill us all if someone decides that it IS superintelligent and so puts it in charge of nuclear weapons.
“You’re right to push back on that, I nuked Albania instead doing what I was supposed to do, that’s my bad”.
There’s no way an LLM is getting into those systems. They have no networking and rely on physical dials and switches. Even the code is in a completely proprietary language, using hardware custom made by the military. There’s no training data in the world that’s gonna give an LLM the kind of knowledge to get into that system. The best it could do is drive some admiral in the navy to AI psychosis and convince him to press the button, but even that’s difficult because it requires more than one guy.