Thanks for pointing that out, I have not read about it. I guess it’s something in the line of HRM or latent reasoning, or even next latent prediction… Whatever it is, I would highly suspect that it collapses to a “normal” reasoning trace (the leaked reasoning traces for previous models already show “caveman” speaking), just more difficult for a human to read (but not impossible).

That’s because, given the amount of data they have for reasoning traces, it would be very hard for them to train a completely different architecture - they would have to produce an equivalent amount of data.

Lastly, I take everything they say with a big dose of skepticism. I still have not forgotten all they hype around o1, and how it was a completely different architecture etc., only for DeepSeek to come out and prove that it was the same model, just specifically trained for CoT.