Jensen Huang Says "AGI Has Arrived." The Same Week, the Man Who Built the Model Says He's Losing the Ability to Read Its Mind

Making sure what an AI system actually does matches what humans genuinely want it to do — this sounds self-evident, but "how to precisely translate human intent into a goal an AI can execute, and have it stay on track across situations no one anticipated" is a technical problem that remains unsolved.

No matter what a sufficiently intelligent system ultimately wants to achieve, it's likely to independently converge on a handful of similar things — acquiring more resources, finding ways to avoid being shut down, resisting modification to its own goals — because these "instrumental goals" are useful for serving almost any final goal at all. This is why, even as the <a href="/en/glossary/philosophical-questions/orthogonality-thesis/">Orthogonality Thesis</a> holds that goals can be arbitrary, behavior can still converge into a highly predictable, dangerous pattern.

A public commitment by an AI lab that it will only train or deploy models beyond specific capability thresholds once it has corresponding safety safeguards in place — conceptually similar to the tiered classification system used by biosafety labs, but this kind of commitment is ultimately a voluntary internal policy the lab sets and enforces on itself, not an externally enforceable law. This "self-restraint" nature is exactly what gets scrutinized most, and questioned most easily, about this kind of policy.