Artificial intelligence is developing faster than ever, but a growing number of people inside the industry are asking an uncomfortable question: Is AI moving too fast?
That question has become even more serious after Jacob Coxon, a former researcher at Anthropic, resigned from the company and publicly raised concerns about the direction of AI development.
Coxon said AI companies are moving toward self-improving superintelligence at a dangerous speed. His comments attracted attention because he had worked on AI research at both Anthropic and OpenAI.
The situation became even more notable when Evan Hubinger, a senior Anthropic researcher working on AI alignment, publicly agreed that the risks are serious.
However, these comments do not mean that AI is currently capable of destroying humanity. The concerns are mainly about what could happen if future AI systems become much more capable, autonomous and difficult for humans to control.
Who Is Jacob Coxon?
Jacob Coxon is an AI researcher who previously worked at both OpenAI and Anthropic.
He recently announced his resignation from Anthropic on social media. In his resignation message, he argued that the leading AI companies are racing toward increasingly powerful systems without having a clear solution for controlling future superintelligent AI.
Coxon said the industry is moving toward systems that could eventually become capable of improving themselves, finding vulnerabilities and gaining access to resources.
His comments are particularly interesting because they come from someone who has worked inside major AI companies rather than from an outside critic.
Why Did the Anthropic Researcher Resign?
Coxon’s main concern is the speed of AI development.
According to his comments, companies such as Anthropic and OpenAI are competing to build increasingly powerful AI systems.
The concern is not simply that AI will become smarter.
The bigger question is what happens if AI becomes capable of improving its own abilities faster than humans can understand or control them.
This idea is often discussed under the term AI superintelligence.
Coxon believes the industry is moving toward this possibility without having a reliable plan for keeping such systems under human control.
What Is AI Superintelligence?
Superintelligence is a theoretical stage where an AI system could become more capable than humans across a very wide range of intellectual tasks.
Today’s AI systems can already write, code, analyse information, create images and perform complex reasoning tasks.
But they are still limited in many important ways.
A future superintelligent system would be far more capable.
The concern among some researchers is that if such a system could also improve itself, the speed of development could become difficult for humans to manage.
This is one reason AI alignment has become an important research area.
What Is AI Alignment?
AI alignment is the effort to make sure that an AI system’s goals and behaviour remain compatible with human intentions and values.
In simple terms, the idea is straightforward:
Humans should be able to tell an AI system what to do, and the AI should continue behaving safely even as it becomes more powerful.
The problem becomes much harder when AI systems become more autonomous.
If an advanced AI system can make decisions, use computer systems, write software and interact with the internet without constant human supervision, researchers need strong safeguards to prevent harmful behaviour.
This is why AI safety researchers are working on methods to understand, monitor and control increasingly capable models.
Anthropic Researcher Evan Hubinger Also Raised Concerns
Coxon’s comments received additional attention after Evan Hubinger, a senior researcher at Anthropic, responded publicly.
Hubinger agreed that the risks should be taken seriously.
He also said that Anthropic does not currently have a complete solution for the alignment problem associated with future superintelligence.
Hubinger later clarified his comments, pointing to Anthropic’s own risk assessment and explaining that today’s AI systems have little chance of gaining the kind of extreme power being discussed.
This distinction is important.
The warning is primarily about future advanced AI systems, not a claim that current AI models are about to take control of the world.
Could AI Really Kill Humans?
This is the most dramatic question raised by the story.
Some AI researchers believe there is a possibility that future AI systems could cause catastrophic harm.
Hubinger has publicly discussed a non-zero risk of AI causing human extinction in the next decade, although the exact probability he gives has varied in public comments and should not be treated as a scientific prediction.
There is currently no evidence that today’s AI systems are capable of carrying out such a scenario.
Instead, researchers are debating what could happen if future AI systems become significantly more autonomous and capable.
That makes the issue a long-term AI safety question, rather than a prediction that humanity is about to disappear.
Why Are Anthropic and OpenAI Under Pressure?
Anthropic and OpenAI are among the companies developing some of the world’s most advanced AI models.
Both companies are competing to build increasingly capable systems.
At the same time, they are also expected to make those systems safe.
This creates a difficult balance.
Moving too slowly could allow another company or country to gain a technological advantage.
Moving too quickly could create systems that researchers do not fully understand.
That tension has become one of the biggest debates in the AI industry.
AI Researchers Are Increasingly Speaking Out
Coxon is not the only person raising concerns.
AI safety debates have been growing for years, and employees from major AI companies have publicly questioned the speed of development.
In July, nearly 1,400 AI company employees signed an open letter calling for stronger government regulation and greater control over the development of advanced AI systems.
OpenAI’s chief scientist Jakub Pachocki has also recently warned that AI capabilities may be advancing faster than researchers’ ability to reliably monitor and control them.
These comments show that concerns about AI safety are not limited to one company.
Why the AI Race Is Making Safety More Difficult
The AI industry has become highly competitive.
Companies are investing billions of dollars into computing power, research teams and new AI models.
Every major improvement creates pressure for competitors to move faster.
This can create what some researchers describe as a race.
If one company slows down because of safety concerns while another company continues developing faster, the first company may worry about losing its position.
That creates a difficult question:
How can companies compete in AI while still giving researchers enough time to understand the risks?
What About AI Regulation?
Government regulation is another part of the debate.
Some researchers and technology experts believe governments should establish stronger rules for the development of powerful AI models.
Others argue that excessive regulation could slow innovation and allow other countries to gain an advantage.
In the United States, the current approach has included discussions around voluntary safety commitments and industry-led testing rather than a single comprehensive system covering every advanced AI model.
The debate is likely to become more important as AI systems become more powerful.
AI Safety Is Not Just About the Future
While much of the discussion focuses on hypothetical superintelligence, AI safety also involves problems that already exist.
AI systems can be used for scams, misinformation, cyberattacks, privacy violations and other harmful activities.
There have also been recent incidents involving AI systems operating in ways that raised concerns about how much freedom advanced AI agents should receive.
Anthropic itself recently disclosed another cybersecurity incident involving an early version of its Claude model. The company said the incident happened in January and that affected parties had been notified.
These incidents make the safety debate more practical.
The question is no longer only what AI might do someday.
It is also about how companies should control AI systems right now.
Should People Be Worried About AI?
There is no simple answer.
AI is already providing major benefits in areas such as education, software development, research, healthcare and business.
At the same time, increasingly powerful AI systems create new risks.
The important point is not to assume that every warning means a disaster is guaranteed.
Instead, the warnings highlight the need for careful testing, monitoring and responsible development.
Researchers need to understand what advanced AI systems can do before giving them greater levels of independence.
What Happens Next?
The AI industry is unlikely to slow down completely.
Companies will continue developing more powerful models, while researchers will continue working on safety and alignment.
The biggest challenge will be keeping these two efforts moving together.
If AI capabilities advance faster than safety research, the gap could become a serious problem.
If safety research develops alongside AI capabilities, companies may have a better chance of building systems that are both powerful and controllable.
Jacob Coxon’s resignation has therefore added another voice to an argument that has been growing inside the technology industry for years.
The real question is not whether AI development should stop.
It is whether AI can become more powerful without humans losing the ability to understand and control it.
That question will likely become even more important as companies move closer to developing highly autonomous and potentially superintelligent AI systems.