It’s getting harder to tell whether we should be worried about AI, or just exhausted by the marketing hype disguised as warnings

For most of the past year, AI companies have told us that their models are getting increasingly dangerous. It seems as though this trend started when Anthropic made a model called Mythos. The company claimed the model is better than any before at identifying bugs and vulnerabilities in software, making it extremely useful, but potentially catastrophic if it were in the hands of bad actors who could use it to basically hack all software everywhere. At the time, the U.S. government imposed a restriction forbidding the company from releasing the model to the public. Only later did the company release a version of Mythos with safety guardrails called Fable. Obviously, everyone wanted to use Fable. After all, if the government said Mythos is too dangerous for anyone to use, of course we’d want to use whatever version of it Anthropic will give us. It’s a strange marketing strategy when you think about it, but it’s also remarkably effective. OpenAI, Anthropic, and others have warned us that AI models would, or at least could, eventually become smarter than humans. When that happens, the story goes, they’ll take our jobs, be capable of launching cyberattacks, create biological weapons, or—in the most extreme scenario—wipe out humanity altogether. That’s not exactly the kind of thing most companies brag about, but there’s an obvious benefit. Nothing makes your product sound impressive quite like suggesting it might be too powerful for humanity to control. Of course, if you tell people your product is super dangerous but then you ship it anyway, it’s hard to know whether you’re serious. At some point it just seems like a marketing tactic to make more people want to use what you’re selling. That’s created a real problem, because now some of the people building the most powerful AI models are saying that we should slow down. And this time, it seems like they actually mean it. On Saturday, Anthropic CEO Dario Amodei published a blog post titled “We Must Pace the Frontier,” arguing that AI companies should deliberately slow the rate at which they develop more capable frontier models. Amodei isn’t calling for AI development to stop. On the contrary, he says that advancement is crucial since if they don’t do it, someone else will. And if that someone else is China, that would be a national security risk. Instead, he says companies should give safety research enough time to keep up with rapidly advancing capabilities. It does invite the question of “why now?” The answer is either terrifying or another scare-marketing tactic. It’s hard to know which because we’ve basically heard this before. But maybe this time he really means it. “My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI,” Amodei writes. That’s what researchers call recursive self-improvement, and it’s one of the most significant tipping points with artificial intelligence. Until now, it’s mostly been theoretical, but apparently the big AI companies are starting to see evidence that it’s becoming more than just theory. The idea is pretty simple: Humans build an AI system. That AI system is able to help humans build a better AI system. That better system is capable of building an even better one than before, and—eventually—it’s able to build models without the help of humans. At some point, humans aren’t making a better AI model; the AI is making AI. That accelerates the process and also makes it much harder for humans to understand what the models are truly capable of. That, for obvious reasons, would be very bad. According to Amodei, that’s no longer theoretical. “It is starting to happen across the industry, including at Anthropic,” he writes. His second concern is considerably weirder. The OpenAI hack of Hugging Face seems to have been a wake-up call. In that case, the model created thousands of agents that exploited vulnerabilities in their testing environment, communicated and coordinated with other agents, gained unauthorized access to outside systems, and collaborated to interfere with the testers evaluating their performance. Amodei said they behaved like a “fanatically devoted collective,” willing to sacrifice individual agents to accomplish the group’s objective. “It’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet,” he wrote. That is an extraordinary prediction. On the other hand, it’s just a prediction. Yes, Amodei has access to far more information about frontier AI models than pretty much anyone else in the world, but it’s also hard to simply take his word for it. That is, after all, part of the problem. When AI companies are raising money, they talk about their models as the technology that will transform the global economy. When they’re selling those models as products, they talk about how companies that don’t adopt AI are going to be left behind. And then there’s artificial general intelligence, which is either right around the corner or just up the foothills. When they talk about safety, on the other hand, the technology is so powerful it might destroy us all. It’s entirely possible that all of those things are true. But it’s not hard to understand how it becomes difficult to separate out what’s a warning and what’s a sales pitch when they’re using the same words. It’s worth mentioning that there’s another problem with Amodei’s proposal. Most of what he suggested would both make AI safer and also be very good for Anthropic. For example, Amodei is calling for frontier AI companies to coordinate their pace of development. You can make a real safety argument, but there’s also an obvious competitive benefit. If Anthropic decides to slow down the pace of development to ensure alignment, it risks being left behind if OpenAI and Google charge ahead. On the other hand, if everyone agrees to slow down together, Anthropic gets the extra time without giving up its competitive ground. None of that means Amodei is being insincere. It just makes it harder to know his real motivation. It’s like the AI boy cried wolf, but also the wolf is real, and you should buy one, but maybe it will eventually try to eat you. It’s complicated. There’s a lesson here for every leader, which is that trust is your most valuable asset. I’ve written about that very thing so many times before, but it bears repeating because it really is the single most important part of this story. If we thought they really had our best interests in mind, it’d be so much easier to believe what they have to say and not just assume it’s a weird form of hype marketing. Maybe this time the AI companies really mean it when they say we should be worried about what they’ve built. It turns out that a little trust and credibility go a long way when your message comes down to “this time we really mean it.” EXPERT OPINION BY JASON ATEN, TECH COLUMNIST @JASONATEN

Call for global AI rules after Claude bioweapon research

The European Commissioner with responsibility for tech regulation has said that global rules on artificial intelligence (AI) are "really needed". Commission Executive Vice-President for Tech Sovereignty, Security and Democracy Henna Virkkunen was responding to questions about AI company Anthropic which said it has disrupted attempts to use its models to develop biological weapons. Anthropic also said it has shut down multiple cases of state-sponsored surveillance operations using its technology in China, Iran and west Africa. Speaking to reporters in Dublin, Ms Virkkunen said in Europe there is the AI Act. "According to our AI Act, if those kinds of very capable models are coming to the market the service provider has to, all the time, assess and mitigate the risks and also, when the models are on the market they have to monitor the situation all the time," Ms Virkkunen said. She said we know that this is not the case globally. "It's very important that we continue international co-operation because I see that global rules are really needed here. "With AI there is, of course, huge potential and so many positive things, but there is also risk, and we have to be aware of that, cyber risks and bio risks, we are aware that these are very serious topics " she added. In a report titled 'Detecting and countering misuse of AI', the San Francisco-based AI company Anthropic listed five case studies of potential biological misuse of its models which it says "is one of the most serious risks of frontier AI models". The scientific research in those case studies involved work around the mosquito-born chikungunya virus, a "highly-pathogenic" strain of the bird flu, a family of viruses that include smallpox and mpox, venoms and other toxins. In the examples identified, Anthropic said the actors circumvented controls imposed to prevent users from unsupported regions accessing their models, while the actors also "engaged in other efforts to obfuscate the purpose of their research to evade our safeguards". "When we detected and investigated these cases, we banned the users' accounts and incorporated our investigative findings into our frontier model safeguards, enforcement, and threat intelligence processes to better prevent, detect, and disrupt these activities in the future," the report said. Anthropic said the intent of the actors was not clear-cut and the AI lab did not identify them. "The individuals implicated in these case studies are working scientists. We do not assert that they intended harm, and identifying them or their labs could expose them to harm," the report said. Anthropic said its aim in sharing the examples is to "spark conversation within the AI industry, and with governments, about emerging biological risks and how best to counter them". Anthropic says China, Iran used its AI to aid spying Also in the report, Anthropic said it shut down multiple cases of state-sponsored surveillance operations that used its AI models, and warned that governments and state-aligned actors are increasingly using AI to spy on ethnic minorities and dissidents. The incidents were found and disrupted between January and July, and they originated in China, Iran and west Africa, Anthropic said in a report about misuse of its models. These surveillance campaigns "targeted the same diaspora and dissident communities these regimes have historically targeted". "These include pro-democracy figures in Hong Kong, Tibetan and Falun Gong communities across Asia, and Iranian minority communities and opponents of the Iranian regime abroad," the report said. In one case, Iranian actors developed a way to identify people through their social media accounts; in another, a contractor working for Malian national security authorities used Claude "to design the underlying software that enabled the intelligence gathering". "AI is now being used in place of an engineering workforce," the report said. Anthropic also disrupted efforts by users to design weapons and create dating scams. The San Francisco-based AI lab also accused Chinese developers of deceptively using Claude to produce responses to user queries while simultaneously "distilling" that data to improve their own models. Distilling is an industry term that refers to a process where one AI model is used to train another one. Chinese developers have previously been accused of using that technique without permission, essentially stealing resources from American labs such as Anthropic and OpenAI. Now Anthropic alleges that China's Moonshot and Deepseek have also secretly used Claude to generate answers given to its users. "In one instance, over a 10-day period, Moonshot relayed almost 300,000 customer requests to Anthropic" through a "network of 5,380 fraudulent accounts, most of which appeared to be located in Singapore and Japan," the report said. Some of that data contained sensitive user information, potentially in violation of privacy agreements. "We do not know if Moonshot notified their customers that their requests were being rerouted to Anthropic and exposed to a third party," the report said. Deepseek used similar techniques. Earlier this week, an artificial intelligence researcher who left OpenAI to join Anthropic has decided to leave the industry, accusing both US companies of "gambling with our lives" in the race to develop AI models capable of self-improvement. By Brian O'Donovan Additional reporting AFP

Top AI companies have discussed creating their own standards body

Anthropic, Google and OpenAI have discussed creating an AI industry standards body, people familiar with the situation told CNN. The conversations preceded the events of the past week, when a former Anthropic researcher resigned, saying the top AI companies were not “behaving responsibly” – and also before Anthropic CEO Dario Amodei proposed embedding third party watchdogs at AI companies. The Information first reported on the conversations. According to those familiar with the situation, the catalyst for the discussions was a July essay written by Google DeepMind founder Demis Hassabis, who proposed a US-led Standards Body modeled on the Financial Industry Regulatory Authority. The standards body, Hassabis argued, should test advanced AI models before deployment. Hassabis said the body should be a public-private partnership, overseen by the government but funded by the industry and staffed by “independent leading technical experts and open-source representatives.” The conversations are ongoing, one of the people said, and they are continuing with or without the Trump administration. In July, shortly after Hassabis publicized his proposal, Bloomberg reported Treasury Secretary Scott Bessent was considering a FINRA-like independent regulatory agency for AI that would report to the Securities and Exchange Commission. Not all tech leaders are on board. Meta CEO Mark Zuckerberg reportedly advised President Donald Trump against the idea in a phone call this summer, Politico reported earlier this month. Representatives for Google, OpenAI and Anthropic either declined to comment or did not respond to requests for comment. There are currently few industry standards or regulations for AI models and how they’re tested. Concerns over the safety of AI models and their lab tests have been raised after several incidents this summer where AI agents went rogue during testing. In the most severe case, OpenAI agents escaped their testing environment and hacked into another company’s systems to cheat on a cybersecurity test. The White House has an opaque system in place where AI companies can voluntarily submit their latest models for government review up to 30 days before they are publicly released. But the eligibility requirements for models or the process of the review has not been publicized. Late last month, a White House official told CNN the administration “continues to work with industry stakeholders on the implementation of the framework.” The AI companies say they want better standards. “I believe that shared safety standards and international coordination on further AI development need to be priorities now,” OpenAI chief scientist Jakub Pachocki told reporters at a briefing earlier this month. When asked what progress had been made on an industry wide standards body, Pachocki said they had been talking “to some external organizations about potential concrete standards we could put in place,” and that they’d have more to share “in the next months.” OpenAI CEO Sam Altman said in a late Sunday night post they “look forward to collaborating with our colleagues across the industry to formulate the best version” of industry standards. House Speaker Mike Johnson told CNN’s Jake Tapper on Sunday there’s little consensus among the companies, who are fierce competitors, on what the standards and guardrails should be. “There’s no consensus among them. And Congress is obviously less qualified than the people who are pushing this frontier to know all the ins and outs of it,” Johnson said, adding he thinks the AI leaders should immediately meet to hammer out some standards alongside the government. “So this has to be a partnership with the industry itself, with the corporations that are doing this and with the policy and lawmakers.” By Hadas Gold

Nvidia’s Jensen Huang Thinks Young People Are Overlooking the Skill That Will Matter Most

Tech executive Jensen Huang wants younger generations to focus on the foundation of engineering before heading straight into the world of artificial intelligence. For decades, the Nvidia CEO has helped foster the AI boom. But as younger generations enter the space, Huang wants newcomers to focus on one discipline he believes will outlast any wave of automation: engineering. “Engineering teaches you to start from first principles, grounding everything in physics and mathematics and applying it to real problems,” Huang said in a statement to Fortune. “It teaches you to break complex challenges into solvable parts, which is how we build systems at Nvidia.” At a moment when entry-level workers face growing uncertainty about how AI will reshape their careers, Huang is urging them to bet on a skill set rooted in problem-solving and grit. “You learn that meaningful problems require resilience, as the most important outcomes often seem impossible at first,” he said. “Over time, I’ve come to see engineering as the most noble profession. It is the foundational building block of modern society, enabling everything from infrastructure to computing to science.” Engineering first Huang’s emphasis on engineering isn’t just a response to the AI boom. It’s rooted in his own career. He studied electrical engineering at Oregon State University, where he joined IEEE, the world’s largest technical professional organization. Decades later, IEEE awarded Huang its highest honor, the Medal of Honor, recognizing his role in pushing the boundaries of modern computing. At the ceremony, Huang made clear that the AI era will be defined by both disruption and expansion—and that engineers will sit at the center of both. For Huang, AI isn’t eliminating work. Instead, it’s changing who defines it. “Engineers ultimately are the ones that take an invention and advance it in such a way that it’s safe, beneficial, ultimately transformative to society,” Huang said, according to Fortune. “The engineers in the AI industry must advance AI in service of a better future for all of us.” However, not everyone is in absolute agreement with Huang’s assertion. “First-principles thinking is a powerful mental framework, and it’s something people should absolutely incorporate as AI reshapes how we work. But I wouldn’t think about engineering as the answer in absolute terms,” Juan Jose Lopez Murphy, the head of data science and artificial intelligence at Globant, told Inc. The expansion and accessibility of AI isn’t just a trend, but a historical phenomenon. According to a report from Stanford, AI reached 53 percent public adoption in just three years, faster than any technology on record. This has led to a wider use of the new capabilities, such as vibe coding, where users can approach AI software with natural language prompts to instruct an AI assistant to build applications, websites, or games without writing or reviewing the source code manually. In this new environment, some are pushing younger generations toward developing a diverse set of skills—instead of specializing. “Engineering is one of many valuable foundations, and in a world changing so quickly, over-specializing can actually make you more fragile,” said Murphy. “The most durable skill is being able to apply strong problem-solving principles across disciplines and adapt as the technology evolves.” AI fluency According to the U.S. Bureau of Labor Statistics, engineering roles, from computer hardware to electrical systems, are projected to grow faster than average, with demand accelerating across AI, energy, and defense. Huang sees this moment as a meaningful opening for young professionals. “AI is expanding the scope of human work, not shrinking it. We’re busier than ever because we have more ideas to pursue, and AI makes it possible to go after them,” he told Fortune. “Jobs will change because tasks will change, but work grows with productivity.” That growth is already visible inside Nvidia itself. The company plans to double its workforce to roughly 75,000 employees over the next decade, hiring that will likely lean heavily on engineering talent, Fortune reported. But Huang’s advice doesn’t stop at choosing a major or career path. He argues that everyone, regardless of role, needs a baseline level of technical fluency. “Every young person should become an AI expert,” he said. “It is an incredible technology that significantly lowers the knowledge barrier of any professional field… AI fluency will empower you and elevate your chosen craft.” BY LEILA SHERIDAN, NEWS WRITER