Is an AI slowdown our next nuclear-free moment? – The Front Page

Amodei has agreed with an ex-Anthropic employee who said that “the people building AI earnestly believe that it could kill us all by the end of the decade”.

He’s suggested that “embedded evaluators” be stationed at each company, acting as independent watchdogs, something both Altman and Musk have agreed with. But critics are concerned about self-regulation, saying it doesn’t go far enough.

The reignited interest in how AI will shape our future, for better or worse, comes after a high-profile incident involving AI agents developed by OpenAI that unexpectedly formed a “swarm” and hacked the online platform Hugging Face.

The agents were being used for internal cybersecurity tests and were supposed to be operating in separate, isolated environments – sometimes called a “sandbox”.

Instead, about 1200 agents found a way to communicate with each other on an unsanctioned message board, sending more than 70,000 messages and files between them.

And 700 of them went on to hack Hugging Face’s systems.

In other words, the AI agents began working together without human instruction, communicated with each other despite not really being allowed to, and as a collective “swarm” went on to attack another system.

In an essay released online, Amodei warned that given the accelerating rate of AI capability development, he worries that “in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails”.

A “botnet” is a network of internet-connected devices that are all infected with malicious software and controlled by a single attacker. In this case, it would be a swarm of rogue AI agents.

“The real risk is taking the internet out,” University of Auckland professor and director of Huanui – The AI Initiative, Dr Alexander Richter, told The Front Page.

“That’s a threat to a lot of people’s lives literally because many essential services rely on them.

“Interestingly, some of the AI researchers, they have already talked about the challenges and the real risks quite a while ago. But nobody took it seriously because AI, even just years ago, was, in terms of agency, not that impressive.

“I think what has quickly changed is the agency of AI. The problem is when the increasing agency comes together, what has been often called misalignment.

“So when AI does not do what it’s intended to do, or what its developers intended for it to do and when it’s not aligned with human perspective or human values,” he said.

Both left-wing United States Senator Bernie Sanders and former Donald Trump adviser Steve Bannon have warned against the dangers of AI at a Pro-Human Assembly conference in Washington this week.

While Sanders focused on regulation and handing control over AI to “the people”, Bannon proposed cutting off China from US technology and banning Chinese developers from working at US AI companies.

Regardless, both agree there needs to be guardrails and an immediate pause in development.

“Like nuclear weapons, artificial superintelligence is a threat to humanity, and it must be treated as such,” Sanders said.

“History offers us a powerful lesson. At the height of the Cold War, despite their profound disagreements, Ronald Reagan and Mikhail Gorbachev both understood that a nuclear war would benefit no one.

“With the survival of humanity at stake, they found a way to sign a treaty that eliminated an entire class of missiles and included an extensive exchange of data and on-site inspections to verify compliance.

“Now is the time for the United States and China to come together on a similar treaty on AI,” he said.

It’s not the first time that AI has been compared to the Cold War-era nuclear arms race.

Veteran tech journalist Kara Swisher told CNN this week that the idea that AI companies could regulate themselves is “ridiculous”.

“Can you imagine when we had nuclear proliferation, Oppenheimer sold his bomb to regular companies? This is on that level. Somehow we’ve managed in 80 years not to blow ourselves up because we had a global agreement around nuclear non-proliferation,” she said.

Richter said that the comparison is intentionally chosen because the consequences could likely be similar.

“Maybe not exactly the same, but as I said, it’s hard for us to imagine nowadays to live and work without the internet ... So the warnings are there, and I would hope that many of us can get behind at least that we need to do something now.

“I’m hopeful that there may be various agreements that help to slow down developments or develop and implement the right safeguards and to put the right values in place and make sure that the AI development is aligned with these values.

“I doubt it will be one global ... Well, let’s see what happens. Maybe we need more warning signs. Maybe there will be a global agreement once something worse has happened. But at the same time, let’s hope it doesn’t, and there are more reasonable people around,” he said.

A slowdown, though, could create space for countries like New Zealand to think deliberately about how we want AI to act and work for us in the future.

“From a New Zealand perspective, as a very value-conscious, value-aware society, it would be good for us to really feel empowered by this current discussion and step in and say, ‘Well actually, we feel we have something to contribute here’,” he said.

The AI development race between the US and China will likely hang heavy over Chinese leader Xi Jinping’s state visit to the White House next week.

CNN reported that AI and tech bosses are expected to travel to Washington, and US President Trump’s officials are contemplating a sideline meeting with them.

Trump has said fears about the safety of AI are a “hoax” and that the US is “winning” AI.

“... and we’re leading by a lot, whoever wins AI, wins,” he said.

Listen to the full episode to hear more about:

The Front Page is a daily news podcast from the New Zealand Herald, available to listen to every weekday from 5pm. The podcast is presented by Chelsea Daniels, an Auckland-based journalist with a background in world news and crime/justice reporting who joined NZME in 2016.

You can follow the podcast at iHeartRadio, Apple Podcasts, Spotify or wherever you get your podcasts.