On Monday, people working in artificial intelligence warned New York City's leaders about the risks of this new technology, calling for stronger oversight as local governments try to respond to growing fears that these systems could threaten humanity.

At a New York City Council hearing, former employees of Anthropic, OpenAI, and Google DeepMind recently described a reckless culture at these firms, stating that the biggest tech companies are racing to create technology they do not know how to control.

"The companies operate on a startup mindset: move fast, break things, fix them later.

That works for a photo-sharing app.

It does not work for building the most powerful technology ever built," said Jacob Coxon, a former engineer at Anthropic who has become a leading voice for regulation.

State and municipal authorities have been exploring avenues for regulating AI, given that the Trump administration has adopted a hands-off stance and Congress has demonstrated a lack of initiative.

President Donald Trump has brushed aside apprehensions regarding AI, branding them a “hoax”, and late last month signed an agreement outlining a series of voluntary measures for companies to undertake, thereby largely permitting them to self-regulate.

Coxon was a relatively unknown engineer at Anthropic until he announced his departure through a series of social media posts that drew global attention.

He told the Council that AI could bring enormous benefits, but without transparency and safeguards, it is more likely that events will spiral out of control.

"On the path we're on now, I think it's more likely than not that humanity will lose control to these AIs, and it could end in human extinction," Coxon said.

Days after Coxon resigned and voiced his fears about worst-case scenarios, leading executives—including Anthropic's Dario Amodei, OpenAI's Sam Altman, and SpaceXAI's Elon Musk—said they agreed that AI's development needs to slow down.

AI companies have detailed the steps they have taken to prioritize safety, but a coordinated industry slowdown faces several obstacles.

A panel of AI company staffers — representing tech giants OpenAI, Anthropic, Google and Meta — also testified at the Manhattan hearing on Monday, telling officials that their systems are already enhancing people's daily lives and emphasizing that they are laser-focused on safety.

Anthropic and OpenAI have said their advanced AI products need to be regulated, and company representatives at the hearing expressed openness to some of the Council's suggestions.

"I share your commitment to ensuring that technology is developed safely and that its benefits are widely shared," said Morgan Dwyer of OpenAI.

At one point during the hearing, Julie Menin, the City Council speaker, asked each of the company representatives to “quantify the risk posed by AI in the worst case catastrophic scenario.”

Dwyer responded: “I don’t know — nor do I think it matters whether there’s a 1% or 10% or 20% chance that something catastrophic will go wrong.

None of these levels is remotely acceptable.

We should not train models that we cannot make an extremely strong case that we can keep under human control.”

Menin condemned the response as “flippant at best.”

“I would argue that the percentage chance is irrelevant.

What matters is the commitment to safety, and that is exactly what OpenAI is committed to doing,” Dwyer said.

The City Council is considering a set of AI regulations, including proposals to require companies to disclose information about AI tools, to require third-party approval for AI systems in the city, and to allow whistleblowers to receive a share of fines from AI companies that break the law.

Separately, New York Gov. Kathy Hochul, a Democrat, has taken steps to impose reporting and transparency standards around AI, as have governors elsewhere, although their power is somewhat limited, given that AI technology can reach across state lines and around the globe.

Logan Graham, representing Anthropic, informed the Council that the company embraces “smart regulation” and that state and local governments have a part to play in the formulation of frameworks.

“AI is powerful enough that getting this wrong has real consequences, for individuals, institutions, public trust,” he said.

“We have to work relentlessly to do the science of mitigating the risks.”