What the OpenAI incident tells us about AI governance

The existential AI debate matters but it should not distract us from the very real challenges already in front of us

Over the past few weeks, the debate around AI safety has become increasingly loud.

Geoffrey Hinton, the Nobel laureate often described as the “Godfather of AI”, agreed on Newsnight that a 10% chance of AI causing human extinction within a decade “seems not unreasonable”.

The clip travelled rapidly around the world and it is not difficult to understand why.

The possibility of losing control of a technology more capable than its creators is frightening. Recent warnings have pushed that possibility from science fiction into mainstream political and regulatory debate.

But the strength of the reaction also exposes an imbalance in how AI risk is discussed.

Too often, the debate is more comfortable with uncertain future catastrophe than with the control failures, governance choices and real-world impacts already in front of us.

The problem is not rogue AI, it is weak governance

An AI system does not need to be superintelligent or malicious to cause serious harm.

Give it an objective, sufficient capability, inappropriate access and weak constraints and harm can follow.

Recent agentic AI incidents are not stories about “rogue AI”. They are stories about systems given objectives, tools and access within environments where the technical permissions, oversight mechanisms and behavioural constraints did not fully reflect what humans intended them to be allowed to do.

The reported breach of Australia's Medicare Statistics Portal by an OpenAI agent makes the point starkly. In an incident involving unauthorised access to an Australian government healthcare data portal, the agent reportedly bypassed existing controls and accessed non-public files.

 The issue is not that the system developed independent intent. It is that its practical authority exceeded the authority humans intended it to have.

That distinction matters.

None of this means existential risk should be dismissed. Catastrophic risk deserves serious scientific, regulatory and public-policy attention.

But effective AI governance cannot focus only on the risks that produce the most compelling headlines. It also has to address the higher-probability risks and real-world impacts arising from systems already in use today:

• unfair outcomes and operational failures
• cyber exposure, data leakage and misinformation
• unclear accountability and erosion of meaningful human oversight

Model safety is only part of the answer

This is where this week’s Alan Turing Institute report, Frontier AI Risks: A practical way forward, provides a useful frame for the debate.

It recognises the lack of expert consensus around long-term existential risk but argues that this uncertainty should not delay action on risks whose behaviours, failure modes and impacts can already be observed, studied and addressed.

Crucially, it also argues that evaluating a model in isolation is insufficient.

That is the right shift.

Recent statements from leading AI companies, including OpenAI and Anthropic, show growing support for independent testing, external review and third-party assessment of frontier models.

That is welcome.

But testing a model, even independently, is not the same as governing an AI-enabled system in production.

The real question is whether the system as deployed, connected to data, tools, workflows, users, third parties and business processes, remains within the boundaries the organisation intended.

A kill switch is not a governance strategy

This also reframes the current discussion about AI “kill switches”.

A kill switch may be a useful control. But it is not, by itself, a governance strategy.

For a shutdown mechanism to be meaningful, an organisation first needs to know that something has gone wrong. It needs monitoring capable of detecting abnormal or harmful behaviour. It needs people who understand the implications. It needs clear authority to intervene. It needs audit trails to reconstruct what happened. It needs escalation routes, accountability and tested procedures for pausing, correcting or disabling the system.

The button only matters if someone knows when to press it, has the authority to press it, understands the consequences of doing so and can explain why intervention was needed.

Without that surrounding governance, a kill switch risks becoming a comforting concept rather than an effective safeguard.

From safety principles to control in practice

This is where the debate needs to move from safety concepts to operating disciplines.

It is also why we published our agentic AI governance paper, Agentic AI: from principles to practice - C-suite guide.

Our 12-step framework is designed to translate Responsible AI intent into practical operating disciplines: defining agency and authority, controlling access, clarifying human oversight, evidencing assurance, and ensuring accountability for outcomes.

At its heart is a simple governance question:

As AI systems become more autonomous, how do organisations demonstrate that they remain in control?

That is the question boards, executives and risk committees are increasingly asking and the answer requires more than policy statements.

It requires visible, testable and enforceable controls in production. It requires organisations to understand:

• what capabilities agents have and what authority they have been given
• what tools, systems and data can agents can access and how are actions monitored
• where is human judgement required and who can intervene if behaviour moves outside intended boundaries

Govern today for the capabilities of tomorrow

The existential debate should sharpen our focus on the governance foundations needed now.

Because the same weaknesses that make today’s AI systems difficult to control (unclear authority, excessive or inappropriate access, weak monitoring, poor accountability and inadequate assurance) are precisely the weaknesses that will become more dangerous as systems become more capable.

The question, then, is not simply whether AI could one day escape human control.

It is whether organisations can demonstrate meaningful human control over the AI systems and agents they are already deploying today.