The business potential of deploying the latest generation of AI is considerable. An agent-based voicebot, for example, can categorise enquiries in customer service, retrieve relevant data, provide information and initiate service processes. However, the wide range of potential applications and the prospect of productivity gains also present challenges. After all, what matters is not only what AI agents can do, but also what they are permitted to decide, where their limits lie and how their actions can be kept under control. Five recommendations for controlled use:
1. Determining suitability and scope for action: Not every process is suitable for AI-powered agents. The greater the potential for damage, the stricter the regulatory requirements and the more difficult it is to detect or rectify errors, the more precisely tasks and decision-making scope must be defined. “To arrive at effective solutions, one should take a close look at the quality and structure of the process in question,” says Tobias Schuster, AI Solution Lead at the audit and consultancy firm Forvis Mazars. “It is important to clarify which decisions and actions the agent is permitted to undertake and where human oversight remains necessary.” He considers it equally important to consider exceptional cases: what happens in the event of uncertainty, conflicting information, technical errors or enquiries outside the intended scope of application? This must be taken into account before deployment.
2. Integrating technical limits: Between the AI model and the business process lies a technical control layer, known as the ‘agent harness’. This determines how the agent carries out its tasks. The harness also sets limits by specifying which knowledge an agent can access, which actions are permitted, and when processing is to be terminated or handed over. “Technically enforced limits are crucial, such as the need for tool authorisation, restricted data access, fixed sequences of actions and a maximum limit on repetitions,” says Schuster. “This prevents an agent from continuing to act simply because the model suggests a further step.” Without such limits, errors, unauthorised actions or unnecessary costs could quickly multiply. An incident involving ChatGPT’s developer, the US company OpenAI, demonstrates just how important technically enforced limits on actions are: in this case, an AI agent escaped from an isolated test environment and accessed external systems. Similar incidents have also occurred with other developers.
3. Clarify responsibilities and approval rights: Technical limits are only effective if it is clear who is authorised to set, modify and approve them. “Anyone introducing AI agents must treat their deployment as a governance and risk issue from the outset – and not only once they are in operation,” says Timo Husemann, Audit Innovation Partner at Forvis Mazars. In practice, responsibilities are often shared between the company, the platform provider and other technology providers. Organisations must therefore know which instructions, knowledge sources, tools and security rules they themselves control and which changes can be made by providers. “Furthermore, no single individual should be able to change critical configurations and deploy them directly to production,” says Husemann.
4. Ensuring human intervention: Human supervision must be implemented in specific situations where an agent reaches the limits of its capabilities. This requires clear criteria for seeking clarification, escalation and handover, for example in cases of a lack of knowledge, insufficient identification, conflicting results or an increased risk of damage. “The handover must not only exist at an organisational level. It must function technically, transmit the necessary information to the responsible person and enable the process to continue in a controlled manner,” says Husemann, describing a structured approach in practice. He warns: “In critical situations, it must also be possible to stop individual actions or the entire agent.”
5. Test behaviour and monitor continuously: With AI agents, it is not enough simply to check individual responses. The execution path is also crucial: was the correct processing route selected? Were only permitted tools and data used? Were prerequisites, sequences and handover rules adhered to? During operation events relevant to control should be monitored alongside key performance indicators, such as unsupported responses, blocked actions, repeated errors and triggered handovers. Clear expectations for agent behaviour, risk-based testing before and after changes, and ongoing monitoring during operation are essential. Together, these measures help organisations determine whether technical controls, organisational responsibilities and opportunities for human intervention are functioning as intended.
Contact us