AI agents need more than AI Act checklists

X is currently full of two conversations happening at the same time: AI agents are becoming more real inside companies, and the EU AI Act is getting closer as an operational pressure point. Between those two conversations sits a topic that matters more to German product teams than the next tool comparison.

AI agents need architecture.

Not architecture as a diagram that lives in a slide deck. I mean the practical question of how a system behaves when AI does more than generate text: when it reads data, calls tools, prepares decisions, changes code, triggers workflows or influences internal operations.

For founders, CTOs, product owners and Mittelstand teams, this is a good moment to slow down slightly. Not to avoid AI. The upside is too large for that. But to put it into the business in a way that does not later become a risk, a compliance burden or an unmaintainable black box.

The EU AI Act is not the real reason to do this properly. It simply makes visible what good software delivery already needs: inventory, boundaries, traceability, review paths, responsibility and operating models.

In shorter form: if you want to use AI agents seriously, do not start with a compliance checklist. Start with the system architecture.

Why the current X trend matters

The discussion over the last few days has a clear pattern. Companies are no longer only experimenting with chatbots at the edge. They are looking at agents that do work: research, support, software development, process automation, compliance workflows, internal tools, data preparation and reporting.

At the same time, X is full of posts about auditability, logging, AI Act readiness, traceability, runtime control planes and agentic security. Some posts are exaggerated, and some blur the legal timeline. But the underlying problem is real.

An AI agent is operationally different from a conventional feature.

A conventional feature, if designed well, does exactly what the code says. An agent works with probabilities, context, tools, instructions and often several intermediate steps. It can take different paths even when the user asks the same kind of task. It can interact with data whose sensitivity only becomes clear at runtime. And it can produce outputs that humans use as decision support.

That does not make agents uncontrollable. It means control has to be designed explicitly.

This is where the AI Act, software architecture and product responsibility meet.

The AI Act is not a PDF project

Many teams will be tempted to treat the EU AI Act as a legal documentation exercise. At some point, someone writes a policy. A vendor questionnaire gets filled out. A risk assessment lands in a folder. Done.

For many real AI systems, that will not be enough.

The important questions are not only legal. They are technical and operational:

• Where is AI already used inside the product or company?
• Which systems influence users, customers, employees or business decisions?
• Which data can an agent see?
• Which tools can it call?
• Who reviews critical outputs?
• Which model, prompt, tool and data version was involved?
• How is an error detected, explained and rolled back?
• Who is the provider, deployer, user or responsible owner in the actual process?

If the team has to reconstruct those answers after the fact, that is an architecture problem. Not only a compliance problem.

For many German companies, this will be the uncomfortable part. AI usage has often grown faster than the system map. Individual teams use copilots, chatbots, automations, internal scripts, SaaS features, OpenAI- or Azure-based helpers, maybe even local models. Each tool feels small on its own. Together they can become an operational AI system without a clear inventory.

The first step is therefore simple but powerful: write down where AI is actually involved.

An AI inventory is architecture work

A useful AI inventory is not a spreadsheet for filing away. It is a working document for product, engineering, operations and compliance.

It should contain more than tool names. The useful questions are:

• What task does the AI perform?
• Is it only assisting, or does it influence a decision?
• Which user group is affected?
• Which data flows in and out?
• Which external providers are involved?
• Which actions can the system trigger?
• Is there human approval?
• Are there logs, tests, monitoring and rollback?
• Who owns the system functionally and technically?

These questions may sound dry. In practice, they save time because they make later discussions concrete. An AI assistant that suggests marketing copy needs different controls from an agent that screens applicants, prioritises support cases, prepares invoices or changes production code.

Not every AI system is high-risk. Not every system needs the same level of effort. But without an inventory, a team cannot make that distinction responsibly.

At McDougall Digital, this is exactly where architecture and product judgement need to meet. The question is not “How do we make everything as bureaucratic as possible?” The question is: which decisions are important enough that the system needs better boundaries, evidence and review paths?

Agents need boundaries, not just better prompts

Many AI projects start with prompt engineering. That is understandable, but too narrow. A good prompt helps. It is not access control, an audit log, a rollback plan or a security model.

Once an agent can use tools, the boundary matters more than the wording.

A sensible agent should not simply run with a human user’s permissions. It needs its own identity, a limited scope and clear rules for what it can read, write, execute and trigger. That applies to internal business tools as much as to coding agents in software teams.

In practice, this means:

• separate agent identities instead of shared user accounts
• least privilege instead of broad access
• short-lived tokens instead of permanent secrets
• approval for risky actions
• separate environments for test, staging and production
• hard blocks around especially sensitive data or system areas
• traceable logs for important actions

This sounds like security, but it is also product quality. An agent that can do too much makes the system harder to understand. An agent with clear boundaries can be tested, operated and improved.

This matters especially for Mittelstand teams. Many companies want pragmatic AI solutions, not enormous platform programmes. That is sensible. But pragmatism does not mean handing an agent a master key. Pragmatism means building the few boundaries that actually matter.

Logging is not an optional extra

If an AI agent works inside a relevant process, the team must later be able to understand what happened. Not because AI should be distrusted by default, but because this is normal operational responsibility.

Good logs answer questions such as:

• Who or what started the agent?
• What task was it given?
• Which data sources were used?
• Which tools were called?
• Which model or prompt version was active?
• What output was produced?
• Which human review happened?
• Which change was accepted?
• Which follow-up action was triggered?

This information is not only useful for compliance. It helps with debugging, support cases, security analysis, product improvement and the question of whether an AI workflow is reliable enough in the first place.

Many teams add logging only after something goes wrong. With AI, that is especially expensive because errors without traces are hard to explain. If an agent gives the wrong recommendation, edits the wrong file or prioritises a process incorrectly, “the model did it” is not an acceptable answer.

The better approach is to build evidence into the workflow. Not excessively, and not at the same level everywhere, but wherever decisions, permissions or customer impact are involved.

Human-in-the-loop must be specific

“A human checks it” sounds reassuring. In many systems, it is too vague.

Who checks what? Before or after the action? Against which criteria? With what information? Is the review logged? Can the person actually reject the decision, or only approve it after the fact? What happens when they are uncertain?

A real review path is a product detail, not a slogan.

For an internal reporting agent, it may be enough for a team member to read the summary before it is sent. For an agent that flags contract risk, subject-matter review may be needed. For a coding agent that creates database migrations, the path may require tests, code review, staging and clear deployment rules. For an HR or credit process, additional legal requirements may apply.

The point is simple: human-in-the-loop has to match the task.

A good architecture process therefore distinguishes between low, medium and high impact. Not every AI step needs the same hurdle. But critical steps need more than trust.

What teams should do now

For most German product teams, the useful next step is not a giant AI Act programme. It is an honest technical readiness check.

Five questions are enough to start:

1. Where do we use AI today in the product, development, operations or internal processes?
2. Which of those systems see sensitive data or influence meaningful decisions?
3. Do those systems have clear owners, boundaries and permissions?
4. Can we later explain which AI output led to which action?
5. Which workflows would we need to change if a customer, auditor or partner asked for evidence tomorrow?

If the answers are fuzzy, that is not a reason to panic. It is a useful signal about where architecture work will have the most leverage.

McDougall Digital can help with the practical version of this work: mapping current AI usage, sorting risk by product impact, designing agent workflows, defining permissions and review paths, adding logging and monitoring, and evolving existing software so AI support does not work against maintainability or operational reliability.

The valuable output is not a thick folder of abstract rules. The valuable output is a system that works in everyday use.

The real opportunity

The AI Act is often framed as a brake. For strong teams, it can also be a filter.

It forces questions that should be asked anyway: Which AI features create real value? Which are just experiments? Which decisions may be automated? Which need human responsibility? Which data and permissions are truly necessary? How do we explain the system when it matters?

Teams that answer those questions early are not just building software that looks compliant. They are building better software.

AI agents will move deeper into products, internal tools and delivery processes over the next few years. The winners will not be the teams that automate the most. They will be the teams that design automation so people can trust it, operate it and improve it.

That is not a legal side issue. It is architecture.