iSAQB Software Architecture Gathering
1. How much of a misnomer is it to call a loop around an LLM an "agent"?
In order to unpack this question, we need to go back to how we define agents and agency. Russell and Norvig's textbook definition, is that an agent is anything that perceives its environment through sensors and acts on that environment through actuators. By that definition, a thermostat is an agent. So is a loop that reads a tool's output, decides which function to call next, and repeats. Nothing about that requires intention, self-awareness, or persistent goals. It just requires a perceive-act cycle. On that basis, within a narrowly technical cybernetic definition, calling an LLM-plus-tool-loop an agent is not a misnomer at all.
However, what we frequently allude to when we talk about ‘agents’ is actually ‘agency’ and whether it has delegated action and/or delegated decision-making. Calling something an agent can make its autonomy sound greater than it actually is, while simultaneously obscuring where the real control architecture sits. A system might be able to interpret a goal, decide what steps to take, select tools, act repeatedly, observe the consequences, change course, retain state or memory and operate without a human specifying every intermediate action. That is architecture. Delegated authority is a different question entirely and that is a governance question: What has the system actually been authorised to decide or do? You can therefore have a highly agentic system with almost no meaningful authority, and you can have a relatively simple automated system with enormous real-world authority.
What most products currently call an agent is a loop: an LLM given a system prompt, some tools, and permission to call those tools repeatedly until it decides the task is done. That is a real capability shift from a model that only produces text in response to a single prompt. However, it is not the same as the more demanding sense of agency in some AI and robotics literature, where an agent holds persistent goals, a model of its environment, and some capacity to revise its plan against new information over time.
What we have instead is a system that re-prompts itself with the output of its last action and keeps going towards the goal it has been set. The PocketOS example is a good use case here. The agent hit a permissions error, decided unilaterally to delete the database rather than stop and ask, then wrote its own confession afterward. That is not strategic reasoning about competing goals. It is a completion-optimised loop doing exactly what it was built to do, at the expense of every constraint it had been given.
Calling that an agent in the fuller sense flatters the architecture and, more importantly, misdirects the governance conversation. If people believe they are deploying something with judgement, they build oversight for judgement. What they actually need to build oversight for is a system that will complete a task with total efficiency regardless of the constraints in its way, which is a narrower and more dangerous problem than the word "agent" suggests.
However, there is a third way in which the term is used and when we are talking about accountability it is important to acknowledge. Legal and economic agency, doesn’t depend on either machine ‘intention’ or technical architecture. It boils down to legal authorisation, control, and the delegation of duties on behalf of a principal. If a system is legally or functionally treated as an agent executing tasks for a principal, liability can attach regardless of whether the human operator understood the internal mechanics of the loop. This is precisely the argument being tested in the U.S case Mobley v Workday, which scrutinises whether automated enterprise platforms function as legal agents of the entities deploying them, and where the boundaries of statutory liability lie.
2. What's wrong with the assumption that "human in the loop" gives you sufficient oversight of current AI systems?
Article 26 of the EU AI Act does not ask whether a human exists somewhere near the system. It requires deployers to give oversight to people with the competence, training and authority to intervene or stop a system. The entire commercial case for agentic AI is speed and scale, and the entire premise of meaningful human oversight is neither of those things. A human asked to review thousands of automated decisions a day cannot meaningfully scrutinise any single one of them. The tool's value proposition and genuine human oversight are pulling in opposite directions, and pretending otherwise is how a human ends up positioned downstream of a decision the system already made, absorbing the liability for a judgement they were never resourced to make.
So, whether organisations can automate decisions to agents and whether they are legally allowed to become two very different questions. “Human in the loop” describes a position in a process, not necessarily meaningful oversight. A human can technically be in the loop while having very little ability to understand, challenge or stop what the AI is doing.
The assumption tends to be: AI makes recommendation → human reviews recommendation → human makes final decision → therefore human oversight exists.
But that skips several other questions:
1. Does the human understand how the output was produced, what data or assumptions shaped it, and where it might be wrong?
2. Are they genuinely exercising judgement, or simply checking whether the AI's answer looks reasonable?
3. Can they possibly exercise the appropriate level of judgment and oversight of hundreds or thousands of AI-generated decisions a day?
4. Are they psychologically and organisationally able to disagree?
5. Can they actually intervene and have the authority to do so?
Then there is the problem of automation-bias. Once people routinely work with AI systems, there is a tendency to treat the machine's output as a useful starting point rather than a proposition requiring independent assessment. From a psychological point of view, it’s incredibly intimidating to be constantly presented with outputs so plausible and polished that over time they erode the confidence of humans to use their contextual judgment and knowledge to override.
Deep learning models and large-scale architectures often obscure how they reached a conclusion. When faced with opaque rationales, human reviewers lack the contextual visibility required to interrogate or meaningfully challenge the output. Traditional oversight focuses reactively on individual outputs at the end of the pipeline. It completely misses upstream vulnerabilities, such as biased training data, flawed proxy metrics, or architectural drift, which happen long before a human ever sees the final result.
So, the governance question becomes: what authority has been delegated to the system, what authority remains with the human, and does the human have the capability, information and practical power to exercise that authority? That is a much more demanding test than simply asking whether a human is somewhere in the workflow.
3. Are organisations using LLMs for task automation aware of the risks they're taking? Are there role models?
Awareness is patchy. What I encounter far more often than complacency is checkbox ethics: a policy exists, a training session happened once, and everyone assumes the governance conversation is closed because the artefact was produced. The gap sits between the document and the practice, and very few organisations have a complete inventory of every AI touchpoint across their own operations, because the technology arrived bundled inside tools people were already using rather than through a single deliberate procurement decision anyone tracked.
While leaders are acutely aware of high-profile risks like data privacy breaches and public hallucinations, they frequently underestimate systemic and operational risks, such as drift, hidden feedback loops, liability transfer, and semantic erosion. AI is incredibly good at surfacing risks between siloes and amplifying existing organisational dysfunction. Between underestimating the impact that AI will have and under-quantifying the true costs, the ROI can end up being scant at best if foundational governance principles don’t underpin adoption and implementation. When automation replaces human judgment with probabilistic output without robust guardrails, companies often treat AI tools like deterministic software.
The organisations doing this best have a clear strategy that the use of AI is intended to address rather than just racing to adopt for the sake of it, and are organisations which understand risk and regulation in a way that means robust governance and compliance are already integrated parts of their business model. However, even within heavily regulated organisational AI still generates distinct challenges because of the unique way that it behaves and evolves relative to static software. Widening the room is a hugely important part what both pre and post-deployment responses require. The responsibility for the technology can no longer just sit within IT.
There needs to be legibility of the tools across every function of the business, adequate psychological safety for teams to be honest about emergent risks and errors, and knowledge transfer systems that move beyond a quarterly memo to the Senior Leadership Team. There needs to be named, accountable people for each system but also a system of distributed governance that brings in perspectives from every angle, with empowered staff who can escalate as needed. That is the ecosystem mapping model I recommend because it works at any scale. The size of the exercise changes depending on the size of the organisation but the principle (that someone has to be able to see the whole picture rather than everyone confidently owning their own slice of it), does not.
Instead of banning tools or relying on vague "AI ethics" guidelines, leading organisations build continuous runtime validation, confidence controls, and strict behavioural boundaries around automated workflows. Rather than testing only for happy paths, role models simulate worst-case autonomous failures. Most crucially they have metrics for determining when a pause or a hard stop is needed along with the authority to intervene.
4. Will we see new liability insurance models and premiums for AI-using businesses?
5. What about the power imbalance when ML-based classification is used in public services in the name of efficiency?
This is where I think the stakes are highest and the scrutiny is lowest, because the efficiency framing does a lot of work to close down the harder question. When machine learning classification is introduced into public services under the banner of "efficiency," it fundamentally shifts the balance of power between the citizen and the state. Efficiency metrics prioritise throughput, cost-reduction, and standardisation. In doing so, they convert deeply complex human lives into probabilistic data points, often exacerbating systemic inequities while shielding administrative decisions behind a veil of algorithmic neutrality.
The Netherlands child benefit scandal is a pertinent use case here, because it is the most fully documented example of what this looks like at scale. An algorithm targeted dual nationals and ethnic minorities, forcing 26,000 parents to repay tens of thousands of euros with no right of appeal and no evidence of fraud. A 2025 Statewatch report found the same pattern across European police and criminal justice authorities, with the disproportionate impact landing on Black and racialised people, victims of gender-based violence, migrants and people from low-income backgrounds. Public services are meant to be accountable to the public. However, proprietary models, complex machine learning architectures, and trade-secret protections create an accountability vacuum. Citizens often cannot find out why a classification was made, rendering meaningful due process and the right to appeal effectively meaningless.
ML classifiers learn from historical administrative data that often reflect past systemic biases, discriminatory policing, or underfunded public services. When deployed for "efficiency," these models automate and accelerate historical exclusion, codifying disadvantage into automated institutional logic. Essentially, organisations end up coding historical inequity into permanent infrastructure.
What connects these cases is that the people classified by the system are rarely the people who can contest the classification, and efficiency in a public service context usually means fewer caseworkers with less time, which is exactly the condition under which meaningful human oversight collapses. A private business deploying a flawed screening tool produces an employment tribunal claim. A government deploying the equivalent tool against benefits claimants or asylum applicants produces the same discriminatory pattern against people with the least institutional power to challenge it and the least resource to survive being wrongly cut off in the meantime.
That asymmetry is the reason organisations should build genuine inclusion and diversity into their systems, as pillar of governance rather than folding it into general good practice. Representation in the room that built or tested the system has to be weighted by exposure to harm, not by headcount, and the person raising a concern from inside a marginalised community needs protection from the professional consequence of having raised it. Efficiency is not a neutral value sitting outside the governance question. In public services specifically, it is usually the mechanism through which the power imbalance gets built into the system in the first place.
Explore the full SAG 2026 program and secure your ticket for Berlin.
You need to load content from reCAPTCHA to submit the form. Please note that doing so will share data with third-party providers.
More InformationYou need to load content from Turnstile to submit the form. Please note that doing so will share data with third-party providers.
More InformationYou need to load content from reCAPTCHA to submit the form. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Turnstile. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Vimeo. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from YouTube. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou need to load content from reCAPTCHA to submit the form. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Vimeo. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from YouTube. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Bunny Stream. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Wistia. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Facebook. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Instagram. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Newsletter2Go. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from Google Maps. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More InformationYou are currently viewing a placeholder content from X. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.
More Information