Until Conference Starts: ✓ 10% Team Discount ✓ Additional Discounts Available

Blog

iSAQB Software Architecture Gathering

Bert_Jan_Schrijver
Bert Jan Schrijver
CTO at OpenValue
Published on September 25, 2026

Who’s Responsible When AI Writes the Code? Bert Jan Schrijver on Trust, Accountability, and Software Quality

Ahead of his SAG 2026 keynote “Software Engineering in the Age of AI Agents: Who’s in Control?”, Richard Wallintin (WPS – Workplace Solutions) spoke with Bert Jan Schrijver about responsibility and trust in AI-assisted software development, the growing importance of independent verification, and what it will take to turn faster software delivery into genuinely better software.

1. Bold (not entirely serious) take: Many software developers have always assumed only limited responsibility for their work. Why should they now that Claude writes the code?

I’ve always seen responsibility for a piece of software as a team effort. It’s not just the person who wrote the code who is solely responsible, but the people who reviewed, tested, designed, architected it should take responsibility as well. When AI agents take over part of the software delivery life cycle, the responsibility doesn’t magically shift from the developer to the agent - because the agent can’t be held accountable. The developer stays responsible - even when agents write the majority of the code. When you use AI to produce more software faster, the human responsibility only increases. AI coding agents are powerful tools, but they don’t replace humans entirely - especially in high stakes situations and regulated environments.

2. What is your estimate on how many people are aware of the fundamental mechanisms of how LLMs work and their probabilistic nature?

My honest, entirely unscientific estimate: under ten percent of working engineers - and far less in the wider public. I wouldn’t necessarily say that that’s a problem, though. For example, look at software development with open source libraries. How many Java developers are aware of the fundamental mechanisms of how the Spring framework works internally? Not that many.

You don’t need to understand the internals of a tool you’re using, as long as you are aware of its general characteristics, its limits, and the risks involved. I do feel that, as an industry, we have some work to do here, explaining limits and risks of using AI tools to the people using them.

And if you do want to learn about the anatomy of an LLM - this blog post by my colleague Roy van Rijn is an excellent start: https://openvalue.blog/posts/2026/06/03/meet-the-experts-roy-van-rijn/

3. While it used to be en vogue to emphasise the human in the loop (last year), there is a current trend to (try to) remove the review bottleneck in what is called agentic engineering. Do we finally get all software teams to automate their tests? Or have AI write automated tests? (And maybe execute them?) Does that solve the bottleneck?

Whenever you’re automating one part of a process, bottlenecks shift and something else becomes the new bottleneck. With AI agents, quickly producing code for a well-specified task (clear intent, goal, and verification steps) is easier than ever. But the goal of software development is not to produce code. The goal of software development is to provide solutions to business problems.

So when we’re using AI agents to write code or generate tests, the point is that we verify that the behavior of the system conforms to the behavior we intended. When you ask an AI to generate tests for the code it has just generated, it will generate tests that verify what the AI thought you meant - not what you actually meant. It’s basically a self-congratulation machine: both sides agree, both sides are wrong.

Look at what actually unlocks autonomy. One of the most impressive agentic projects I've seen this year was a complete rewrite of a major open-source tool, done in eleven days by AI agents. The enabler wasn't a better model - it was the >50k tests in the test suite already in place, acting as an independent source of truth the agents could run against. The more independent verification you have, the more autonomy agents can safely earn.

The real bottleneck was never test execution. It's knowing what "right" looks like and expressing it in a way that can be mechanically verified. AI doesn't dissolve that bottleneck; it drags it to the center of the stage.

4. When we talk about responsibility, we also touch accountability and maybe liability (as in legal). Do you expect more conversations about compensation and insurance in the software industry? Have software vendors and consultancies had good processes/mechanisms in place in the past, will they need to step up their game?

Did we have good mechanisms in place in the past? Partly. Our responsibility approach (reviews, CI/CD gates, testing, incident processes) was real, but built for human speed. Agents break the speed assumption without breaking the responsibility assumption.

So yes, vendors and consultancies need to step up, in two directions at once: first, more rigorous verification and guardrails so failures stay rare. And second, being honest about AI-assisted work: who reviews, what gets verified, who owns the outcome. It's the same principle I’ve used in the past: if you bring it to a review, it's your code. And who knows, maybe we’ll see liability insurance for AI-assisted delivery become a real market in the future.

5. Have you ever too blindly trusted an LLM to do the right thing? What was the impact? Where did your trust come from? Is trust a good concept in that space?

I think that anyone who seriously works with these tools has experienced this to some extent: the moment you lean on output not because you verified it, but because it was fluent and confident and you were short on time. You trusted it to be right.

The interesting part is where the (misplaced) trust came from. We trust fluency. Generated text feels researched and certain and right, and an LLM's confidence is a design property, not a signal of correctness.

So is trust a good concept? Trust is unavoidable. You can't get anything done without some of it. But it has to be calibrated, and above all verifiable. Replace trust with evidence: review the output, run the tests, challenge the assumptions. In the age of probabilistic systems, verification is the only form of trust that still holds.

6. Do you see a chance that the quality of software products in total advances through generative AI?

Maybe. But only if we stop the tools from making us lazier rather than better. I think we have a long way to go there. So far, AI-generated code tends to bring more static-analysis warnings, higher complexity, and more technical debt. We’re producing more code faster than ever. But are we producing better software faster, or just faster?

The effect is ours to choose. A team that keeps its review bar exactly where it was gets better specs and better tests - and the product genuinely improves. A team that replaces review with AI output gets more code and less quality. Quality is not automatic. But we do have access to very powerful tools now that, when used appropriately, can have a positive impact on software quality. How you use these tools is up to you.

Many thanks to Bert Jan for the great interview!

Explore the full SAG 2026 program and secure your ticket for Berlin.

More from the speaker

Software Engineering in the Age of AI Agents: Who’s in Control? (Keynote by Bert Jan Schrijver)

Return to Overview