Trust, but Verify: Running an Open-Source AI Pentester in a Regulated Environment
What an autonomous AI agent actually saves in real penetration testing work, and where we still need a person watching it.
Our team does offensive security testing for medical devices and the software around them. Web apps, APIs, source code, and the systems holding the whole infrastructure together. None of us is a doctor, but the work protects patients all the same. The portfolio grows every year, and we want to keep pace with it, which is what got us looking at AI agents in the first place.
For the past few months, we’ve had an open-source autonomous penetration testing agent running alongside us on live engagements. The question we wanted answered was whether it actually saves us time or just moves the work somewhere else.
The answer? It does save time, though, on a narrower slice of work than you would hope. It also needs supervision. We run it against the same target several times before we believe what it tells us, because two runs can come back with a different set of findings.
Pick your poison: speed or judgment
Until recently you picked one of two approaches (or their combination).
Scanners are fast and cheap, and they cover a lot of ground in minutes. They also hand you a queue where a large portion of the findings are false positives, and nothing in that queue knows anything about your business logic.
Manual testing gets you the issues that matter: a broken authorization flow, or three small things that only add up to something in this particular product. However, it costs more and it doesn’t scale.
Medtech makes the gap worse. The attack surface is large and heavily regulated, and our team will never be as big as the product portfolio. Therefore, we need both at once.
What we’re running
We picked StrixAI for the pilot. It’s an open-source agent under the Apache 2.0 license, available on GitHub, and two things about it made it worth more than a one-time test.
It acts; rather than describing what an attacker might do, it runs code, probes endpoints and actively tries to get into it.
And it proves what it finds. Every reported issue comes with a working proof of concept, a short Python script. That on its own did more for our false positive rate than any prompt tuning we tried before. Against the same targets, a general-purpose chatbot produced noticeably more noise, and the Strix agent came in a little under what we expected.
Open source mattered for a practical reason; we forked it internally. That let us add the medtech pieces ourselves, such as DICOM testing along with our own frameworks and checklists, without waiting on new features being added into the tool itself.
The agentic loop
The behavior underneath is simple. StrixAI runs roughly the loop you run in your head when you test something.
- Plan. Work out what the target is and what’s worth trying.
- Act. Pick a tool and run it.
- Analyze. Read the output and figure out what it means.
- Iterate. Adjust, then try the next thing.
Steps three and four are where it stops resembling a scanner. A scanner walks its checklist regardless of what comes back. The agent reads the response, decides what to do with it, and keeps going until it hits the objective or runs out of ideas.
It drives the tools we already use
The agent isn’t inventing new hacking techniques. It drives a toolkit any tester would recognize: an HTTP proxy for intercepting and replaying traffic, a real browser for client-side work, templated scans, a sandboxed shell, and a Python runtime for throwaway exploit code.
Two things follow from that. The tooling is known and trusted, so nothing exotic touches the target. And every action goes through those predefined tools, which means I can go back afterwards and read exactly what has been executed.
A small pentest team, made of agents
The second structural idea is specialization. Several agents run in parallel and share context between them.
- Recon maps the surface and finds the endpoints worth testing.
- Injection works the inputs and whatever else looks weak.
- Validation confirms each finding with a proof of concept.
- Reporting writes it up.
The shared context is what makes this useful. Whatever recon turns up becomes the injection agent’s starting point, and the whole thing covers ground quickly because the individual agents run in parallel.
What a run actually looks like
For the demo below we used OWASP Juice Shop, a deliberately vulnerable web app, in an isolated local container.
The agent gets a brief – a set of instructions – before it starts. Scope, the hosts that are explicitly out of scope, a set of hard safety rules, and the objective for the run. The safety rules need to be stated explicitly: no DROP, no DELETE, no mass data modification. That brief is the most important input in the whole workflow.
On the right is the agent tree. SQLi discovery, XSS discovery and access control / IDOR each ran, with validation and reporting agents spawning underneath them as findings came in. The list below shows what came out of it: unauthenticated access to an endpoint leaking user data, unauthenticated access to the admin configuration endpoint, and an SQL injection authentication bypass on login.
The main pane is one of those findings written up. A short Python script that reproduces the issue, the values it pulled back as evidence, and remediation steps a developer can act on.
Where it fits today
Tool write-ups usually skip this part, and it’s the part that decides whether you get any value out of the tool at all.
Web apps, APIs and source code review work well today, and the results there are solid enough to lean on.
Infrastructure is functional, but the maturity of the tool and its methodology is still not at the level where our team would reach for it.
Access is the hard blocker. Anything behind a jumphost is out of reach, and so is anything in a properly segmented environment. If the agent can’t reach the target, then none of the rest matters, and we test those the way we always have. And in a regulated enterprise a surprising share of the estate sits behind exactly that kind of infrastructure.
What it actually buys us
It absorbs the repetitive part of the job, the recon and surface mapping that has to happen on every engagement before anything interesting starts. That work isn’t really where a tester’s judgment and brainpower pays off.
It also covers more ground than we would by hand in the same window, and it does that in the background while we’re on something else. Because every finding arrives with a PoC, triage now takes a fraction of the time it used to.
The obvious question therefore is whether you can just run the tool and walk away.
Trust, but verify
You can’t, and here is a list of what gets in the way.
The judgment is opaque. I can see every command it ran. I can’t see why it chose that path or what and why it decided to skip, so I can’t tell you the coverage was complete.
It’s non-deterministic. Three runs against the same target can produce up to three different result sets, so we run it several times before we trust the picture. That eats into the time it saved and obviously into the costs.
It’s dangerous when the scope is loose. Left alone it will drop tables and fire delete requests, which is why the brief and scoping matter as much as it does.
And it needs supervision. It might get stuck in loops, and every so often a person has to step in and point it somewhere else.
So we run it in parallel with the way we’ve always worked. We test as normal, the agent grinds through the easier findings in the background, and that frees us up for the exploitation work that genuinely needs a person on it. Everything it produces gets reviewed, manually verified and signed off by a human before it lands in a report.
Where we are now
Strix, and agentic penetration testing overall, is still in a pilot phase in our penetration testing process. It fits inside our existing processes and doesn’t introduce a new risk surface, which is the compliance question that usually kills this sort of thing early. We are collecting data across real engagements to see where it holds up, and looking at other AI tooling at the same time.
If you want to try this yourself
A few things I’d tell anyone starting the same experiment:
- Start with web apps and APIs. That’s where agentic tooling actually works right now, and an infrastructure test that never had a chance will give you the wrong impression.
- Scope it tight, and run it somewhere you can afford to lose. Assume it does the destructive thing at least once.
- Run it more than once. Non-determinism isn’t something you configure away, so repeat runs are part of the method.
- Keep your reporting standards where they are. If the evidence wouldn’t survive your normal review, it doesn’t ship.
Final takeaway
So where does that leave us? The agent is in our workflow, it’s useful on a specific slice of the work, and it’s supervised at every step. For a regulated medtech environment that’s a reasonable place to be right now. Ask me again in a year and the first half of that sentence will probably have moved. The second half probably won’t.
