CloudThinker Cyber vs AWS Security Agent on OWASP Juice Shop: 65% More Findings at 94% Lower Cost

CloudThinker builds Frontier Investigation Agents. They find the why behind every production issue and propose the fix. Your engineers approve. CloudThinker Cyber is our Frontier Security Agent, and its job fits in one line: it hacks you before hackers do.

Cyber tests your apps and APIs the way a hacker would, so your team catches critical issues first. Every finding is proven. Every fix arrives ready to merge.

If you're new to this kind of testing: a penetration test (pentest) is an authorized, simulated attack on your own app, run to find the holes before a real attacker does. Traditionally a security firm does it by hand once or twice a year, and the report arrives weeks later. Cyber runs the same kind of test on demand. You give it the address of a web app or API you own. It maps what's exposed, tries the attacks a hacker would try, and keeps only the findings it can prove with a working exploit. Production is excluded unless you add it.

AWS launched its own AI pentester, AWS Security Agent, so we ran the two side by side on the same app to see how they compare on results, speed and cost.

Both agents ran against the same target this week, one run each, with default settings. CloudThinker Cyber finished in 27 minutes, reported 38 confirmed vulnerabilities and used 571.75 credits, about $57. AWS Security Agent finished in 5 hours 9 minutes, reported 23 findings and used 19 hours 57 minutes of task time, about $998 at its $50 per task-hour list price.

On this run, CloudThinker Cyber came out ahead with 65% more confirmed findings, about 11 times faster, at 1/17th of the cost, or about $1.50 per finding against $43. Below are the method, each tool's raw results, where the cost difference comes from, and the limits of a single-target comparison.

The setup

The target was OWASP Juice Shop, a fake online store built on purpose with dozens of known security holes. The security community uses it to train people and to test tools, because everyone knows what should be found. We deployed and own the instance both agents tested. Its hostname and run IDs are redacted from the screenshots below.

01Target

OWASP Juice Shop

A deliberately vulnerable app, deployed on infrastructure we own.

02Access

Black box

A URL and nothing else. No source, no credentials, no API schema.

03CloudThinker Cyber

Oct 11, 2026

One pentest run, billed in CloudThinker credits.

04AWS Security Agent

Oct 13, 2026

One full pentest run, billed per task-hour.

Same target, same starting point. Each tool ran once with its default settings.

Both runs were black box, meaning each agent got only the web address, the same view an outside attacker has, and had to discover everything else on its own. We didn't hand either one source code, test credentials or an OpenAPI file. Juice Shop is a fair benchmark for this because its vulnerabilities are known and documented, so a finding is easy to check by hand, and neither tool can claim credit for something that isn't there.

The results

01Confirmed findings

38 vs 23

CloudThinker Cyber found 15 more, with 2 more held for verification.

02Run time

27 min vs 5 h 9 min

Wall-clock time from start to completed report, about 11 times faster.

03Cost of the run

$57 vs $998

571.75 credits against 19 hours 57 minutes of task time at $50 an hour.

04Cost per finding

$1.50 vs $43

Run cost divided by confirmed findings for each tool.

CloudThinker Cyber first, AWS Security Agent second in each cell.

Here's the raw breakdown from each console.

	CloudThinker Cyber	AWS Security Agent
Critical	2	5
High	17	9
Medium	14	9
Low	5	0
Total reported	38 confirmed, 2 need verification	23, plus 10 lower-confidence or informational (hidden by default)
Run time	27 min	5 h 9 min
Billing	571.75 credits	19 h 57 min task time, 20 h cap
Cost	about $57	about $998

CloudThinker Cyber overview for the Juice Shop BlackBox target. A banner reads 2 critical issues are open, last pentest 1 day ago, with 2 Critical, 17 High, 14 Medium, 5 Low, 38 Confirmed and 2 Need verification. Below it, OWASP Web Top 10 shows 4 of 10 categories clean, a fix queue lists SQL injection in the login email as item 1, and the detail pane shows finding F-0026 with an attack flow, a description and a Create PR with agent button.

AWS Security Agent penetration test run for juice-shop-blackbox on October 13, 2026, status Completed, type Full pentest. The run summary shows a duration of 5 hours 9 minutes 18 seconds, task hours of 19 hours 57 minutes against a max of 20 hours, and test scope Web application. A severity donut shows 23 total findings: 5 Critical, 9 High and 9 Medium. A risk types chart lists privilege escalation, database modification, business logic, information disclosure, local file inclusion and authentication bypass.

Read the severity column with care

AWS labeled 5 findings Critical. We labeled 2. That isn't evidence that AWS found three critical bugs we missed: each tool has its own severity scale, and Juice Shop has several issues (SQL injection in login, SQL injection in product search, JWT weaknesses) that one tool calls Critical and another calls High. Our 17 High includes account takeover and cross-user tampering that a stricter scale would push up.

So we compare confirmed findings, not labels. On that count it's 38 to 23. If you add AWS's 10 hidden lower-confidence findings, it's 38 to 33, and AWS classifies those 10 as lower-confidence or informational.

Where the cost difference comes from

AWS Security Agent is priced at $50 per task-hour. A task-hour is one hour of the agent actively working, and the agent runs tasks in parallel. That's why a run of 5 hours 9 minutes wall-clock billed 19 hours 57 minutes: several workers were exploring at once, and each one was on the meter.

Two things follow from that model.

Cost tracks effort, not results. The bill grows with how long the agent spends exploring, so a larger attack surface or a longer exploration costs more regardless of how many findings it produces. This run used 19 hours 57 minutes of a 20-hour cap.

Scale multiplies it. One run at $998 is fine for a quarterly check. Run it on every release and the arithmetic changes fast: weekly runs on one app come to about $52,000 a year at this rate, against about $3,000 for the same cadence on CloudThinker Cyber.

CloudThinker Cyber bills in credits for the work the agents do, not for wall-clock time per worker. On this run that came to $1.50 per confirmed finding, against about $43 on AWS.

One finding, end to end

The top item in our fix queue is a good example of what a confirmed finding looks like. It's a SQL injection: the app pastes what a user types straight into a database query, so a carefully crafted input can change what the query does. Juice Shop's login endpoint is POST /rest/user/login, and the question for any pentester is whether the email field reaches the database unescaped. There are three readings:

Parameterized query. The email is bound as a value. A quote in the input is just a character, and login fails as normal.
Escaped but concatenated. Quotes are escaped. Injection fails, but a verbose error might leak the query shape.
Raw concatenation. The email lands in the SQL string as is. A '-- ends the statement early and the password check disappears.

The agent tested the third reading with a single request:

curl -s -X POST https://<juice-shop-host>/rest/user/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@juice-sh.op'\''--","password":"anything"}'
# if parameterized: 401, "Invalid email or password."
# observed: 200, with authentication.token for user id 1 (admin)

The server returned a valid admin session token (a JWT) for a wrong password: anyone on the internet could log in as the administrator. That's the proof, and only then did the finding get its Confirmed tag.

01Input

Email field

A login request with a SQL comment appended to the admin's email.

02Flaw

Raw SQL

The login route concatenates the email into the query string.

03Effect

Password ignored

The comment cuts off the password clause, so admin row 1 comes back.

04Proof

Admin JWT

The server issues a valid admin session token. Finding confirmed.

Finding F-0026, as CloudThinker Cyber recorded it. No proof, no Confirmed tag.

The write-up names the cause precisely: login() in routes/login.ts:34 builds the query by template-literal concatenation of req.body.email and the MD5 hash of req.body.password into SELECT * FROM Users WHERE email = '${...}' AND password = '${...}' AND deletedAt IS NULL. Juice Shop is open source, so the agent could tie a black-box result back to the exact line. The finding is tagged OWASP A03 and CWE-89 (the standard industry labels for injection flaws and SQL injection), and it works without logging in, from anywhere on the internet.

From there, Create PR with agent opens a merge request with a parameterized query, and the next run retests the route. Changes like that go through Auto Mode, so you decide what runs on its own and what waits for approval.

What the comparison shows, and what it doesn't

	CloudThinker Cyber	AWS Security Agent
Confirmed findings on Juice Shop	38 (plus 2 pending verification)	23 (plus 10 lower-confidence)
Run time	27 min	5 h 9 min
Cost of the run	about $57	about $998
Pricing model	Credits for the work done	$50 per task-hour, parallel tasks billed separately
Cost per confirmed finding	about $1.50	about $43
Output	Fix queue, attack flow per finding, OWASP Top 10 map, merge request from the finding	Findings with severity and risk type, exportable report

The deeper difference is the pricing model. AWS Security Agent bills for agent time, so the cost of a run depends on how long exploration takes. CloudThinker Cyber is designed so a pentest costs little enough to run on every release, with each finding proved, prioritized and handed to the developer as a fix. Teams that pentest a few times a year may weigh this differently from teams that want a run per release.

Limits of this comparison:

One target, one run each. Juice Shop is a standard benchmark, but it isn't your app. Run both on your own staging environment before you decide.
Two days apart. The runs were on October 11 and October 13, against the same Juice Shop deployment.
Free trial. New AWS Security Agent customers get a two-month free trial with up to 200 task-hours a month. The $998 is the list price for this run once that's used up.
Neither tool found everything. Juice Shop has well over 100 documented challenges. Both agents left plenty on the table, which is a good reminder that an AI pentest complements a human red team rather than replacing it.

How to reproduce this benchmark

Anyone can rerun this comparison. Here's everything we used.

	CloudThinker Cyber	AWS Security Agent
Target	OWASP Juice Shop v20.2.0, default configuration, public HTTPS URL	Same instance
Access	Black box: target URL only, no credentials, source or API schema	Same
Run type	One pentest, target type Domain, default settings	One penetration test, type Full pentest, test scope Web application, max task hours 20
Date	October 11, 2026	October 13, 2026
Pricing used	571.75 credits at $0.10 per credit	19 h 57 min of task time at the $50 per task-hour list price

How we counted:

Confirmed findings are the findings each console shows by default. For CloudThinker Cyber, that's findings with Confirmed status. For AWS Security Agent, it's the run's total with "Show lower-confidence and Informational findings" switched off.
Run time is wall-clock time from the start of the run to the completed report.
Cost is the billed usage of the run at list price, before any free trial or discount.

Our instance's hostname is redacted from this post. Any fresh deployment of Juice Shop v20.2.0 works as a target.

The cost of waiting

Look again at finding F-0026. It's a pre-auth admin login bypass, reachable from the internet, and the agent proved it with a single HTTP request. Nothing about it needed insider knowledge or a long engagement.

That's the part that should change how you schedule testing. The agents that found these 38 issues aren't exclusive to defenders. Anyone with a target list can point the same class of tool at your login page tonight, and they won't wait for your next pentest window.

Most teams still test on a calendar. With a quarterly pentest, a bug merged the day after the test stays in production for about 90 days before anyone looks for it. Your release cadence has moved to days; the attacker's cadence has moved to minutes. A test cycle measured in quarters is the slowest part of that picture.

At $57 a run, the economics no longer force that trade-off. A weekly pentest on one app costs about $3,000 a year, and at 27 minutes a run it fits inside a release pipeline instead of beside it. Neither cost nor time is a reason for a release to ship untested.

Try it on your own app

Run your first pentest at app.cloudthinker.io/cyber.