Inside the FPT Cloud Evaluation: How CloudThinker AI Code Review Held the First Quality Gate
Enterprise engineering organizations face a persistent tension in code review. Review quality depends on senior engineers, the scarcest resource in the organization, while release velocity pushes review depth down at exactly the moments when risk is highest. For providers operating multi-tenant platforms the stakes compound: a single cross-tenant data-access defect is a compliance incident, not a routine bug.
This post is the engineering account of how FPT Cloud evaluated CloudThinker AI Code Review over five weeks on production merge requests. We describe the challenge encountered in each phase, the decisions that addressed it, and the measured outcome: roughly 97% precision, full coverage of the backend logic defects in the evaluated set including every security finding, and 68.6% of all verified defects caught at the merge request before human review or QC testing.
The challenge
Three constraints shaped the evaluation:
- Security and tenant isolation are non-negotiable. The platform serves multiple enterprise tenants, so defects that cross tenant boundaries carry regulatory and contractual consequences.
- Senior review capacity is limited. Every merge request requires experienced review, but senior engineers cannot scale linearly with delivery volume, and review depth degrades under deadline pressure.
- Quality claims must be measurable. In a compliance-driven environment the review process itself has to be auditable. Adopting an AI reviewer requires quantitative evidence on real workloads, not anecdotes.
So the evaluation was structured as a controlled measurement rather than a demonstration. CloudThinker AI Code Review ran on production merge requests, every finding was verified by FPT Cloud engineers, and coverage was measured against a ground truth assembled from all defects identified by any quality layer: AI review, human review, or QC testing.
Week 1: Baseline evaluation and the context gap
The first week established a baseline and surfaced the central technical challenge of the engagement. Accuracy was initially low, and the limiting factor was context rather than model capability.
An AI reviewer operating only on the diff has no visibility into intended behavior. It cannot see the tenant-isolation rules, the valid value ranges, or the domain conventions the team has accumulated over years of development. The baseline reflected that: findings that were technically plausible but wrong for the business domain, and missed defects in logic that can only be judged with domain knowledge.
That diagnosis set the roadmap for the following two weeks. Rather than tuning the model, the team focused on systematically supplying the business context the reviewer lacked.
Week 2: Integrating requirement specifications from code and Jira
In the second week the reviewer was integrated with the two systems of record where intended behavior is documented:
- Requirement specifications maintained in the codebase, so each review evaluates the change against documented behavior rather than code hygiene alone.
- Jira issues, so each merge request carries its acceptance criteria, value bounds, and business rules into the review context.
This changed the question the reviewer answers, from "is this code well formed?" to "does this change implement the documented requirement, without unintended side effects?" That reframing is what enables detection of static business-logic defects: a chart legend silently re-sorted, a column-width setting silently dropped, a configuration change applied to the wrong target.
Week 3: Automated rule generation with senior engineer curation
The third week addressed FPT Cloud's internal engineering standards, most of which existed as institutional knowledge rather than documentation. Instead of asking senior engineers to author a rulebook by hand, the team inverted the process:
- CloudThinker generated candidate rules automatically, derived from patterns in the codebase and from recurring themes in historical review feedback.
- Senior engineers curated the generated rules, keeping those that reflected genuine intent, correcting inaccuracies, and adding the judgment-based standards only they could articulate.
The result is a machine-readable rule set the reviewer loads as context on every merge request. Senior expertise stops being a per-review bottleneck and becomes a durable asset applied consistently across the organization.
Weeks 4 and 5: Evaluation methodology and results
With the context integrations in place, the final two weeks measured performance on a representative set of production merge requests. Engineers verified every AI-reported finding, and coverage was computed against the complete set of defects identified by any layer.
| Metric | Result | Interpretation |
|---|---|---|
| Precision | ~97% | False positives were rare, so engineers acted on findings instead of triaging them |
| Backend logic coverage | Full | Every defect observable in the code itself was detected, security findings among them |
| Share of all defects | 68.6% | Caught at the merge request, before human review or QC testing |
Two methodological notes accompany these results:
- The figures are preliminary. A subset of the evaluated merge requests had not completed QC testing at the time of measurement, so coverage on that subset represents an upper bound.
- Coverage is deliberately reported in two scopes. The 68.6% figure charges code review with every defect found anywhere, including UI and runtime defects that only QC can see. The backend logic figure describes the defect class code review is actually responsible for. Both numbers are needed for an accurate assessment.
Detection coverage across quality layers
Figure 1: Defect detection across the three quality layers. AI review identified 68.6% of all defects before human review and QC testing.
The code-layer defects the AI reviewer did not report were all code-convention violations. Each maps directly to rules that had already been generated in week 3 but had not yet been loaded into the review context at the time of measurement. With the curated rule set applied, that gap closes.
Complementary coverage: AI review and QC
The distribution of defects across layers showed a consistent division of responsibility rather than overlapping effort.
Defects identified by AI review (code and diff, in scope):
- Technical and code quality: a
NaNpx CSS value, duplicated logic, and a configuration change applied to the wrong target - Static business logic: a silently re-sorted chart legend and a silently dropped column-width setting
- Security, every finding in this category: cross-tenant IDOR vulnerabilities and credentials stored in plaintext
Defects identified by QC (runtime execution, outside review scope):
- An application crash when expanding all rows over large datasets
- Layout regressions and distorted gauges following font and padding changes
- Drill-through, pagination, and page-load performance issues
The pattern is precise: the AI reviewer's blind spot is runtime behavior and UI rendering, not business logic. Every defect observable in the code itself was detected. The QC-identified defects only manifest when the code executes and the interface renders. The two layers are complementary by scope, and neither substitutes for the other.
Severity distribution of verified findings
The severity profile of the verified AI findings is as follows:
| Severity | Share of findings |
|---|---|
| Critical | 0% |
| High | 37% |
| Medium | 37% |
| Low | 26% |
A single merge request accounted for nearly half of the High-severity findings, including:
- IDOR vulnerabilities permitting cross-tenant access to workspace data
- A partial failure incorrectly reported as
Succeeded - A silent data-copy failure that left the destination table empty
For a multi-tenant provider these are the highest-cost defect classes that can reach production: a tenant-isolation breach, an integrity failure in status reporting, and a silent data-loss condition. All were identified at the merge request, before human review, QC, or deployment.
Path to production
The evaluation concluded with three recommendations for rollout, ordered by expected leverage:
- Load the curated rule set into the review context. This recovers the missed convention findings at negligible cost.
- Attach the feature specification to every merge request. This extends coverage from static logic into dynamic behavior and value-bound validation.
- Continue reporting coverage in two scopes. Keeping the in-scope and blended figures separate preserves an accurate view of code review performance as the system operates at scale.
Conclusion
This engagement shows that the determining factor in enterprise-grade AI code review is not model selection but context engineering. The week 1 baseline, the specification integration in week 2, and the curated rule generation in week 3 map directly onto the movement from low initial accuracy to roughly 97% precision with full backend logic coverage.
Three conclusions generalize beyond this engagement:
- Low initial accuracy is usually a context problem. A reviewer that reads only diffs cannot evaluate business intent. Supplying specifications, issue context, and codified standards is what closes the gap.
- Institutional standards can be codified efficiently. Automated rule generation with senior curation captures engineering judgment without imposing a manual documentation burden.
- Evaluation rigor is a requirement, not an option. Verified findings, ground-truth coverage, dual-scope reporting, and disclosed limitations are what make results usable in a compliance-driven adoption decision.
Where merge requests routinely touch tenant data and senior review capacity is constrained, the operating model is straightforward. AI review serves as the first quality gate, catching what is observable in the code at the least expensive point in the pipeline, so human reviewers and QC can concentrate on the work that requires them.
Read the FPT Cloud case study for the business view of this engagement, or visit the Code Review product page and contact our team.