AI code review walkthrough: a recorded finding, a tested fix and the limits

What we actually ran through CodeStax

AI code review is useful when a reviewer can connect a finding to changed code, inspect the reasoning and test a proposed fix. This walkthrough follows that process with a fresh execution of the deployed CodeStax review engine. The input is a small Python lookup function written for the demonstration. The outputs are live model responses, not the deterministic replay fixture used in our earlier feature-page example.

We called the deployed PR AI analysis function directly with two diffs: first the vulnerable function, then a patch that binds the email value separately. This exercises diff preparation, model inference, structured-output validation and finding normalization. It does not exercise repository authorization, webhook receipt, queue dispatch, full scanners, database persistence, GitHub or Bitbucket comment delivery, or a merge gate. We supplied no repository-wide code graph.

An initial OpenAI attempt returned unavailable. We then captured the pair through the already-enabled Anthropic route without changing production configuration. That failed attempt is disclosed in the evidence. An unavailable AI response with no findings is not a security pass, and choosing a working route does not establish reliability across providers.

Inspect the recorded review evidence

Download the recorded AI review evidence (JSON) to inspect both patches, all accepted findings, output-validation results, sample code and the database check results. The extract includes artifact hashes and hashes of the deployed review implementation. The model identifier is recorded as returned, but its revision is an unversioned alias. Do not interpret that identifier as an immutable model build or a guarantee that a future call will return the same answer.

Read the original code and the AI finding

def find_users(connection, query_args):
    """Look up users from a request query-parameter mapping."""
    email = query_args.get("email", "")
    query = f"SELECT name FROM users WHERE email = '{email}'"
    return connection.execute(query).fetchall()

The query_args mapping represents request query parameters. At line 3 the function reads email; at line 4 an f-string inserts that value into SQL syntax. At line 5 SQLite executes the assembled statement. If the email value contains SQL syntax, the database receives the resulting statement rather than a separately bound email value. Type annotations or a longer explanation in the review would not change that boundary.

The actionable part of the response was the link between request-controlled data, string interpolation and a SQL execution call. We tested that claim against an in-memory database containing two synthetic users. With the email input shown below, the original function returned both records instead of selecting a single matching email. The test is confined to the sample database; it does not establish how an unseen production endpoint is authorized or exposed.

' OR '1'='1

Check the AI explanation instead of accepting every claim

The model's SQL finding also claimed that stacked statements could drop a table and described the issue as complete database compromise. Those statements overreach the evidence for this exact SQLite API. Our test passed a multiple-statement input to connection.execute; the API rejected it and the table remained intact. A filtering bypass is demonstrated. Table deletion through that same call is not. Keep that distinction when writing the review decision.

Another finding suggested an empty email could return the entire table. The query still contains WHERE email = a value, and in our fixture an empty email matched no rows. That result does not prove empty strings are appropriate business input; it does show why the stated all-rows consequence needs correction. The model also commented on missing authorization and exception handling, but this isolated diff does not show the caller or application boundary.

Apply the narrow parameterized fix

def find_users(connection, query_args):
    """Look up users from a request query-parameter mapping."""
    email = query_args.get("email", "")
    if not isinstance(email, str):
        raise ValueError("email must be a string")
    return connection.execute(
        "SELECT name FROM users WHERE email = ?", (email,)
    ).fetchall()

The change keeps the query text constant and passes email in a one-element tuple. SQLite binds that value as data through the question-mark placeholder. The type check rejects unsupported input shapes before query execution. The Python sqlite3 documentation (opens in a new tab) explains parameter binding. The placeholder syntax belongs to the database adapter; do not copy it unchanged into an adapter with a different parameter interface.

This is an editorial fix proposed for the demonstration and then submitted to a second live engine call. It is not an automatically applied product patch. We did not adopt every model recommendation. Replacing the data layer, introducing an ORM or adding broad exception handling would require application context and a separate design decision. The focused change addresses the observed SQL construction problem while preserving legitimate lookup behavior.

Verify the behavior independently of the model

These groups contain 11 individual checks. We ran them in the scanner container's Python 3.11.15 environment using an isolated in-memory SQLite 3.46.1 database. No production user table was queried or modified. The verifier loads the exact source samples retained with the model capture. That connects the test outcome to the displayed code rather than to a similar-looking rewrite.

The tests establish the behavior of this sample and fixture. A real application also needs tests for authorization, input limits, database failures and the intended caller contract. If an empty email is valid for your business, test its intended meaning; if it is invalid, reject it explicitly. Avoid turning an arbitrary model suggestion into a requirement without checking how the product is supposed to behave.

Read the follow-up review without inventing a clean pass

The follow-up AI response recognized the parameterized query as an improvement and did not emit sql-injection-string-interpolation. It still emitted two accepted findings: a low-severity input-validation observation and an informational separation-of-concerns observation. The output validator rejected another comment because its end_line was not an added line in the diff. The response therefore has degraded AI health.

The narrow conclusion is supported: the bound query prevents the demonstrated bypass, and the same SQL rule was not returned in this paired call. The broader conclusion that the application is secure is not supported. Response health, context coverage, finding count and behavior tests answer different questions. One analyzed diff with one missing context record does not mean that a second file was skipped; it means the same changed file lacked repository context.

Follow the workflow on your own pull request

1. Connect an enabled repository provider and import the intended repository. Confirm access to the repository and the pull request you intend to review.
2. Open Dashboard → Reviews and start a review for that repository and pull request, or use configured webhook automation. Check the review's source commit so the findings refer to the code you are evaluating.
3. Inspect AI policy, provider execution evidence and analyzer coverage. A completed job with unavailable AI or missing required analysis does not establish sufficient review coverage.
4. Open an actionable finding and inspect its source, location, explanation and suggestion. Separate deterministic findings from AI output and verify the surrounding code.
5. Apply a focused change in a branch, run regression tests and review the changed commit again. Record whether the specific issue is resolved and what remains uncertain.
6. Verify provider delivery and any required merge checks separately. This demonstration does not establish that a comment was posted or that a branch rule blocks merging.

Use the PR review guide (opens in a new tab) and AI privacy and execution documentation (opens in a new tab) for your configuration. The GitHub SAST walkthrough explains the distinction between a published check and merge enforcement. The TypeScript SAST walkthrough follows a separate recorded static-analysis finding, which should not be presented as AI model output.

Keep a review record that another developer can verify

Retain the reviewed revision, selected finding, proposed change, test results and follow-up coverage together. Record disagreements with the model rather than silently rewriting its output into a success story. In this example, the meaningful outcome includes both a demonstrated SQL fix and limits on the model's explanation. That is more useful to the next reviewer than a lower finding count without context.

For a tool evaluation, repeat this process on representative changes in your own languages and frameworks. One small demonstration does not measure false-positive rate, recall, consistency or delivery reliability. Use CodeStax AI code review alongside SAST, application tests and human review, with each source of evidence clearly identified.