📝 Turning pytest into an “Incident-Prevention Ledger” — Phase 1: From 3 Simple Tests to 9 Regression Tests

Update

This article records Phase 1, when I expanded the pytest suite from 3 tests to 9.

At that time, AI responses were rendered using a combination of createTextNode() and <br> elements.

After publication, a reader suggested using innerText. I tested that approach in my own environment and confirmed that, for this use case, it preserved line breaks while keeping HTML-like strings from being interpreted as HTML. The current implementation therefore uses innerText.

The XSS regression tests have also been updated to match the current implementation. They now verify that innerText remains in use and that dangerous HTML sinks such as innerHTML have not returned to the relevant rendering paths.

For that reason, references to createTextNode() and <br> in this article should be understood as a record of the implementation at the time Phase 1 was completed.

This article was originally published in Japanese on Qiita and has been translated and adapted for DEV Community.

GitHub:

https://github.com/tosane932/sales_data_app

pytest Improvement Series

Phase 1

https://qiita.com/tosane932/items/f3de1e190873a90de39f

Phase 2

https://qiita.com/tosane932/items/b91261e7103df5792f7d

Phase 3

https://qiita.com/tosane932/items/6d1ca5490979c8cf9d62

Phase 4

https://qiita.com/tosane932/items/372270330e73583a227f

Introduction

Hello from Japan! 🇯🇵

I have been reviewing the security and maintainability of a Flask application I am developing, with help from Codex.

So far, I have fixed issues such as:

• XSS risks in AI response rendering
• Stored XSS in dynamically generated product rankings
• A broken migration history that could not build the application from an empty PostgreSQL database

While working through those fixes, one question started bothering me:

Is my current pytest suite far too simple?

At the time, I had only three pytest tests.

All three focused on a single function:

build_sales_prompt()

So I decided to change how I thought about pytest.

Instead of treating it only as:

A tool for checking whether the application currently works

I wanted to develop it into:

An incident-prevention ledger that records bugs, vulnerabilities, and near misses so the application cannot silently return to the same dangerous state later.

For Phase 1, I added or strengthened:

• Full pytest discovery in GitHub Actions
• XSS regression tests
• Jinja autoescape regression tests
• Prompt contract tests
• A test confirming that the complete prompt is actually sent to Gemini

The Original pytest Suite

Before this improvement, only three tests existed in test_prompts.py.

They mainly checked that:

• Sales data appeared somewhere in the prompt
• The phrase requesting three suggestions existed
• The return value was a string

At first glance, that sounds like a test suite.

But after asking Codex to review the tests statically, I realized how weak they were.

In an extreme case, an implementation like this could potentially still pass:

return sales_summary + " Give me three suggestions. Three bullet points."

That would still satisfy:

• The input sales data exists
• A fixed phrase exists
• The return value is a string

But important parts of the real prompt could disappear, including:

• The AI's role
• Analysis perspectives
• Number of recommendations
• Response format
• Expected answer length

and the tests might still pass.

In other words:

Tests passing did not necessarily mean the intended specification was protected.

Another Problem: CI Was Not Running Every Test

During the investigation, I also found a problem in GitHub Actions.

The CI workflow was running:

pytest test_prompts.py -v

That meant if I later added:

test_security.py
test_sales.py
test_api.py

GitHub Actions would not run them.

They might pass locally but be completely ignored by CI.

That is a serious problem if pytest is supposed to become an incident-prevention system.

I changed the command to:

pytest -v

This lets pytest use its normal test-discovery rules and automatically collect newly added test files.

The actual change was only one line:

- pytest test_prompts.py -v
+ pytest -v

But for the future of the test suite, that one line was important.

The Goal of Phase 1

The goal was not simply:

Increase the number of tests.

Instead, I wanted this behavior:

If a bug or vulnerability that I have already fixed returns in the future, pytest should fail immediately.

For this phase, I deliberately avoided changing application behavior.

I focused on converting already-fixed behavior into regression tests.

Growing the Suite from 3 Tests to 9

By the end of Phase 1, the suite had grown from three tests to nine.

1. Verify That Sales Data Appears in the Correct Section

The first strengthened test was:

test_build_sales_prompt_places_sales_data_in_its_section

Previously, the test only checked whether the sales data appeared somewhere in the prompt.

The new test checks that it appears in the correct sales-data section.

This makes it harder for a broken prompt structure to pass unnoticed.

2. Protect the AI Role and Analysis Contract

The next test was:

test_build_sales_prompt_preserves_role_and_analysis_contract

The prompt contains important instructions defining:

• The AI's role
• The perspectives the AI should use when analyzing the sales data

Previously, a few surviving fixed strings could be enough for the tests to pass.

Now, the test protects the broader analytical contract of the prompt.

3. Protect the Output Contract

I also added:

test_build_sales_prompt_preserves_output_contract

This verifies important response requirements such as:

• Number of recommendations
• Bullet-point formatting
• Expected answer length
• Conciseness

I intentionally did not compare the entire prompt character-for-character.

A full exact-string comparison would make the test excessively fragile.

Even a harmless wording improvement could break it.

Instead, I test the important pieces of the contract.

Does the Complete Prompt Actually Reach Gemini?

One test I considered especially important was:

test_generate_ai_advice_sends_complete_sales_prompt

Previously, I only tested:

build_sales_prompt()

in isolation.

But even if that function works perfectly, it is useless if the actual Gemini request does not use the prompt it creates.

So the new test verifies that:

• The content generated by build_sales_prompt() is actually passed to Gemini
• The configured model name is used
• Gemini's returned text is passed back correctly

This connects the unit-level prompt logic to the actual AI integration path.

Mocking Gemini Instead of Calling the Real API

I did not want pytest to call the real Gemini API every time the suite ran.

That would introduce unnecessary problems:

• Real API keys would be required
• External network access would be required
• Tests could consume API quota
• Results could depend on network availability
• External service failures could make local tests fail

So I used:

unittest.mock.Mock

to mock the Gemini client.

I configured a dummy API key and replaced:

genai.Client

with a mock.

The response was also fixed:

SimpleNamespace(
    text="Mocked AI advice"
)

This created a test with:

• No real API key
• No external communication
• No real Gemini request
• No dependency on network conditions

Regression Tests for XSS

One of the most important goals of this phase was:

Preserve the XSS fixes I had recently made.

I added:

test_dynamic_ranking_product_name_uses_text_dom_api
test_dashboard_ai_responses_use_text_dom_api
test_input_ai_response_uses_text_dom_api

Stored XSS in the Dynamic Ranking

Previously, dynamically displayed product names were inserted using:

innerHTML

I had already changed that code to use safe DOM APIs and:

textContent

The regression test protects that specific rendering path so that a future refactor does not accidentally return product names to a dangerous HTML sink.

XSS in AI Response Rendering

AI business advice and AI greetings had also previously used:

innerHTML

At the time Phase 1 was implemented, I had replaced that with:

• createTextNode()
• <br>
• DOM APIs

so that the AI response itself would never be interpreted as HTML.

I added source guards to ensure that those rendering paths did not silently return to innerHTML.

After publishing the earlier article, I tested and adopted innerText.

The regression tests have since been updated to match the current implementation.

They now verify:

• innerText remains in the relevant rendering path
• innerHTML does not return there
• outerHTML does not return there
• insertAdjacentHTML does not return there

The implementation changed.

The protected security property did not.

I Did Not Ban the Word innerHTML Everywhere

I thought carefully about this.

It would be very easy to write a test like:

Fail if the string "innerHTML" appears anywhere in the file.

But that would be too broad.

A future feature might have a legitimate and safely controlled reason to use innerHTML.

A global ban could create false failures unrelated to the original vulnerability.

So I limited the regression tests to the actual external-string rendering paths that previously caused problems:

• Product rankings
• AI business advice
• AI greetings

The goal of pytest is not:

Ban the word innerHTML.

The goal is:

Prevent the application from returning to the same dangerous state that previously existed.

Turning Jinja Autoescape into a Regression Test

The initial AI response display had previously used:

| safe

I removed it and returned to Jinja's default autoescaping behavior.

So I added:

test_dashboard_initial_ai_binding_does_not_disable_autoescape

This source guard checks that:

{{ ai_advice }}

is used and that:

| safe

has not returned to that rendering path.

This makes it easier to detect a future regression where autoescaping is accidentally disabled again.

Actually Rendering HTML-Like Text Through Jinja

I added another test:

test_dashboard_initial_ai_advice_autoescapes_html_like_text

This one goes further than inspecting the template source.

For example, I pass strings such as:

<b>Test</b>
<img src=x>

into the Jinja template.

The test then confirms that they do not become real:

<b>

or:

<img>

elements.

Instead, they must remain escaped text.

This test actually renders the template and inspects the generated HTML using BeautifulSoup.

So I now had both:

• A source guard
• A behavior-level rendering test

protecting the same security boundary from different directions.

pytest Results

First, I checked test discovery:

pytest --collect-only -q

At the time Phase 1 was completed:

9 tests collected in 8.23s

All new tests were successfully discovered.

Then I ran:

pytest -v

The result was:

9 passed in 6.37s

Summary:

Passed: 9
Failed: 0
Skipped: 0

These nine tests represent the suite at the end of Phase 1.

I have continued adding regression tests since then as I discover more bugs and specifications worth protecting.

Phase 1 Did Not Use PostgreSQL, Docker, or the Real Gemini API

For this first phase, I intentionally focused on lightweight regression tests.

The tests did not require:

• PostgreSQL
• Docker
• The real Gemini API

They also performed no real database operations.

My priority was to build:

Fast tests with minimal external dependencies.

Database-related integration testing would come later.

Files Changed

Phase 1 changed four files:

.github/workflows/test.yml
test_prompts.py
test_ai_integration.py
test_xss_regressions.py

The diff was:

4 files changed, 195 insertions(+), 13 deletions(-)

I did not modify the application code in:

app.py
prompts.py
models.py
templates/
migrations/

for this commit.

The goal was not to change the existing implementation.

It was:

Lock the already-correct behavior in place with tests.

Commit

After final verification, I created:

01d76e8 test: strengthen regression coverage

and pushed it to:

origin/main

What I Learned

1. The Number of Tests Matters Less Than What They Protect

Previously, I had three pytest tests.

But all three mostly examined one function.

Numerically:

3 tests
        ↓
9 tests

looks like the main improvement.

It was not.

The important improvement was that the protected surface expanded from:

Basic prompt string checks

to:

XSS
Jinja autoescape
Prompt contracts
Gemini integration
Full CI test discovery

Test count is easy to measure.

Protected behavior is what actually matters.

2. “HTTP 200” Is Not Enough

This work also reinforced another lesson:

HTTP 200

alone is a weak definition of success.

A request can return successfully while:

• The database ends up in the wrong state
• HTML is rendered unsafely
• Duplicate records are created

So future tests need to check not only the response itself, but also:

What state exists after the request finishes?

3. pytest Can Become an Incident-Prevention Ledger

The biggest change was how I started thinking about pytest.

Before this work, my mental model was:

pytest
=
Check whether things work correctly

Now I think of it more like:

A record of previous incidents, bugs, and near misses that automatically stops the application from returning to the same dangerous condition.

For example:

Discover XSS
        ↓
Fix it
        ↓
Add a regression test
        ↓
A future change recreates the unsafe state
        ↓
pytest fails
        ↓
CI turns red

That means I do not need to remember every security fix forever.

The test suite remembers for me.

Not “Never Make the Same Mistake Again,” but “Never Recreate the Same Dangerous State”

This way of thinking comes partly from my day job.

When an incident or mistake happens, saying:

I'll be more careful next time.

is not enough.

That still relies on human attention.

The stronger question is:

How can we prevent the same dangerous condition from existing again?

I found the same principle useful in software development.

For XSS, the goal should not be:

Do not enter dangerous HTML.

Instead:

Even if HTML-like input arrives,
do not allow it to execute as HTML.

And pytest adds another layer:

If the safe implementation is broken later,
automatically stop the change.

That turns a lesson into a system.

What I Planned to Improve After Phase 1

At the time this Phase 1 article was written, I had focused on lightweight regression tests that did not require external APIs or a database.

Areas that were still largely unprotected included:

• PostgreSQL empty-database migrations
• Migration behavior on existing databases
• Authentication and authorization
• CSRF
• Invalid sales input
• Soft deletion and historical data preservation
• Duplicate daily-sales prevention
• Gemini API 429 responses
• Gemini API 503 responses
• General Gemini API exceptions
• API response behavior
• Dashboard aggregation

Instead of changing all of these areas at once, I planned to expand coverage gradually using this cycle:

Choose one dangerous state
        ↓
Write the test first
        ↓
Fix the implementation
        ↓
Run the full test suite

For database-related tests in particular, I also started paying more attention to:

The state of the database after invalid input or a failed save

rather than checking only HTTP responses.

Authentication, authorization, CSRF, and some other areas would be addressed in later phases.

Conclusion

During Phase 1, my pytest suite grew from 3 tests to 9.

But increasing the number was not the real objective.

The goal was to take bugs and vulnerabilities I had already discovered and preserve them as:

Regression rules that prevent the same dangerous states from returning.

The workflow became:

Discover
        ↓
Understand the cause
        ↓
Fix
        ↓
Add a regression test to pytest
        ↓
Verify automatically in CI

I want pytest to evolve from:

A tool that confirms normal behavior

into:

An incident-prevention ledger.

This article records only the first phase.

Since then, I have continued expanding the areas protected by tests using the same idea.

The AI-response rendering implementation also evolved after publication.

A reader suggested innerText, so I tested it instead of accepting the suggestion immediately.

That reinforced another principle I want to keep using:

Whether a suggestion comes from a human or an AI, verify it in your own environment before adopting it.

Related Articles

pytest Improvement Series

https://qiita.com/tosane932/items/f3de1e190873a90de39f

https://qiita.com/tosane932/items/b91261e7103df5792f7d

GitHub

https://github.com/tosane932/sales_data_app

Qiita

https://qiita.com/tosane932