RAG Test Script Generator

Overview

Regulated software validation requires formal test scripts: precise, traceable steps that prove a system behaves as specified. Writing them is repetitive, format-heavy work — yet it demands accuracy, because auditors read them. This platform generates draft test scripts from an organization’s own historical validation assets using retrieval-augmented generation (RAG), so every generated script inherits the house style, structure, and domain conventions reviewers expect.

The engineering problem

Validation engineers spent hours per script formatting and phrasing content that followed predictable patterns from past work. The knowledge to do it well already existed — in hundreds of previously approved scripts — but it was locked in documents nobody could efficiently search. Generic LLM output was unusable: wrong terminology, wrong structure,

Architecture

<rect x="20" y="120" width="160" height="56" style="fill:var(--paper);stroke:var(--ink)"></rect>
<text x="100" y="144" text-anchor="middle" style="fill:var(--ink)">New Requirement</text>
<text x="100" y="160" text-anchor="middle" style="fill:var(--dim)">spec / change input</text>

<rect x="280" y="70" width="170" height="66" style="fill:var(--paper);stroke:var(--red)"></rect>
<text x="365" y="94" text-anchor="middle" style="fill:var(--red)">RAG Pipeline</text>
<text x="365" y="110" text-anchor="middle" style="fill:var(--ink)">retrieve similar cases</text>
<text x="365" y="126" text-anchor="middle" style="fill:var(--ink)">+ few-shot selection</text>

<rect x="550" y="30" width="200" height="56" style="fill:var(--paper);stroke:var(--ink)"></rect>
<text x="650" y="54" text-anchor="middle" style="fill:var(--ink)">Azure OpenAI</text>
<text x="650" y="70" text-anchor="middle" style="fill:var(--dim)">constrained generation</text>

<rect x="550" y="120" width="200" height="56" style="fill:var(--paper);stroke:var(--ink)"></rect>
<text x="650" y="144" text-anchor="middle" style="fill:var(--ink)">Validation Layer</text>
<text x="650" y="160" text-anchor="middle" style="fill:var(--dim)">structure + terminology checks</text>

<rect x="280" y="170" width="170" height="56" style="fill:var(--paper);stroke:var(--ink)"></rect>
<text x="365" y="194" text-anchor="middle" style="fill:var(--ink)">Engineer Review</text>
<text x="365" y="210" text-anchor="middle" style="fill:var(--dim)">edit → approve → reuse</text>

<path d="M180,58 L280,90" style="stroke:var(--ink);fill:none;marker-end:url(#ah3)"></path>
<path d="M180,148 L280,116" style="stroke:var(--ink);fill:none;marker-end:url(#ah3)"></path>
<path d="M450,90 L550,66" style="stroke:var(--ink);fill:none;marker-end:url(#ah3)"></path>
<path d="M650,86 L650,120" style="stroke:var(--ink);fill:none;marker-end:url(#ah3)"></path>
<path d="M550,148 L450,178" style="stroke:var(--ink);fill:none;marker-end:url(#ah3)"></path>
<path d="M280,198 L100,176" style="stroke:var(--red);fill:none;marker-end:url(#ah3)"></path>

Technical design

• Asset base — approved historical scripts are parsed, chunked, embedded, and indexed with rich metadata (domain, script type, system area) so retrieval can be filtered, not just ranked.
• Few-shot selection — the pipeline retrieves the most similar approved examples and injects them as few-shot context. The model imitates real house style instead of a generic idea of a test script.
• Constrained generation — output must conform to the script template schema; structure violations are rejected before human review.
• Feedback loop — engineer edits are captured, and newly approved scripts re-enter the asset base, so generation quality compounds over time.

Key engineering decisions

• Why RAG over fine-tuning? The corpus changes constantly and per-domain; retrieval keeps knowledge current and traceable to real approved artifacts, which fine-tuning cannot cite.
• Why few-shot from retrieved examples? Format compliance was the biggest failure mode of zero-shot generation; showing the model 2–3 genuine exemplars outperformed any amount of instruction.
• Why a validation layer before review? Engineers should review content, not catch malformed structure. Cheap deterministic checks protect expensive expert attention.

Challenges & solutions

• Similar-but-wrong retrievals — scripts from adjacent domains looked similar and misled generation. Fixed with metadata filtering and stricter similarity thresholds.
• Confident hallucination of steps — grounded generation against the requirement document and required explicit traceability from each generated step to a source.

Results

• Regulated validation effort reduced from hours to minutes per script (~40% improvement) in the target workflow.
• Generated scripts required structure fixes at near-zero rate after the validation layer shipped; expert time shifted to technical review.
• The pattern (retrieval + few-shot + deterministic validation) became the template for other AI-assisted engineering tools in the organization.

What I learned

For structured professional writing, retrieval quality is generation quality. The model is a stylist; the corpus is the substance.

What I would change today

I would build the feedback loop earlier — approved edits are the highest-signal training data a system like this can get, and we captured them too late.

Technologies

Python · RAG · Azure OpenAI · Vector Search · PostgreSQL · FastAPI

and no traceability to the approved norms.

My role

I designed the retrieval strategy, the prompt architecture (including few-shot selection), the generation pipeline, and the review workflow that turns generated drafts into approved scripts.