Use Case in Software Testing: How to Turn One Into Test Cases
Every path through a use case becomes at least one test case. A worked bank-transfer example shows the test table, and ISTQB's 2023 and 2025 syllabus changes explain…
AI Model Validation and
Testing Services
Vervali validates machine learning and generative models before and after they reach production: accuracy and calibration against held-out data, bias and fairness across protected groups, drift detection, explainability, robustness to adversarial and malformed input, and evaluation of large language model output against a defined rubric. Findings come with the evidence a model review board will ask for.
150+ clients
450+ projects
275+ engineers
15+ years
ISO/IEC 17025:2017Accredited testing laboratory
CMMI Maturity Level 3The process is written down and repeats
ISO 9001:2015Quality management
ISO/IEC 27001Information security
What Model Validation Covers
Evidence that a model is accurate, calibrated, fair, explainable and robust on data it did not train on, with drift and leakage risks made visible before approval.
Accuracy and Calibration
Is the model right, and does its confidence mean anything. Held-out data, not the training set. A 0.9 score that is wrong as often as a 0.5 score is a calibration failure, and a review board will treat it as one. Precision, recall and error slices sit here so a global number cannot hide a bad group.
Bias and Fairness Testing
Does it behave differently across groups it should not. Protected attributes, subgroups, and the operating point you will actually ship. More than one fairness measure, because a single metric can bless a slice that still fails. The trade-off with accuracy is reported, not buried. Detail lives on AI bias and explainability testing.
Drift Testing
Has the world moved since training. Inputs may no longer match the training distribution, or the relationship between inputs and outcomes may have changed. We define the baseline, thresholds and response for continuous data drift testing.
Explainability Testing
Can a decision be defended to a customer or a regulator. Feature attribution, worked examples, and what a model card contains. If you cannot explain why this applicant was declined, or why this KYC image was flagged, the model is not ready for a review board even if the accuracy looks fine.
Robustness Testing
What happens with malformed, adversarial or out-of-distribution input. A model that is accurate on clean test rows and brittle on a cropped document, a noisy audio clip or a prompt injection is not production-ready. We name the attacks and the malformed cases in the design, then run them.
Data Quality and Leakage
The failure that makes a model look excellent in testing and useless in production. Target leakage, duplicate rows across train and test, labels that were filled in after the fact. If the sandbox score cannot be reproduced on a truly held-out set, that score is not evidence.
Need evidence a model review board will accept, not a sandbox score?
Free Model Review (One Model, One Hour)Validating LLM and Generative Systems
A versioned evaluation set exposes hallucination, weak grounding, unsafe behaviour and silent prompt regression. End-to-end workflow coverage is available through AI and machine learning testing.
The Evaluation Set
Built from your own traffic, not a public leaderboard. Tasks, expected behaviour, and the rubric a reviewer will score against. If the set is not stored, the next prompt change has nothing to fail against. We version it the way a regression pack is versioned, so a model or prompt bump is a diff, not a vibe.
Hallucination and Grounding
Does the output stay inside the sources you allowed. Citations that do not support the claim, numbers invented from tone, and answers that skip retrieval entirely are fails. Grounding is scored against the documents in scope, not against whether the prose sounds confident.
Safety and Refusal
What the model must refuse, what it must escalate, and what it must still answer. Jailbreaks, out-of-scope asks and the prompts that should produce a clean refusal. A model that refuses the work you hired it for is also a fail. The allowed and refused lists are signed before execution.
Prompt Regression and Retrieval
A prompt or model change silently alters output, and without a stored evaluation set nobody notices. That is the discipline most teams skip. Retrieval quality is tested separately from generation, because most failures are retrieval failures wearing a generation costume. If the wrong chunk arrived, the fluent answer is still wrong.
Voice and Call Agents
A call agent is not a chatbot with a speaker. Recognition, latency, interruption and task completion sit on top of the same output evals. The checklist lives in Testing AI call agents. One of the cases below is an AI calling engine Vervali built and then validated.
Bias, Fairness and Explainability
Performance by group, threshold effects and explainable decisions are reported together, so product and risk leaders can approve the trade-off with evidence.
Fairness Measures
Group and subgroup performance on the attributes that matter for the decision: the groups a reviewer or a regulator will ask about, not a generic list. More than one fairness measure, because equalised error on one metric can hide a threshold effect on another. The operating point you will ship is in the test. A measure computed at a threshold nobody uses is not evidence.
The Accuracy Trade-Off
Sometimes the slice that is fairer is less accurate, or the threshold that treats groups alike rejects more of the work you wanted to automate. We put both numbers on the same page. Product and risk sign which side they will live with. A tester does not pick it in private and dress it as a pass.
Explainability Testing
Feature attribution for the decision in front of you, worked examples a reviewer can follow, and a model card that states intended use, limits, and the groups that were tested. If a customer or a regulator asks why this case was flagged, the pack has to answer in language they can use. A saliency plot with no worked example is not that answer.
Independent Evidence, Not Certification
We do not certify fairness or sign that a model is unbiased. We test, report slices and trade-offs, and retest after a threshold or feature changes. Ongoing checks are covered by AI bias and explainability testing.
Drift Monitoring Hand-Off
The validated baseline becomes the production control: what to monitor, which threshold should alert and when investigation or retraining is warranted. Ongoing work is covered by data drift testing and AI and data model monitoring and retraining.
What to Monitor
The inputs that moved the decision in validation, and the outputs that a human still labels. Input drift is visible without waiting for outcomes. Concept drift needs those outcomes. If you only watch incoming features, you will miss the case where the features look the same and the world has changed.
Thresholds, Alerts, Retraining
A threshold is set from the validation baseline, not from a round number. An alert should name what moved, who owns the next look, and whether the action is investigate, re-validate or retrain. Retraining is warranted when the live relationship has changed and a new held-out set can prove it. It is not a reflex every time a chart ticks. US-hours coverage, 9am to 1pm Eastern, is how a product owner sees that alert in their morning.
How We Work with Your ML Team
An endpoint, held-out data and the intended decision are usually enough to begin. Findings return as failed slices, examples and actions a data scientist can use.
What We Need to Start
The model or an endpoint, a held-out data set you control, the intended use, and the decision the model supports. Group labels if fairness is in scope. For an LLM, a sample of real prompts and the rubric. NDA first. Staging or a scored file, not production write access.
What We Do Not Need
Training code and weights, in most cases. A third-party or vendor model is usually a black box: we score what you can call. If the check requires internals, we say so in the design before we quote, rather than discovering it after kickoff. You are not handing over the family silver to get a first report.
How Findings Are Handed Back
A result a data scientist can act on: the slice, the metric, the example, and whether the next move is a threshold, a feature, a prompt or more data. Leadership gets the review-board view: what would fail a large-customer or regulatory ask. The longer method write-up is in AI and ML model validation testing in 2026.
Case Studies
Vervali-built models validated against business outcomes: less KYC review, more qualified leads and real-time SOP evidence.
Financial services · know-your-customer · deepfake detection
Before: manual review was absorbing KYC volume, and a spoofed identity image was the failure that mattered. Vervali built a deepfake detection model for those checks, then validated accuracy, calibration and the review-board evidence around the decision. After: 75% detection accuracy, and 70% less manual review.
75% detection accuracy 70% less manual review Built and validatedSales operations · AI calling engine
Before: qualification depended on human follow-up, and the cycle was long. Vervali built an AI calling engine and validated the model and the output against the conversations that actually produce a lead. After: 35% more qualified leads, and a 28% shorter sales cycle. Voice-specific checks that sit on top of this work are in the call-agent checklist linked above.
35% more qualified leads 28% shorter sales cycle Built and validatedOperations · real-time SOP image detection
Before: standard operating procedure compliance depended on someone looking at the image later. Vervali built a real-time image detection system for that check and validated it against the SOP cases that fail in the line, not only against a clean lab set. After: the compliance check runs as the work happens, with the evidence the operations owner asked for.
Real-time SOP image detection Line cases, not only lab rows Built and validatedFrequently Asked Questions
Vervali checks the model against held-out data for accuracy and calibration, then bias and fairness across groups it should not treat differently, drift since training, whether a decision can be explained, robustness to malformed and adversarial input, and data leakage that would make a sandbox score look better than production. Findings come with the evidence a model review board will ask for. A free review of one model, for one hour, is the first look.
Yes. Most vendor and open-source models are validated as a black box: an endpoint or a scored file, a held-out set you control, the intended use, and the decision the model supports. Training code and weights are not required in most cases. We report how the model behaves on your data and your groups, not how the vendor scored it on theirs.
No. Supervised classifiers and rankers, unsupervised systems, and generative and large language model systems are in scope. The checks change with the type: labelled accuracy and calibration for a classifier, retrieval and generation evals for an LLM. If a model type is outside what we can evidence, we say so in the design before we quote.
Held-out accuracy, precision, recall and calibration, so a 0.9 score means something. Error analysis by slice, not only a global number. Cross-validation where the data volume allows it. Confidence intervals on the metrics a review board will read. The training set is not the proof. If the sandbox score and the held-out score disagree, the held-out score wins.
We build an evaluation set from your own traffic, not a public leaderboard. Hallucination and grounding checks, safety and refusal behaviour, and regression on prompts so a prompt or model change cannot silently alter output. Retrieval quality is tested separately from generation, because most failures are retrieval failures wearing a generation costume. The stored set is what makes the next change visible.
Group and subgroup performance, more than one fairness measure because a single metric can hide a slice, and threshold effects that appear only when you pick an operating point. When the fair choice and the accurate choice differ, we report the trade-off. Vervali does not decide your risk appetite. Method detail sits on AI bias and explainability testing.
Validation is a point in time. Drift is the world moving after that point: inputs that no longer match training, or a relationship between inputs and outcomes that has changed. We say which inputs and outputs to monitor, what threshold should fire, what an alert should trigger, and when retraining is warranted rather than reflexive. Continuous monitoring lives on the data drift and monitoring pages.
Judge a vendor on whether they test on held-out data you control, whether bias and LLM output are in the same engagement as accuracy, whether findings are evidence a review board can read, and whether they are independent of the team that trained the model. Tool vendors sell software. Vervali is a testing laboratory: ISO/IEC 17025 accredited, US-hours coverage, delivery from India, and we will also build the model when that is the job.
Work is scoped to the model, the data and the checks you need, then quoted before we start. Typical work is $25 to $49 per hour. A free review of one model, for one hour, is the first look. A one-off validation is a fixed price. Ongoing drift monitoring is billed separately. Extra groups, evaluation sets and LLM red-teaming are named in the estimate.
Email Us
Call Us
