How Vervali Load Tests Shopify Stores Before Black Friday
How Vervali load tests Shopify and Shopify Plus stores before Black Friday: the Shopify rules, what gets tested, when to start and what it costs.
AI Bias and Explainability
Testing Services
Vervali tests fairness across protected groups, runs explainability checks and produces documented evidence for a model review board. We report the trade-off when the fair choice and the accurate choice differ, and we do not decide your risk appetite. The full model pack is on AI and machine learning testing.
150+ clients
450+ projects
275+ engineers
15+ years
ISO/IEC 17025:2017Accredited testing laboratory
CMMI Maturity Level 3The process is written down and repeats
ISO 9001:2015Quality management
ISO/IEC 27001Information security
Fairness Across Protected Groups, and Evidence a Review Board Can Read
What is tested, and what you receive at the end of it.
Group and Subgroup Performance
Whether the model behaves differently across groups it should not, with results reported for every group and subgroup you name, not only the overall average.
More Than One Fairness Measure
A single metric can hide a slice, so we report several fairness measures side by side and show exactly where they disagree with each other.
Threshold Effects
Some differences appear only at a chosen operating point, so we test fairness at the threshold you actually use in production, not a default cut-off.
Explainability
Whether a decision can be defended to a customer or a regulator, using feature attribution, worked examples and the contents of a model card.
The Trade-Off, Reported
When the fair choice and the accurate choice differ, we report the trade-off in plain terms. We do not decide your risk appetite for you.
Evidence for a Review Board
The slice that failed, the examples, the trade-off and what was not tested, written so a board can read it without a data scientist.
Need fairness and explainability evidence a review board can read?
Free Model ReviewHow Fairness and Explainability Are Tested
Fairness is one slice of model validation. These are the neighbouring packs.
AI and ML Testing
The wider pack: accuracy, calibration, drift, robustness, data leakage and LLM output, with fairness as one slice. See AI and machine learning testing.
Model Validation
Independent validation of the model under review against held-out data you control, with results a board can read. See model validation testing.
Data Drift
Continuous checks on live inputs and outputs, so fairness that held at launch does not quietly fail later. See data drift testing.
Monitoring and Retraining
Ongoing monitoring after launch and retraining when it is warranted rather than reflexive, so the model keeps behaving as it was validated. See AI model monitoring and retraining.
Already know the model and the groups? A QA lead can scope the pack.
Book a CallWhat We Can Test
The checks change with the model type, and we say so in the design if one is outside what we can evidence.
Third-Party and Vendor Models
Checked as a black box: an endpoint or a scored file, a held-out set you control and the groups that matter. Training code is not usually required.
Supervised and Unsupervised
Classifiers, rankers and unsupervised systems each get checks suited to their type, and we say in the design if one is outside what we can evidence.
LLM and Generative Output
Whether the model treats groups differently, and whether a refusal or an answer can be defended. Hallucination and grounding are on AI and machine learning testing.
Proof From Delivery
Three Vervali AI builds, sector named. They show Vervali has built and validated models at that scale and were not fairness-only jobs.
Financial services · KYC deepfake detection
Manual review was the bottleneck on identity. A detection model for know-your-customer cut that load: 70% less manual review at 75% detection accuracy. Whether the model could be defended sat in that pack.
70% less manual review 75% detection accuracyAI calling engine · lead qualification
An AI calling engine, built and validated by Vervali. After: 35% more qualified leads and a 28% shorter sales cycle. The checks were whether the engine actually qualified the lead.
35% more qualified leads 28% shorter sales cycleOperations · real-time SOP image detection
A real-time image detection system for standard operating procedure compliance, built and validated by Vervali. The check was whether the system caught the SOP miss on the floor.
How an Engagement Starts
From one model to evidence a review board can read.
-
01
Share the Model and Groups
Tell us the model, the groups that matter and the decision it supports, with a held-out set you control.
-
02
Get the Fairness Pack
We agree the fairness measures, the operating point and the explainability checks, and what is out of scope.
-
03
Start With a Model Review
A free review of one model, for one hour, is the first look. Most engagements land at $25 to $49 an hour blended, and price follows the groups and checks.
-
04
Get the Evidence
The failed slices, examples, the trade-off and what was not tested, 9am to 1pm Eastern with delivery from India.
Frequently Asked Questions
For fairness and explainability we test group performance, more than one fairness measure, explainability evidence and the trade-off when the fair choice and the accurate choice differ. Accuracy, drift and robustness are on AI and machine learning testing. Findings come with the evidence a review board will ask for.
Yes. Most vendor models are checked as a black box: an endpoint or a scored file, a held-out set you control and the groups that matter for fairness. Training code and weights are not usually required. We report how the model behaves on your data and your groups, not on the vendor's.
No. Supervised classifiers and rankers, unsupervised systems, and generative and large language model systems are in scope for fairness and explainability. The checks change with the type, and if a type is outside what we can evidence we say so before we quote. The full list is on AI and machine learning testing.
We check whether the model treats groups differently and whether a refusal or an answer can be defended. The stored evaluation set, hallucination, grounding and prompt regression are on AI and machine learning testing. A free model review is the first look.
We test group and subgroup performance with more than one fairness measure, because a single metric can hide a slice, and we test threshold effects that appear only at your operating point. When the fair choice and the accurate choice differ, we report the trade-off. Vervali does not decide your risk appetite.
Validation is a point in time, and drift is the world moving after it, including fairness that held at launch and failed later. We say which inputs and outputs to monitor. Continuous monitoring is on data drift testing and AI model monitoring and retraining.
Judge a vendor on whether they test on held-out data you control, whether bias and explainability are in the same engagement as accuracy, whether findings are evidence a review board can read, and whether they are independent of the team that trained the model. Vervali is an ISO/IEC 17025 accredited testing laboratory.
Most engagements land at $25 to $49 an hour blended. Work is scoped to the model, the groups and the checks you need, then quoted before we start. Extra groups and extra evaluation sets are named in the estimate, and a free review of one model shows the surface first.
Email Us
Call Us
