Before a release that has to stay up for days, a soak test shows what happens to the system after hours of ordinary traffic. It holds a steady, average load for several hours or even days and watches for the problems that only appear with time: memory that leaks, connection pools that run dry, disks that fill and response times that drift upward. A load test that runs for under an hour ends before any of that surfaces.
Endurance testing is the name the International Software Testing Qualifications Board (ISTQB) gives the same test, and its glossary defines it as testing to determine the stability of a system under a significant load over a significant period of time within the system's operational context.
Related reading: Vervali's guide to load testing tools, the head-to-head of JMeter, Gatling and k6, and load testing on a startup budget.
What is a soak test in software testing?
Soak testing, as Grafana's k6 documentation describes it, is a variation of the average-load test that focuses on extended periods and assesses a system's reliability and performance over that time. k6's procedure is to raise the load to an average number of users or throughput, then hold it for considerably longer than any other test type runs. A soak test might also be called an endurance or stamina test, according to Grafana's guide to load test types.
What the test is looking for is spelled out in ISTQB's performance testing syllabus: endurance testing verifies that there are no resource capacity problems, such as memory leaks, database connections or thread pools, that may eventually degrade performance or cause failures.
How is a soak test different from a load, stress or spike test?
Performance tests differ in how much load they apply and for how long. Grafana's k6 documentation groups the common types this way.
| Test Type | Load Level | Duration | Goal |
|---|---|---|---|
| Load | Average production | Mid (5 to 60 minutes) | Check that performance holds at average use |
| Stress | High (above average) | Mid (5 to 60 minutes) | See how the system performs at its limits when load exceeds the expected average |
| Spike | Very high | Short (a few minutes) | See whether the system recovers from a sudden burst of peak load and returns to a steady state |
| Soak | Average | Long (hours) | Check the system under prolonged continuous use, typically after changes |
A soak test and an average-load test use the same load, and Grafana's guide to load test types puts the main difference at duration. Stress tests push load above the average to see how a system performs at its limits, and Vervali runs them under its stress testing service.
What does a soak test find that a short load test misses?
A soak test checks for defects that show only after extended use, and Grafana's k6 documentation names response time degradation, memory or other resource leaks, data saturation and storage depletion among them. Each grows slowly, and a short run ends before any of them crosses a limit anyone would notice. In Grafana's example, a five-minute stress run cannot expose a leak that adds 50 MB an hour, and a ten-minute breakpoint ramp may not reveal a connection pool that loses handles only under sustained traffic.
When response times degrade over a run, the underlying causes ISTQB's syllabus lists include memory leaks, disk fragmentation, network load that increases over time, growth of the file repository and unexpected database growth.
Some failures depend on timers. HikariCP, a Java connection pool, retires pooled connections once they reach their maxLifetime, 30 minutes by default, and removes a connection that is in use only after it is closed, so a run shorter than 30 minutes never sees the pool replace its connections. HikariCP's documentation also asks for maxLifetime to sit several seconds below any connection time limit set by the database or the network in between, and a soak run is the test that exposes a wrong setting. Microsoft Entra access tokens add a second clock: each is assigned a random default lifetime ranging between 60 and 90 minutes, and when both the client and the resource support Continuous Access Evaluation, the token lifetime may be automatically extended to 24 to 28 hours. A test that stops at twenty minutes never triggers the token refresh path or evaluates the system's response to expired sessions.
Leaked file handles and threads run into limits the operating system enforces. On Linux, open(2) fails with EMFILE once a process reaches its per-process limit on open file descriptors, and pthread_create(3) fails with EAGAIN when a system-imposed limit on the number of threads is reached. A process that leaks a file handle or a thread on some requests reaches those limits only after enough hours of leaking.
How long should a soak test run?
ISTQB ties the duration to the system's operational context, so the right length differs from one system to the next. Vervali's advice is to pick a run long enough to pass the longest periodic event in the system at least once, whether that is a connection lifetime, a token lifetime, a scheduled job or log rotation. Then keep it running long enough to see whether the memory left after a full garbage collection keeps climbing. With HikariCP's 30-minute default and a 60 to 90 minute token, a run of several hours crosses each of them more than once, and a release that has to survive a daily batch job needs a run that covers one.
For comparison, Grafana's k6 documentation on soak testing lists typical durations of 3, 4, 8, 12, 24, and 48 to 72 hours. Production soaks often run 2 to 8 hours or longer, according to Grafana's soak testing learning path.
How do you run a soak test, step by step?
Start with smoke tests, then move to higher loads and longer durations. k6 advises against running a soak test before the smoke and average-load tests have run, because each test uncovers different problems and running a soak first may cause confusion and resource waste.
Reuse the average-load script where you can and change only the hold duration, with the plateau set to the virtual-user count from the baseline or average-load test. The environment and data need to look like production: ISTQB's syllabus calls realistic environments, data and loads critical for accurate results at system level, and a test database a fraction of the production size can hide the slow queries that data growth produces. Take a baseline before the run as well: the syllabus warns that results cannot be judged without a baseline set of measures for acceptable performance.
Backend monitoring should be running before the load starts; k6's documentation calls it especially important for soak tests. Decide the pass and fail criteria before the run too, which in k6 means thresholds. The load-profile decisions to settle in writing before any performance run, meaning concurrent users, ramp, duration and think time, are listed in Vervali's software testing checklist. On a multi-hour soak, Grafana suggests delaying abort evaluation by 10 minutes or more, so the warm-up period, when caches fill and connection pools open, does not trip an early abort.
In k6's own soak example, the test ramps to 100 virtual users over 5 minutes, holds 100 users for 8 hours, then ramps down over 5 minutes:
export const options = {
stages: [
{ duration: '5m', target: 100 },
{ duration: '8h', target: 100 },
{ duration: '5m', target: 0 },
],
};
To adapt k6's example, replace 100 with your own average load and 8h with the duration your system's clocks call for.
Which metrics decide whether a soak test passed?
Oracle's guide to troubleshooting memory leaks says to watch the live set, the Java heap or Metaspace still in use after a full garbage collection. If the live set keeps rising after the application has reached a stable state under a stable load, Oracle calls that a strong indication of a memory leak.
| Metric | Steady run | Warning sign | Where to read it |
|---|---|---|---|
| Memory | Live set holds level after each full collection | Live set keeps rising; latency climbs with it | GC logs, jvm.gc.live.data.size, process_resident_memory_bytes |
| Connection pool | Few or no threads waiting | Latency stable, then a jump with timeout or connection-refused errors | HikariCP pool.Wait and pool.PendingConnections |
| Disk | Log and temp space reclaimed | Performance changes as disk space or log volume nears its limit | Disk usage on every host |
| Container restarts | Zero | Restarts, last state OOMKilled | kube_pod_container_status_restarts_total |
| File descriptors | Count holds level | Count keeps rising | process_open_fds |
| Throughput | Holds level at a constant load | Falls while virtual users stay constant | The load tool's request rate |
| Errors | None, or a steady background rate | Absent early, present later | The load tool's error rate |
On a Java service, garbage collection logs carry the same memory picture; Oracle documents -Xlog:gc for logging collections. Watch memory for each process as well as the total for each server.
How do you read soak test results?
Grafana's advice is to record what changed, when it started and how fast it progressed, so the next run can be compared with this one.
Grafana's soak testing guide sorts degradation into categories, each with a follow-up. Memory-related degradation shows as latency rising gradually while application memory or GC metrics climb on the system under test. The follow-up is to review application-level caching and session storage for unbounded growth, and to check for long-lived collections that accumulate per request, such as logging buffers, metrics registries or listener lists. Oracle adds that heap dumps are the most important data to troubleshoot memory leaks.
With connection-related degradation, latency stays stable and then jumps when the pool is exhausted, and the errors look like timeouts or connection refused. The guide's advice there is to compare the pool maximum to the sustained request rate. When a HikariCP pool reaches its maximum size and no idle connections are available, calls to get a connection block for up to the configured connection timeout before timing out.
Disk-related degradation is performance that changes when disk space or log volume approaches limits; for that, the guide says to review log rotation and temp-file cleanup and to set disk usage alerts from the fill rate you observed.
An upward trend matters even when the 95th percentile stays below the threshold. In Grafana's example, a p95 that climbs from 200 ms to 490 ms over two hours without crossing a 500 ms threshold is still a warning sign. Falling throughput at a constant number of virtual users means each request is taking longer to complete, and errors that are absent early in the run and appear later may indicate resource exhaustion. Grafana suggests recording response time, failure rate and request rate at a baseline after warm-up, at the start of the degradation and at the end of the test.
When every metric stays level for the whole run, keep the result as the reference for the next code or infrastructure change, and extend the hold next time; Grafana's example goes from a 30-minute plateau to 4 or 8 hours on a production soak.
Which tools can run a soak test, and what changes for a long run?
A multi-hour run tests the load generator as well. A tool set up for a ten-minute test can run out of memory or disk before the system under test shows anything, and results gathered while the generator is short of CPU or memory have to be thrown away.
Apache JMeter
The current release listed on the download page is Apache JMeter 5.6.3. Follow the JMeter manual's best practices for load tests: run in CLI mode, write results to a CSV file and use as few listeners as possible, keeping View Results Tree and View Results in Table for scripting and out of the run. To watch a multi-hour run as it happens, send results through the Backend Listener; JMeter ships an InfluxDB client for it, and the manual notes that InfluxDB data can be viewed in a browser through Grafana.
Give JMeter more heap before a long run: it runs with a 1 GB heap by default, and the manual says that might not be enough for your test. If the plan uses BeanShell elements, the manual also notes that some long-running tests may cause the interpreter to use lots of memory, and suggests the reset option.
Set the run length in the Thread Group's scheduler: with a Duration set, JMeter stops the thread group when the duration or the loop count runs out, whichever comes first, and a Loop Count of infinite keeps the test running until it is stopped by hand or the thread lifetime ends. To hold a rate, add a Constant Throughput Timer, which introduces variable pauses calculated to keep total throughput, in samples per minute, as close as possible to the figure you set; throughput will be lower if the server is not capable of handling it. For a schedule in stages, the Ultimate Thread Group, published by the JMeter-Plugins.org project, offers an infinite number of schedule records, each with its own ramp-up time, shutdown time and flight time.
Test data has to last the whole run. By default, JMeter's CSV Data Set Config starts the file again from the top when it reaches the end, so on a multi-hour run the same users and records come round again; check whether reused records cause errors of their own. With recycling and stopThread both off, the variables are set to <EOF> once the file runs out.
For large-scale load testing, the manual suggests running multiple CLI JMeter instances on multiple machines using distributed mode. Remote mode uses more resources than running the same number of CLI mode tests independently, and with many server instances the client JMeter can become overloaded, as can its network connection.
k6
The generator needs CPU and memory to spare. According to k6's documentation on running large tests, the machine should be sized for at least 20 percent idle cycles, because at 100 percent CPU the test hits throttling limits and the result metrics can show a much larger response time than in reality. Memory swapping on the load generator will likely invalidate the results, since the generator then performs differently in different parts of the test. By default k6 also loads the response body of each request into memory, and discardResponseBodies avoids that.
The choice of executor decides whether the load stays constant while the system slows. Under constant-vus, a fixed number of VUs execute as many iterations as possible for a specified amount of time. In a closed model, when the target system is stressed and starts to respond more slowly, the test waits: iteration durations increase and the arrival rate of new VU iterations tapers off. The constant-arrival-rate executor starts a fixed number of iterations over a specified period of time, and as an open-model executor it starts them independently of system response. For a soak test that should hold the arrival rate steady, the arrival-rate executor keeps the load where you set it as response times grow.
Gatling
Gatling makes the same choice in its injection profile. For an open model, constantUsersPerSec(rate).during(duration) injects users at a constant rate for a given duration. constantConcurrentUsers(nbUsers).during(duration) is the closed-model form and keeps the number of concurrent users in the system constant.
Locust
Headless Locust takes its run time from -t or --run-time, for example 1h30m.
LoadRunner
In LoadRunner, the Duration action keeps the Controller running the scenario in its current state for the amount of time you specify.
Can you run soak tests in a CI/CD pipeline or on Kubernetes?
Long soaks fit a nightly or pre-release schedule on a dedicated runner. Run as a GitHub Actions job, an 8-hour soak needs a self-hosted runner: GitHub-hosted runners stop a job at 6 hours of execution and fail it, and self-hosted runners allow up to 5 days per job.
On Kubernetes the kernel enforces memory limits with out-of-memory kills, and only when it detects memory pressure, so a container can run over its limit for a while before it is killed. The kubelet then restarts it, with a back-off capped at 300 seconds, and the killed container's last state shows reason OOMKilled with exit code 137. A restart frees the memory the leaking process held, so the leak can vanish from a memory graph while it keeps happening. Count restarts during the soak with kube-state-metrics' kube_pod_container_status_restarts_total.
Which systems need soak testing most?
A soak test earns its hours on systems that run for days between deploys: back ends that hold pooled database connections, services that rely on expiring tokens, platforms with nightly batch or scheduled jobs, and anything that writes heavy logs. Many e-commerce back ends and SaaS platforms with many concurrent users match that profile.
In October 2012, a reporting agent on AWS's EBS storage servers in US-East lost contact with a data collection server, and a latent memory leak bug in the agent began to slowly consume memory as it kept retrying, according to AWS's summary of the service event. AWS's monitoring watched aggregate memory on each server and did not alarm on the leak. By Monday morning the leak had consumed enough memory that the affected servers could not keep up with normal request handling.
An anonymised Vervali engagement: what performance engineering changed
Vervali's performance engineering work for a UAE SME finance-management platform, one that handles collections, receivables and financial tracking, used JMeter for load injection, Grafana for observation and PostgreSQL tuning for the database layer. The engagement ran load and stress testing, and reliability was validated under concurrent load. According to the results the client reported, payment-transaction response times at peak improved by 55 percent after the tuning and API latency fell by 65 percent. A soak test runs on the same tool chain: a load generator, a dashboard over the whole run and a database the team can tune when the graphs point at it. The work sits under Vervali's performance testing service, and more engagements are on the case studies page.
When is a soak test the wrong test to run?
A soak test is the wrong test while the build still has functional defects. ISTQB's syllabus notes that generating a performance test load depends on the absence of functional defects in the system under test. A soak run that fails on a broken workflow in its first hour has to be thrown away.
How many users the system can carry is a capacity question, and ISTQB defines capacity testing as determining how many users or transactions a system will support while still meeting its performance objectives. Stress testing and scalability testing answer that question.
Some defects take longer to appear than any run a team can afford. The FAA's 2015 directive on the Boeing 787 describes a software counter in the generator control units that overflows after 248 days of continuous power, an issue Boeing identified during laboratory testing. For a defect on that time scale, review every counter and timer in the code for overflow, and run tests that move the clock forward.
A soak test also needs an environment that stays up for the whole run: a shared environment that is redeployed or reset during the day will interrupt a multi-hour soak, and an interrupted run has to start again.
Vervali's load testing team works with JMeter, Gatling, k6, LoadRunner and Locust and can embed performance tests in Jenkins, GitHub Actions or GitLab CI pipelines. To plan a long run before your next release, get in touch through the contact page.
Email Us
Call Us
