When I first started in the world of test automation, sanity tests were run manually on local machines. It seemed like a quick way to get feedback, and the test suites were small enough that running them on a developer’s machine wasn’t a huge issue. But as our product evolved, and so did the test suite, this approach quickly became unsustainable.
Sanity testing, which was meant to give us confidence in basic functionality before moving into deeper testing, was becoming a bottleneck. As the tests grew in number, running them on local machines was time-consuming, error-prone, and unreliable. If a developer was tied up or a machine was under heavy load, the tests could fail simply due to environment issues.
That’s when we made the decision to automate daily sanity testing in the cloud. The goal was simple: increase efficiency, minimize downtime, and ensure that we could continuously validate basic functionality without manual intervention. What started as a challenging transition soon became one of the most impactful decisions in our automation journey. Let me share how we made this shift and the lessons we learned along the way.
The Challenges with Running Tests Locally
When I look back at running tests locally, the challenges were abundant and often frustrating:
1. Inconsistent Environments
Each developer’s machine had a slightly different setup. Dependencies like browser versions, OS configurations, and the presence of background processes often led to test failures that weren’t caused by the application.
2. Limited Machine Resources
Running sanity tests locally was feasible with a small test suite, but as our test cases expanded, developers’ machines struggled to handle the load. This led to slow test execution, and sometimes machines even crashed mid-run.
3. Human Error
Manual execution of sanity tests often relied on individual developers to trigger the runs at the right time. Human error crept in, like starting tests late, missing key test cases, or skipping them entirely when under tight deadlines.
4. Lack of Parallelism
Running tests on a single machine meant tests had to execute sequentially. This dragged down the overall speed of feedback. Waiting hours for the sanity tests to finish before developers could begin deeper testing was inefficient.
5. Environment Drift
Tests run locally often had issues with mismatched environments. A test could pass on one developer’s machine but fail on another, or worse, fail on the CI server due to different configurations or dependencies.
We knew we needed to automate sanity testing, but we also needed a way to ensure that it was consistent, fast, and scalable.
Other Recommended Reads: api automation interview questions
The Shift: Moving to Cloud Execution
The solution we adopted was cloud-based test execution, which allowed us to run our sanity tests on consistent, scalable infrastructure. Here’s how we made the transition:
1. Selecting the Right Cloud Platform
We needed a platform that could handle the scale of our tests while providing flexibility in browser and OS configurations. After evaluating several options, we decided to go with Azure DevOps integrated with Azure Pipelines. This offered powerful cloud-based test execution with integrations into our CI/CD pipeline, as well as support for parallel test execution.
2. Setting Up a Cloud-based Grid
Instead of running tests on a single machine, we moved to a Selenium Grid setup in the cloud. This allowed us to run multiple tests in parallel across various combinations of browsers, operating systems, and devices. Setting up the grid in Azure was relatively easy, and we could scale it up or down as needed without worrying about resource constraints.
3. Integrating with CI/CD Pipelines
With Azure Pipelines, we set up automated jobs to trigger sanity tests every day at the same time, first thing in the morning. No more waiting on individual machines. The tests were kicked off automatically whenever a new build was pushed. This integration meant that sanity testing was no longer something we did manually but a seamless part of our CI/CD process.
4. Optimizing Test Execution with Parallelism
By leveraging the power of the cloud, we could run sanity tests in parallel across different virtual machines. This drastically reduced the execution time of our test suite, and we could now get immediate feedback from multiple browsers and OS configurations, all within minutes.
5. Managing Test Data and Dependencies
One challenge we encountered in the cloud was managing the test data and ensuring that the environments were clean and consistent. We solved this by setting up containerized test environments that were created from scratch for each run. This approach ensured that the environment was always consistent, and we could control dependencies more effectively.
Additional Resources: automation testing interview questions
The Benefits We Gained
After the transition to cloud-based test execution, the impact was immediate:
1. Speed and Efficiency
Tests that once took several hours to execute now ran in minutes. Parallel execution meant we could test different browser versions simultaneously. This saved precious time for the development team, who could start deeper testing much sooner.
2. Consistency Across Environments
Running sanity tests in the cloud meant that the environment was standardized. We eliminated the inconsistencies that arose from local machines. Now, the tests ran in the same environment every time, with the same configurations, ensuring reliable results.
3. Scalability
As our test suite grew, so did the need for increased resources. The cloud allowed us to scale our infrastructure easily. If we needed to run tests on new browsers, devices, or configurations, we could add them to our cloud-based grid without any additional hardware or setup.
4. Reduced Human Error
Automating sanity tests through the cloud removed the need for manual intervention. Tests ran without fail, exactly when they were supposed to. Developers no longer had to remember to start tests manually or worry about missing test cases. The process was automated end-to-end.
5. Improved Developer Confidence
With daily sanity tests automatically triggered in the cloud, developers gained confidence in the basic functionality of the application right from the start of the day. This early feedback allowed them to catch critical issues before diving into more complex testing.
Don’t Miss Out: Epam interview questions
Lessons Learned
While the transition to cloud-based test execution brought significant benefits, there were several lessons we learned along the way:
1. Cloud Resources Are Not Infinite
While scaling in the cloud is easy, it’s important to monitor resource usage. We initially faced some unexpected cost overruns when we didn’t manage parallel execution efficiently. It’s important to balance the speed of execution with budget constraints.
2. Build a Robust Test Suite
A robust set of sanity tests is essential for cloud execution. Inconsistent or unreliable tests can cause your automated cloud runs to fail, which can make it harder to trust the system. We spent time refining our tests to ensure they were stable and effective for automation.
3. Use Test Result Analytics
Cloud-based test execution gives you access to rich test result data, including logs, screenshots, and videos. We implemented a system to analyze failures automatically, which helped us identify flaky tests or environmental issues faster.
4. Collaborate with DevOps Early
Moving to the cloud requires collaboration with the DevOps team to configure the infrastructure properly. Ensuring the cloud test grid and pipeline integration are seamless is key.
From Chaos to Confidence
Automating our daily sanity tests in the cloud was a pivotal moment in our automation journey. It not only improved the efficiency and consistency of our testing but also allowed us to scale as our product grew.
What began as a cumbersome, local, and manual process is now a fully automated, reliable, and scalable part of our CI/CD pipeline. We gained speed, confidence, and peace of mind, knowing that basic functionality was tested every day without fail, and without relying on individual developer machines.
For any team still running sanity tests manually on local machines, my advice is simple: take the leap to the cloud. It may take time to set up, but the long-term benefits in terms of speed, scalability, and consistency will pay off immensely.
FAQs
1. What is sanity testing in automation?
Sanity testing ensures that key functionalities of an application work as expected before deeper testing begins, saving time and reducing risk.
2. Why move sanity tests to the cloud?
Cloud execution offers speed, consistency, scalability, and parallel testing — helping teams eliminate environment issues and human dependency.
3. Which tools are best for cloud-based test execution?
Tools like Azure DevOps, Selenium Grid, and Jenkins integrate easily with CI/CD pipelines to enable automated cloud testing.
4. How does parallel execution improve testing efficiency?
Parallelism allows multiple tests to run simultaneously, reducing execution time from hours to minutes and accelerating developer feedback.
5. What are key challenges in cloud automation?
Managing cloud resources, maintaining clean environments, and ensuring test stability are crucial for cost-efficient, reliable automation.
We Also Provide Training In:
- Advanced Selenium Training
- Playwright Training
- Gen AI Training
- AWS Training
- REST API Training
- Full Stack Training
- Appium Training
- DevOps Training
- JMeter Performance Training
Author’s Bio:
Content Writer at Testleaf, specializing in SEO-driven content for test automation, software development, and cybersecurity. I turn complex technical topics into clear, engaging stories that educate, inspire, and drive digital transformation.
Ezhirkadhir Raja
Content Writer – Testleaf
