Continuous Testing
Continuous testing is the process of performing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. Continuous testing was introduced as a way to reduce the wait time for feedback to developers.
For continuous testing, the scope of testing ranges from validating low-level requirements or user stories to evaluating system requirements associated with larger business goals.
History of Continuous Testing
In 2010, the software has become a key business differentiator. As a result, organizations now expect software development teams to deliver more and more innovative software within shorter delivery cycles. To meet these demands, systems development teams have turned to learning methodologies such as Agile, DevOps, and Continuous Delivery to try to accelerate the lifecycle. After accelerating other aspects of the delivery pipeline, teams typically appear to that their testing process is preventing them from achieving the expected benefits of their SDLC acceleration initiative. Testing and the overall quality process remain problematic for several important reasons.
- Traditional testing processes are too slow. With the growing popularity of Agile, DevOps, and Continuous Delivery, iteration lengths have changed from months to weeks or days. Traditional testing methods, which rely heavily on manual testing and automated GUI tests that require frequent updates, cannot keep pace.
- Even as more automation is added to existing test processes, managers lack adequate insight into the level of risk associated with an application at any given point in time. Understanding these risks is critical to making rapid go/no-go decisions involved in continuous delivery processes. If tests are developed without understanding what the business considers an acceptable level of risk, So it is possible to have a release candidate that passes all available tests, but that business leaders do not consider ready for release. The test results are used to accurately indicate whether each release candidate meets business expectations, the test design approach should be based on the business’s tolerance for security, performance, reliability, and compliance risks. In addition to conducting unit tests that test code from the very granular bottom-up level, a broad set of tests is required to provide a top-down assessment of the business risk of a release candidate.
- Even if testing is automated and tests effectively measure the level of business risk, teams without an integrated end-to-end quality process will have trouble meeting business expectations within today’s compressed delivery cycle. Efforts to eliminate risks at the end of each iteration have been shown to be significantly slower and more resource-intensive than building quality into a product through defect prevention strategies such as development testing.
Organizations adopt continuous testing because they recognize that these issues are preventing them from delivering quality software at the desired speed. They recognize the increasing importance of software as well as the increasing cost of software failure, and they are no longer willing to make trade-offs between time, scope, and quality.
Objectives and Benefits of Continuous Testing
The purpose of continuous testing is to provide rapid and continuous feedback regarding the level of business risk in the latest build or release candidate. This information can be used to determine whether the software is ready for development through the Anytime Delivery pipeline.
Because testing begins early and is performed continuously, application risks are exposed immediately after introduction. Development teams can then prevent these problems from escalating to the next phase of the SDLC. This reduces the time and effort that needs to be spent on finding and fixing defects. As a result, it is possible to increase the speed and frequency at which quality software is delivered while also reducing technical debt.
Additionally, when software quality efforts and testing are aligned with business expectations, test execution creates a prioritized list of actionable tasks. This helps teams focus their efforts on quality tasks that will have the greatest impact based on their organization’s goals and priorities.
Additionally, when teams are continuously executing a broad set of tests throughout the SDLC, they collect metrics regarding process quality as well as the state of the software. The resulting metrics can be used to re-evaluate and improve the process itself, including the effectiveness of those tests. This information can be used to establish a feedback loop that helps teams incrementally improve the process. Frequent measurements, tight feedback loops, and continuous improvement are key to DevOps.
Scope of Continuous Testing
Continuous testing includes the validation of both functional requirements and non-functional requirements.
For testing functional requirements, continuous testing often includes unit testing, API testing, integration testing, and system testing. For testing non-functional requirements – to determine whether the application meets expectations about performance, security, compliance, etc, including static code analysis, and security testing, performance Tests should be designed to provide early detection of risks that are most critical to the business or organization releasing the software are important.
Teams often find that the test suite can run continuously and effectively assess risk levels, requiring a shift in focus from GUI testing to API testing because APIs is considered to be the most stable interface. for the system under test and GUI tests require considerable rework to keep pace with the changes typical of a rapid release process. API layer tests are less brittle and easier to maintain.
Tests are performed during or alongside continuous integration at least daily. For teams practicing continuous delivery, tests are typically performed several times a day, each time the application is versioned is updated in the control system.
Ideally, all tests are performed in all non-production test environments. To ensure accuracy and consistency, testing should be performed in the most thorough, production-like environment. Strategies to increase test environment stability include software virtualization and test data management.
The general behavior of Continuous Testing
Testing should be a collaboration of development, QA, and operations aligned with line-of-business priorities within an integrated, end-to-end quality process.
Tests should be logically partial, incremental, and repeatable. The results should be conclusive and meaningful.
All tests need to be run at some point in the build pipeline, but not all tests need to be run all the time because some tests are more resource intensive than others.
Eliminate test data and environmental constraints so that tests can run continuously and continuously in a production-like environment.
To reduce false positives, minimize test maintenance, and more effectively validate use cases with multi-tier architectures in modern systems, teams should emphasize API testing over GUI testing.
Challenges of Continuous Testing
Because modern applications are highly distributed, the test suites that use them typically require access to dependencies that are not readily available for testing (for example, third-party services, mainframes that are only available for testing in a limited capacity or at inconvenient times, etc.).
Additionally, with the adoption of agile and parallel development processes, it is common for end-to-end functional tests to require access to dependencies that are still being developed or not yet implemented. This problem can be solved by simulating the application under test interactions with missing or unavailable dependencies using service virtualization. It can also be used to ensure that data, performance, and behavior remain consistent across different test runs.
One reason teams avoid continuous testing is that their infrastructure is not scalable enough to run the test suite continuously. This problem can be solved by focusing tests on business priorities, dividing the test base, and parallelizing testing with application release automation tools.
Continuous testing vs automated testing
The purpose of continuous testing is to apply “high automation” in a stable, production-like test environment. Automation is essential for continuous testing. But automated testing is not the same as continuous testing.
Automated testing involves the automated, CI-driven execution of whatever tests the team has collected. , and regularly execute these tests in the context of a stable, production-like test environment. Some differences between automated and continuous testing:
- With automated testing, test failures can indicate anything from a major problem to a nominal quality violation. With continuous testing, test failures always indicate a significant business risk.
- With continuous testing, a test failure is addressed through a clear workflow to prioritize defects versus business risks and address the most critical risks first.
- With continuous testing, every time a vulnerability is identified, there is a process of uncovering all such defects that may have already been introduced, as well as potential future recurrences of the same problem.
Continuous testing tools
Research firms Forrester Research and Gartner made continuous testing a central consideration in their annual evaluation of test automation tools. Gartner published an updated version of the research in 2019.
Gartner reviewed 10 tools that met their criteria for enterprise-grade test automation tools. The evaluation included inquiries from Gartner clients, surveys of tool users, vendor responses to Gartner questions, and vendor product demonstrations. Gartner requires native Windows desktop application testing and Android or iOS testing support, as well as tools to support 3 of the following: Responsive Web Applications, Mobile Applications, Packaged Applications, and API/Web Services. Here are the results of the 2019 Magic Quadrant Research:
- Leaders: Eggplant, SmartBeer Software, Tricents
- Challengers: IBM, Microsoft
- Visionary: Broadcom, Parasoft
- Niche players: FrogLogic, Renorex, Worksoft
In 2020, Forrester Research reviewed 15 tools that met the criteria for enterprise-grade test functional automation tools.[30] Forrester determined 26 criteria based on past research, user needs, and expert interviews, then Evaluated products against products based on vendor responses to Forrester questions, vendor product demonstrations, and customer interviews. Forrester required tools for cross-browser, mobile, UI, and API testing capabilities. Here are the results for the 2020 Forrester wave:
- Leaders: ACCELQ, Eggplant, Parasoft, Tricentis
- Strong Players: Broadcom, IBM, Mabel, Microsoft, Perforce, SousLabs, SmartBear Software
- Contenders: Cyara, Expiretest, Worksoft
- Challengers: Rinorex