You also need to ensure your tests measure all relevant aspects of risk and quality for your applications. You will probably need multiple tools to achieve this.
Your first set of tests should verify the most important aspects of the system. Make sure key use cases still work when a new release goes out. Capture as much information as you need when you run your tests. If you have enough information about what was going on in the app to determine what the problem is, you will spend less time trying to reproduce test failures.
Then build out test coverage based on the desired risk and quality information - not necessarily by "test category." More tests are not always better. Clearly, there is a tradeoff between getting more information and increasing runtime cost, maintenance cost, complexity of the overall picture, and so on.
Choose tests according to the context. For example, run more tests in the areas of the system that are being changed in the current pipeline run. And you want to actively manage your test set. Strive to minimize the signal-to-noise ratio of the test output — a test that never fails can be as wasteful as a test that never succeeds.
What Do I Need To Do Now?
When you are getting started, the first thing to do is critically examine existing tests to see how long they take to run and if they are compatible with the targets you have set for your delivery pipeline.
Then, determine the most critical use cases for your application and ask the following: What is the likelihood of breaking this use case? How bad is it if this use case no longer works? How quickly can we fix it if it breaks? Do we have tests to cover this use case? Manual or automated? How many of them do I need to run to be confident enough that the use case still works well?
If you are missing tests, add time and budget to add these to your pipeline initiative. You may also decide to invest in better mechanisms to fix things quickly. Choose tests based on use cases rather than test categories — and ensure that you capture information about the systems under test as the test runs. Also, it's smart to ensure your tests are linked to the use cases and parts of the system they cover - otherwise, it's hard to choose the most appropriate test set.
Continuous Delivery is all the rage. Its benefits are easy to grasp, but doing it properly hinges on the maturity of your testing. A new version of an application should be rigorously tested to ensure that it meets all desired system qualities. It is important that all relevant aspects — whether functionality, security, performance or compliance — are verified by the pipeline. If in doubt, test and test again.
Sign up for MIS Asia eNewsletters.