TIPS FOR SCALING TESTING TO MEET YOUR BUDGET NEEDS – OR SETTING PRIORITIES FOR TEST EXECUTION
A critical aspect of testing is planning how many tests the team executes during the time scheduled. It is important to make the most of the testing time by testing efficiently and thoroughly without testing everything. How can you increase your testing coverage and determine what’s the most important functions of the application to test? Basically, you need to set test cases to a priority based on the impact to the application’s function. In other words, you need to determine what functions are critical to your end users and the application’s function. Plan to start your testing by executing all the critical tests first followed then by the high priority, and finally as many of the low priority tests as possible. You may also consider executing any critical or high priority defects that were fixed during the previous release. The most important factor is to determine the priority for each group of application functions.
Determining Test Priority
One way to determine test priority is by searching through your customer support cases and defects and finding which areas of the application break the most often or are used by customers the most. Define each application function and group common functions together. For each group, determine the priority of each function within the group as well as the group as a whole. Move through the entire application from the front end UI workflows and include all the back-end processing functions that are feasible.
Another method of determining test priority is by determining which test cases fail the most often. Smoke test suites are typically considered critical tests for general functionality. Depending on your application, performance and load tests can be included and prioritized as well.
Planning Test Execution by Priority
Once you have your application grouped and prioritized you need to match up or organize any existing test cases into the same groupings. Now, plan how many tests you have in the critical, high, and low priority categories. Balance that number with the testing time schedule. Determine how many tests can be executed within the scheduled time frame and then determine how to split your test execution. For example, if you have 1500 test cases total, 400 critical, 600 high and 500 low but you only have time to execute 600 tests, determine how you are going to divide them up by priority. Do you need to run all the critical tests? Alternatively, can you run 300 critical and 300 high? If tests are truly critical, then it may be wise to execute all your critical tests and a portion of both your high and low. Defects are likely found in each priority, but it is most important that your customers do not experience critical defects. The other option is to extend the testing time, but that is not always possible. Another option to consider is performing continuous regression testing where suites of test arranged by priority are executed throughout the development and release cycle. In this way, tests are executed where time allows, and the team works through the entire test suite over time. Testing services teams are an excellent choice for continuous regression testing efforts.
Randomizing test suites to increase test coverage
Consider using the testing services team to execute either scheduled or continuous test executions. In order to reduce the monotony of running the same tests repeatedly, plan your test executions by randomizing your test suites. For example, for the first run prior to a release, perhaps have the team execute all the critical test cases. After the first test suite is executed, move to executing the high priority tests followed by the low. Alternatively, mix the priorities together randomly, so the team is testing across priorities at any given time. Test suites can be continuously randomized into various test suites and still incorporate new feature and functionality tests.
Your test suites grow over time, so planning out randomized test suite execution is an optimal plan to get testing completed by the priority mix you choose. A testing service team can be a valuable tool for planning and executing randomized suites of test case executions whether it be continuous or on a timed schedule.
Testing by priority ensures test execution is efficient and practical while still adhering to the release schedule. Test execution by priority also provides increased test coverage without extending the time needed. Granted, you cannot test everything, but you can plan testing effectively by organizing it based on priority.