Right Sourced Testing: Getting Value for Money
Your right sourced testing team is in place but what is their impact? To understand their value to the business, you must measure their efforts. Here we give you some tips on how to do that.
What you should measure
Measure productivity and cost (this is all part of managing the budget for operating expenses anyway). Also consider adding one or more metrics over time. These should be metrics which are of specific importance for your business. This will really demonstrate value and validate the cost.
How you should measure it
Decide upon the key metrics. Then decide upon any additional metrics above those required by your organisation. Next obtain metric data from your ALM tools. You can also get this from task and test management tools. These include TFS/MTM, Quality Centre and JIRA.
Test data metrics are varied. Measure performance and response with load conditions, then measure without. Measure the number of test cases executed during a given release cycle. To truly assess the value of right sourced testing, look at individual productivity metrics. Base these on the team level and include measurements based on:
Number of tests executed
Test code coverage
Keep in mind these numbers may need additional data analysis to be accurate. For example, you’ll need to perform adjustments based on data analysis for the type of tests being executed. Keep in mind that many functional workflow tests have a higher step count than many other types of tests. If you have automated test execution in your mix, then consider the time it takes to analyse failures. Determine whether or not they are application defects. In fact, consider creating separate test execution metrics for automated tests. That way, the failure analysis time will not adversely affect the test execution numbers.
Top Tip for right sourced testing
Be aware of downtime!
Another factor that impacts testing schedules is downtime on the QA environment(s). Be sure to calculate a number of tests executed. Then note how often and how long the QA testing environment was down, if at all. Included in QA environment factors to consider is test data creation and database response. In other words, the QA team’s test execution numbers may be small if the application database is slow to save or update.
Defects entered is straightforward. The only factor that influences defect numbers is if they are application defects or user errors. If your organization has a change control board or another mechanism for reviewing defects, it may be more accurate to count only “approved” or “accepted” defects. Otherwise, you may end up counting defects that end up being removed or are duplicates of existing defects in the system.
The test code coverage metric refers to measuring what percentage of the code base is covered by test executions. In other words, how much of the code is actually tested by test execution cycles. Again, most development management tools include report metrics for code coverage or total lines of code that are accessed during a defined period of time. With this metric, if your developers have unit tests within in the code, you may need to separate that data from the execution of tests from the test team.
Finally, you’ll need to determine the total cost of the outsourced testing team. Be sure to include internal resources that assist the team with issues and questions or provide task management support.
Once you determine the metrics that provide value to your business, you’ll have the data to determine if you’re getting your money’s worth from the outsourced testing team. You’ll also be able to analyze the metric data to improve test coverage by adding more types of testing or randomizing your test execution schedule. Basically, if you don’t measure data, you won’t know how well the team is doing or where and when you need to make adjustments to improve overall performance. Metrics provide answers to business questions using real data from your system. Use them to prove the value of the outsourced testing service team’s contribution.