Now that your outsourced testing services team is up and testing, what’s the impact? In order to get a realistic understanding of the outsourced testing services value to the business, you need to measure their success. Consider adding one or more metrics over time to ensure you’re getting your money’s worth. In most cases, measuring productivity and cost is part of the team’s yearly budget for operating expenses. When your outsource testing services team performs well, show your pride by using data centered metrics to show the project’s business value.
First, determine if you need additional metrics above those required by your organization. You can pull metric data from all ALM tools, as well as from many task management or test management tools like TFS/MTM, Quality Center, or JIRA, amongst others.
Test data metrics come in many flavors. For example measuring the performance and response of the application with and without load conditions, or how many stories are set to do in an agile release cycle, or the number of test cases executed during a given release cycle. If you’re looking for metrics to prove the outsourced testing services provides business value, you’re going to have to expand into individual productivity metrics. Productivity metrics should be based on the team level and include measurements based on each testing cycle similar to the below:
Keep in mind these numbers may need additional data analysis to be accurate. For example, you’ll need to perform adjustments based on data analysis for the type of tests being executed. It’s much faster to execute smoke tests or functional workflow tests; then it may be to perform the same number of security, integration, or regulatory-based tests. Keep in mind that many functional workflow tests have a higher step count than many other types of tests. If you have automated test execution in your mix, then consider the time it takes to analyze failures and determine if they are application defects. In fact, consider creating separate test execution metrics for automated tests, so the failure analysis time doesn’t adversely affect the test execution numbers.
Another factor that impacts testing schedules is downtime on the QA environment(s). Be sure to calculate a number of tests executed, while noting how often, and how long, the QA testing environment was down, if at all. Included in QA environment factors to consider is test data creation and database response. In other words, the QA team’s test execution numbers may be small if the application database is slow to save or update.
Defects entered is straightforward. The only factor that influences defect numbers is if they are application defects or user errors. If your organization has a change control board or another mechanism for reviewing defects, it may be more accurate to count only “approved” or “accepted” defects. Otherwise, you may end up counting defects that end up being removed or are duplicates of existing defects in the system.
The test code coverage metric refers to measuring what percentage of the code base is covered by test executions. In other words, how much of the code is actually tested by test execution cycles. Again, most development management tools include report metrics for code coverage or total lines of code that are accessed during a defined period of time. With this metric, if your developers have unit tests within in the code, you may need to separate that data from the execution of tests from the test team.
Finally, you’ll need to determine the total cost of the outsourced testing team. Be sure to include internal resources that assist the team with issues and questions or provide task management support. Once you determine the metrics that provide value to your business, you’ll have the data to determine if you’re getting your money’s worth from the outsourced testing team. You’ll also be able to analyze the metric data to improve test coverage by adding more types of testing or randomizing your test execution schedule. Basically, if you don’t measure data, you won’t know how well the team is doing or where and when you need to make adjustments to improve overall performance. Metrics provide answers to business questions using real data from your system. Use them to prove the value of the outsourced testing service team’s contribution.