I am trying to better analyze my test data but found averages sometimes can hide a lot of valuable information. Would percentiles working better here? And how to use percentiles?
Answer by Nyna W. ·
Thank you for the question. As you mentioned dealing with averages in non-trivial data sets can hide a lot of the valuable information. If you would like to dig a bit deeper into this topic I would recommend the APMBlog post “Why Averages Suck and Percentiles are Great” - http://apmblog.compuware.com/2012/11/14/why-averages-suck-and-percentiles-are-great/.
Since October 2013 release you can use various percentiles to analyze your performance in New Portal (portal.dynatrace.com).
So I would highly recommend you to take a closer look at you data – especially the Last Mile data – by using the typical aggregations of 25th percentile, median and 90th percentiles.
For example the 90th percentile of the below charted test would be a great indicator why a lot of the users might be lost on the homepage – having to wait more than 13s is probably very frustrating for many.
When you do chart the percentiles however make sure you do aggregate enough values into each point. Typically using time frames of 1 day are a good. Hourly aggregations with say just 20 or 30 data points would be displaying some large fluctuations especially in the higher percentiles.