.net collections - performance problems - auto sensors?
some good and bad news.. DT 4.1 helped quickly identify a performance issue in our code.. we were using System.Collections.Arraylists, and then doing an Arraylist.IndexOf()
Arraylist.IndexOf():
The ArrayList <http://msdn.microsoft.com/en-us/library/system.collections.arraylist(v=vs.100).aspx> is searched forward starting at the first element and ending at the last element.
This method performs a linear search; therefore, this method is an O
operation, where n is Count
what is curious is i had to scan my source code to find the issue.. ie my MethodA() uses ArrayList.IndexOf.. DT only reported long execution time and high-cpu on this method.. but didn't further explode the root cause.
I was hoping the auto sensor would have shown System.Collections.Arraylist.IndexOf being responsible.. why didnt it?
Hi
I assume that the individual calls to IndexOf didnt take as long as the code executed in your MethodA handling the individual array elements. Auto Sensors do not look at every single method execution. Auto Sensors will pick up methods that have a high execution time. If your IndexOf method is called very often but an individual execution is very fast it will most likely not identify it as a problem but rather identify the calling method.
You can learn more on how Auto Sensors work by watching our recent dynaLearn Webinar on Sensors: dynaLearn Webinar - Advanced Sensor Rules - September 5, 2012
Its good to hear that you found the problem in the end. Also - it is great input for us to hear about a use case like yours. Helps us improve the product to automatically detect problem patterns like the one you described
Andi
Hi Andi,
the individual calls to ArrayList.indexOf() was responsible for 99% of the execution time in MethodA().. as soon as the code was corrected to remove the ArrayList.IndexOf() call, MethodA() now runs under 10ms.. before it would take over a second depending on the size of the Arraylist.. as the documentation explain, IndexOf uses a linear search which is very expensive.
Hence the confusion as to why ArrayList.IndexOf wasn't auto-profiled.
Actually thinking about it some more.. MethodA() was called by another method, say ProcessCode(). While i had ProcessCode() explicitly instrumented in a sensor package, MethodA() was NOT being shown in the purepath even though it was responsible for 99% of the execution time.. i had to explicitly instrument Method() to determine root cause.. which still didn't reveal the .net collection to be responsible.
Hence the concern / confusion that auto-sensors don't seem to work.
The weird behavior now is that since the time we have fixed MethodA().. it doesn't show up in the purepath anymore, even though it is still part of the sensor rules.. i dont see it in the "Methods" dashlet either. Exploring the configured sensors on the agent shows that the method is no longer instrumented.. sometimes when i restart the process, it gets instrumented.. most times, it does not.. very weird behavior.
Definitely looks like a product bug..
That is definitely interesting. Here are some thoughts
a) In case you use dynaTrace 4.2: in the PurePath Dashlet click on "Show all Nodes" - see if that gives you additional visibility. By default we only show "Relevant" nodes, e.g: we hide deep stacks and only show "leave" methods that have a performance significance
b) As for the Sensors that don't work: Have you checked the Deployed Sensor Tab in the Agent Overview? This will show you which methods are actually instrumented. It also shows you which Sensor Group/Pack defines the Rule.
As you explain a very "volatile" behavior I suggest you open a support ticket. When looking at your sessions we will be able to find the root cause of this. It might be a product bug - or hopefully something that can easily be explained
Thanks
Andi
