High inter-tier timings (seen from transaction flow)
Hi all,
We're currently doing tracing on a 3-tier ASP.NET production application. There is a middle tier, and the web tier communicates with it using WCF. We see some interesting bottlenecks in the inter-tier communication, with some requests taking almost 5 minutes (according to transaction flow) between the web and middle tiers. The amount of time the various agents detected as processing time were in milliseconds. A few questions:
- The WCF calls are done over custom serialization, and that doesn't seem to be automatically captured by dynaTrace. Any way to enable that?
- We can't seem to find a way to track the amount of data being sent between the web and middle tiers; any way to do that?
- On the web tier, we have a web server agent and a .NET agent for the w3wp process. We are also seeing very long inter-tier timings between those 2, which is highly surprising as these are within the same machine. We initially thought that it might be that the client sent a huge chunk of data or that the server is sending a huge chunk of data, hence even passing this using a shared memory space is slow, but further analysis showed that the request and response were only about 40kb. Any reason why this might be the case?
Thanks in advance for any help!
jason
Hi Jason
We automatically capture WCF calls but not necessarily show you the time spent in custom serialization. With dynaTrace 4 - and the Auto-Sensors - you should however see into your custom serialization in case there is a performance problem as Auto-Sensors will pick up HotSpots automatically. If you dont see what you expect to see you can always define a custom sensor for your serialization methods.
The amount of data can be captured by the ASP.NET and Web Service Sensor. You have to enable this through the Sensor Properties in your System Profile.
As for the high inter-tier time. Inter-Tier time is the time between the CLient-Side of a WCF call and the Server-Side of the WCF call where Client and Server Side is the instrumented interface done by the dynaTrace WCF Sensor Packs. Between these calls you may have custom serialization or any other modules you have in your IIS that handle things like security, ... - I recommend having a closer look at what else might go on between your Client and Server Tier. Checks what modules you have on IIS.
Hope this helps
Andreas,
I don't see any sensor properties that you can set on the ".NET Web Services" sensor. How do you go about telling it to capture the size of the request & response?
Thanks,
PJ
You are right - I mixed up .NET and Java. For Java Web Services we can capture size information through the Web Service Sensor - we cant do that through our .NET Sensors right now.
We can capture this information using our Web Server Agent which can be installed on IIS and enables User Experience Management - but - I am not sure if you own a Web Server Agent that you could use for this
Andi
Hi Andreas,
Thanks for the reply. I've looked at the methods captured by dynaTrace's auto-sensors and indeed, they don't capture the classes and methods that are used in the custom serializers. Will adding in method sensors help? As far as I know, method sensors only serve to capture arguments and return values (and maybe a few other things), but it doesn't "discover" any method that was not caught by dynaTrace in the first place.
If that is the case, how do I make dynaTrace capture those? At the same time, I also need dynaTrace to capture the performance of 2 HTTP Modules that are implemented on the web front-end (I can't find the methods in the call stack too). Can I configure dynaTrace to do that?
The inter-tier communications (and timings) I was referring to above are actually happening within the same machine, between IIS6 and the W3WP process (the communication channel uses a kernel-mode driver). It seems that something is blocking it, and I'm suspecting that for this case it might be the HTTP Modules.If I am not able to peer into the modules, I won't be able to tell conclusively.
Thanks,
jason
Jason
Auto-Sensors pick up methods that have a performance impact on a single PurePath in case they are called as part of an active PurePath. What does that mean?
a) It can be that a certain method will not show up in 99% of your PurePaths because it has no performance impact but may show up in the 1% where it does.
b) In case you do not have a PurePath for something that is going on in your application, e.g: you dont yet get PurePaths for your background jobs or your custom ASP.NET Modules - AutoSensors will also not pick anything up because they only do this for active PurePaths
Now - in your case it seems you have custom HTTP Modules - I assume these are ASP.NET Modules implemented in .NET and registered as HttpModules in your ASP.NET web.config file - correct? If that is the case you may not see these modules being invoked because they get invoked outside the PurePath that gets started with you ASP.NET Sensor Default Configuration. Try the following: In the ASP.NET Sensor Properties enable "Support for ASP.NET MVC" - it is a checkbox. Then generate some PurePaths and see whether the PurePaths change. They should now start "higher up" the execution chain and should also include your custom HTTP Modules. If that doesnt make any difference let me know - then its possible that we need to find a special entrypoint method for your PurePaths in order to pick up your custom HTTP Module Executions.
As for the Inter-Tier Time: Even if it is on the same machine - your requests have to go through a similar set of processing chain. Its possible that your custom modules may have something to do with this high Inter-Tier time. Once you get to see them in the PPs you should hopefully see how much time they contribute to it.
Andi
P.S.: our r&d labs are closed over xmas and new years - therefore you may not get an immediate response on the forum. Happy New Year
Hi Andreas,
We took your advice and enabled ASP.NET MVC support and indeed it started showing the HTTP modules. At the same time, we looked into the implementation of the modules and specifically placed method sensors on them, and they all started showing up on the purepaths. Apparently what was happening was that one module was occasionally making long calls to an external web service, and that explained the high inter-tier timings.
I also realised that my understanding of the auto-sensors was wrong; I had assumed that they will pick up every method call. It was until I looked at the methods dashlet that I realised that none of the custom code was being detected in the purepaths; then it dawned on me that I had to specifically instrument them.
Thanks for the explanation and suggestion; it has helped me to understand dynaTrace more and how to use it more effectively.
Have a blessed new year!
What exactly does "Enable ASP.NET MVC Support" do? I have a hybrid WebForms and MVC application and it does look like MVC controllers and models aren't being instrumented.
Hi Jason - check out the following post: Tip - Getting Visibility into ASP.NET Modules and Handlers
Hi Jason,
It starts the PurePaths a little bit higher in the call chain, and also traces calls into the HTTP Modules and Handlers.
Andi Grabner wrote a very nice tip, here: https://apmcommunity.compuware.com/community/display/PUB/Tip+-+Getting+Visibility+into+ASP.NET+Modules+and+Handlers
Hope this helps,
Rob
Hi I'm in dynaTrace 5.0 and trying to enable ASP.NET MVC support so I can see HTTP modules. How do I do this? I don't see any "Support for ASP.NET MVC" checkbox.
I am seeing the exact same thing in my purepaths where a lot of time is spent on the Inter-tier communication time between the Web Server Agent and .Net Agent (which are on the same machine!). How can go about diagnosing this?
Hi Melanie
We removed the checkbox "ASP.NET MVC Support" that we used to have as our ASP.NET Sensor is no automatically instrumenting the correct ASP.NET MVC Methods and other Managed Modules that run in IIS
We typically see a lot of time between the Web Server and ASP.NET if you either have
a) native modules that take processing time, e.g: authentication. URL Rewrites, Caching, Compression, ... -> we CANT see into native modules as they are not .NET - thats why the time shows up as Web Server time
b) ASP.NET is out of worker threads and therefore the Web Server needs to wait until the next thread is available - this will also show up as Web Server Time. HEre you can check Windows Performance counters that tell you how many worker threads are currently busy/idle/...
Andi
