Monitoring Long-Running Processes
We have an application that has several threads that poll queues for processing. Just looking at the pure paths for the application gives very little insight in to what the application is actually doing, since all the pure paths are simply just adding objects to the queues, while most of the heavy processing is done asynchronously.
What would be the recommended way to get visibility into the application in this case? Adding several of the services the processing threads have as APIs still doesn't seem to result in anything showing up as far as time spent interacting with those APIs. In fact, nothing seems to show up there at all.
JP,
It sounds like you have two separate problems. The first problem is that you're not seeing the activity in your background threads, and the second problem is how to correlate the activity in the foreground and background threads.
You'll need to solve the first problem first. You didn't say whether you were running in test or production, and the right path depends a bit on which it is. In either case, you can take a CPU dump to see which methods are being called regularly. This will give better results the more traffic you can run through the background threads. In test, you might consider putting method sensors on entire packages – just to see what you can see – but in general you can't do that inn production.
When you're adding method sensors to catch the background processing, remember to set them to "active and start PurePaths" in the method sensor configuration.
If the background threads are making any database calls, you could set the JDBC / ADO .NET sensor to "active and start PurePaths" as well, just to verify that you can see something in the background threads.
Once you have visibility into both foreground and background threads, the way to correlate them is to construct a PurePath business transaction that splits on some transaction identifier that is shared between foreground and background activity.
-- Graeme
