• Forums
    • Public Forums
      • Community Connect
      • Dynatrace
        • Dynatrace Open Q&A
      • Application Monitoring & UEM
        • AppMon & UEM Open Q&A
      • Network Application Monitoring
        • NAM Open Q&A
        • Enterprise Synthetic Monitoring
      • Synthetic Classic
        • Synthetic Classic Open Q&A
  • Home /
  • Public Forums /
  • Dynatrace /
  • Dynatrace Open Q&A /
avatar image
Question by Larry R. · Jun 25 at 11:31 AM · dynatrace saas monitoring usability technologies google cloud platform

Lets be clear when it comes to messaging queues & Dynatrace - What does the future hold?

Just to be clear on the below article, Dynatrace does a great job at tracing requests, however has a very long way to go when it comes to providing what is needed in terms of queue depths and setting thresholds around them.

Breaking down barriers: IBM MQ tracing across queue managers and queue clusters
Michael KoppProduct news · June 24, 2019

Dynatrace already offers rich support for tracing your business requests end to end, including all major queue vendors from ActiveMQ, HornetQ, RabbitMQ, Tibco EMS, Software AG’s webMethods Broker and Universal messaging, Azure Service Bus, and IBM MQ.

We have struggled with trying to monitoring queue depths within Tibco EMS using Dynatrace and to date it has been a failure. In fact, I would say it is the one big reason we are now evaluating other solutions for monitoring our messaging services outside of Dynatrace. We went down the road of custom plugin development even and that is when it became very known to us that Dynatrace does not deal well with trying to tackle queue depth monitoring which frankly is vital in terms of monitoring messaging which is why it leaves me scratching my head. It's not to say that monitoring queue depths of something like Tibco EMS would be impossible with Dynatrace, but there is nothing that would be user friendly and straight forward about it at this time.

We are now moving towards GCP Pub/Sub and currently do not plan to monitor that with Dynatrace due to the lack of information coming from Dynatrace on GCP in terms of roadmap. Keep in mind, this is not because we do not wish to use Dynatrace for it - We would LOVE too! But currently it is not an easy thing to do with Dynatrace and far from out of the box VS something like Prometheus & Grafana.

Example of one of our Tibco EMS queues in Grafana

The above should be easy to do in something like Dynatrace and can be until you want to monitor hundreds of individual queue depths, set unique thresholds on each of them and do it all from within the GUI.

So this brings me to 3 questions for Dynatrace:

#1 Are there any plans to allow for the monitoring of messaging queue depths with the ability to adjust thresholds and be out of the box? If so, for which technologies and what does that timeline look like?

#2 When is GCP going to become just as supported as AWS and Azure and reflect that within the Dynatrace GUI?

#3 This is somewhat related to question #1, but what does the roadmap look like to support GCP Pub/Sub? Again, this is not about "requests end to end" and about the ability to monitor and alert on queue depths.

@Michael K. - perhaps you could shed some light around these things? Thank you!

Comment
Bruce B.
Andrew P.
Justin A.
Hervé R.
Leon Z.
Julius L.
Michael D.

People who like this

7 Show 2
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image Leon Z. · Dec 04 at 03:59 AM 1
Share

Hey Larry , We are yet to compare notes about the EMS and BW plugins, I have extended my EMS plugin to include the following Metrics:

Queue Depth

Pending Messages

outbound Message Rate

Inbound Message Rate

Consumers


I have created a dashboard with custom charts for this, but I am not too happy about the way it is aggregated on the dashboard over a selected time period, so these metrics cannot be seen as real time from the dashboard. It is real time in the process information though.


I am also puzzled about the default process Detection/naming rules for EMS , it depends obviously on the process EXE , I have multiple EMS servers and they are all detected into different process group instances(process startup in different paths per host) ALL named "Tibco EMS". Do you perhaps have any ideas to change the group naming so that it reflects an environment etc ?

@Sebastian K., I am happy to donate the work I have done on the BW ActiveGate and EMS oneagent plugin to the cause.

=

avatar image Larry R. Leon Z. · Dec 04 at 02:20 PM 1
Share

Good morning! Yeah I have been side lined working on more end of the year stuff. I was trying to remember, are you building a OneAgent plugin or ActiveGate plugin?

Asking because when I tried to go the OneAgent route, I found no method to get get to a level of where each queue name could be tagged. This is a requirement for us as that is how we assign ownership. By going the route of the ActiveGate plugin, that allows me to create custom devices so I take each queue and do exactly that, I turn each one into a custom device. This then treats each one independently and allows for auto-tagging. I also placed them under their own technology called "EMS Queues" so that they are separated from the core "Tibco EMS" technology. The reason for this is because there is one team which owns the support of Tibco EMS itself for things such as upgrades, patching, etc. and then the queues are owned by multiple teams.

We are bringing in....

  • Queue Depth
  • Queue Size

The reason for this is due to the amount of custom metrics which each metric consumes. Because we go to a depth of the individual queues, so it adds up fast depending on how many queues you have.

I too am still playing around with the aggregate. I will be honest, this is one area of Dynatrace that always seems to confuse me that I can never keep straight in my head and often just have to play with it. SUM, COUNT, MAX, etc... I always have to go back and look up exactly what each one means.

There are pros and cons to each method.

OneAgent Plugin

Pros

  • AI is utilized.
  • The relation ship between the said queues, services, processes, etc. are understood by Dynatrace.
  • Does not use as many custom metrics as the ActiveGate version due to higher up the tree scope.

Cons

  • All queues are placed under the technology "Tibco EMS" process.
  • Queues are not treated independently to a point that they can individually be tagged manually or through rules.
  • OneAgent plugins do not have the ability to create custom devices.
  • Does not scope deep enough down to each independent queue level.


ActiveGate Plugin

Pros

  • Has the ability to create custom devices, therefore allowing each queue to be turned into its own individual device.
  • Haas the ability to scope down to each individual queue.
  • Because queues can be individual devices, you can manual and / or auto-tag like you would anything else.
  • Queues can be separated out from under the "Tibco EMS" technology group which is a huge benefit if you have multiple teams supporting different things within EMS.
  • Each team can create custom thresholds under "Custom events for alerting" and can be very specific due to the fact that the plugin uses the queue names as a dimension.

Cons

  • The AI is not utilized.
  • The relationships between said queues, services, processes, etc. are not understood or shown by Dynatrace. This one might not stay a con as I think it's due to the IP being assigned more than once. Working on this one.
  • SmartScape shows these custom devices which are queues as hosts.
  • Can utilize a large amount of custom metrics.

I am still working on the ActiveGate plugin and I might be able to reach a point where the relationship between the queue and services is shown. I was talking to some folks at Dynatrace and I think it's due to the fact that the same IP is being used which is why it does not track. So currently I am working on the structure a bit to see if I can get around that.

Both options are a trade off from what I have found. It really depends on the need at the end of the day. For us, we must be able to obtain these metrics, but also assign ownership to each queue and therefore custom devices are a must.

While I can not speak for Dynatrace, I have had a few conversations with these great minds over there about this. The out of the box solution for EMS and any other messaging service for that matter would be for Dynatrace to just go one level deeper. As you can see below, Dynatrace out of the box detects the EMS queue names. It already see's these queues. If you click on them however they just take you to the SmartScape and do not go any deeper. My hope is that Dynatrace will enrich the product more for messaging services so that not only do you see the queues, but can click into them and then see all of the metrics which are vital to customers. Such as queue depth, size, etc. This is a challenge as every messaging service is not the same. For example, when it comes to Tibco EMS, they would need to use the tibemsadmin. For something like Pub/Sub in GCP, they would most likely need to tap into StackDriver. There is a big challenge there, but I think it's one they need to meet due to the fact that if you can not get down to these levels of metrics when it comes to messaging services, it becomes somewhat useless. I do think we will see headway on this from Dynatrace eventually, but for which messaging services and when - Anyones guess.

In terms of the process group detection, are all the servers running the exact same version of EMS? If so, I would bet that it's the different paths thats making Dynatrace separate them which I would expect. That being said, if you go to "Settings", "Processes and containers", and select "Process group detection", you can build a new "detection rule" there which should then allow you to group them up.


Hope that helps! I will release what I have soon. I need to get over to GitHub.

Thanks!

queues.png (97.5 kB)

2 Replies

  • Sort: 
  • Most voted
  • Newest
  • Oldest
avatar image

Answer by Hervé R. · Dec 03 at 03:25 PM

Hi there, I'm in the same situation. We use Dynatrace to monitor our platform on GCP with full stack agents but we really need a Pub/Sub view of any sort and so far nothing seems to be available.

DT is doing great on AWS and Azure but what about GCP for cloud provided services ?

Comment
Larry R.

People who like this

1 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

avatar image Larry R. · Dec 03 at 04:19 PM 1
Share

Funny you should mention that. We too are in GCP and moving more everyday and have the exact same need. The timing could not have been better for your post as I was just finishing an email off to Dynatrace asking for specifics exactly around Pub/Sub. I think when it comes to GCP in general, Dynatrace needs to start releasing more news about their plans. You are correct, there is a strong focus and support on AWS and Azure, but very little information and presence seems to come in the way of GCP. I am not sure if this is just due to business relationships between vendors or what the reasoning is behind it. They do support GCP as we are using them now, but the advertising of that and what enhancements are coming around it are far and few in between. One thing that really bugs me is that you clearly see AWS and Azure right there in the menu on Dynatrace, yet GCP continues to be missing. Yes you have Kubernetes, etc.... but GCP is oddly missing from the collection. I would and I think many others like to see some updates and focus from Dynatrace around GCP and whats coming up shortly.

avatar image

Answer by Sebastian K. · Jun 25 at 06:51 PM

What you’ve posted is really interesting topic. I will consider it as well in our company as candidate for next plugin to Dynatrace. I have some idea how we can make it work but I have to check it :) But I’m curious as well if DT has any plans for it :)

Sebastian

Comment
Larry R.

People who like this

1 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
â–¼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 52.4 MB each and 262.1 MB total.

Join the conversation!

First steps in the forum
Community User Guide

LIVE WEBINAR

"Power Demo: Software Intelligence for Cloud Infrastructure"


DECEMBER 12, 10:00 AM GMT / 2:00 PM ET

Register here

Live webinar: Ensuring Digital Business Availability with Dynatrace

Learn how Dynatrace Real User Monitoring automatically detects errors that impact your end users caused by erroneous 3rd party or CDNs.
December 10, 4:00 pm CET / 10:00 am ET
Register here

Live webinar: Ensuring Digital Business Availability with Dynatrace

Learn how Dynatrace Real User Monitoring automatically detects errors that impact your end users caused by erroneous 3rd party or CDNs.
December 10, 4:00 pm CET / 10:00 am ET
Register here

Live webinar: Ensuring Digital Business Availability with Dynatrace

Learn how Dynatrace Real User Monitoring automatically detects errors that impact your end users caused by erroneous 3rd party or CDNs.
December 10, 4:00 pm CET / 10:00 am ET
Register here

Follow this Question

Answers Answers and Comments

37 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

Is there Dynatrace OneAgent support for ehcache? 1 Answer

Single tenant per environment OR multi-tenant with a mixed bag of on-prem and cloud 0 Answers

Is it possible to monitoring JMS queue depths without the need for custom metrics? 1 Answer

Deploying Dynatrace oneagent on IBM Kubernetes 3 Answers

What is the status of Google Cloud Platform coming to Dynatrace? 0 Answers

Forum Tags

mobile monitoring dotnet iis chat kubernetes servicenow amazon web services mysql rest api cassandra dashboard oneagent sdk cmc application monitoring smartscape request attributes monitoring ui developer community user tagging log monitoring services ufo activegate auto-detection high five award webserver uem test automation license web performance monitoring ios migration management zones web services sso notifications host monitoring reports browser monitors java sap vmware maintenance window javascript appmon ai availability tipstricks automation extensions session replay permissions faq documentation problem detection http monitors easytravel network docker tags and metadata cloud foundry google cloud platform synthetic monitoring process groups account usability dynatrace saas gui paas openshift key user actions administration hosts user actions postgresql synthetic locations oneagent security Dynatrace Managed user management python technologies mongodb openstack user session monitoring continuous delivery citrix configuration alerting NGINX action naming linux nam installation database mission control apache mobileapp RUM php azure purepath davis plugins scripting aix nodejs android
  • Forums
  • Public Forums
    • Community Connect
    • Dynatrace
      • Dynatrace Open Q&A
    • Application Monitoring & UEM
      • AppMon & UEM Open Q&A
    • Network Application Monitoring
      • NAM Open Q&A
      • Enterprise Synthetic Monitoring
    • Synthetic Classic
      • Synthetic Classic Open Q&A