Dynatrace Infrastructure Monitoring

Hi Ken

Great that you are consolidating your tool landscape. I have a couple of customers in mind that are using dynatrace for large scale system monitoring. I will ping them in the hope they share their experiences here on your forum post.

I just got a couple of questions and also answers to some of your points

a) Can you share which tools you are using right now? Also - can you tell us why you consider using scheduled reports for infrastructure vs. looking at dashboards on-demand in case dynatrace alerted on problems?

b) When setting up monitors in dynaTrace you can leverage the "Infrastructure" definitions in the Server Settings Dialog. You said that your envioronment is split between QA & Prod. You could define two "sites" and manage your hosts in both environments which makes it easier to define the monitor plugins when it comes to defining the list of machines to monitor -> I assume you will have different Collectors for these two envirionments

c) Besides using the Unix Monitor also have a look at our Host Agent. This agent automatically monitors your host system (cpu, network, disk, memory). What additional metrics are you looking for? We are looking to expand our host agent capabilities for future versions - so - it would be interesting to learn which additoinal system metrics you are interested in

d) Querying dashboards from several new clients should be that big of a performance impact on the dynatrace server. Especially when these dashboards contain charts where the data comes from the performance warehouse. You should definitely keep an eye on the performance warehouse performance as you will have much more Writes to the database than before - but in general I dont see a problem. But please have a look at the Performance Warehouse Health Dashboards accessible through the Start Center

I hope this helps + I hope I get some additoinal answers from some customers

Andi

Andreas, thank you so much for responding to the post. I hope the clients you have in mind will be willing to respond to the post. I appreciate you reaching out to them.

In response to your questions:

) Can you share which tools you are using right now? Also - can you tell us why you consider using scheduled reports for infrastructure vs. looking at dashboards on-demand in case dynatrace alerted on problems? We are using Compuware Server Vantage. Scheduled reporting is a way of life for us. Users prefer these reports as opposed to using a the client software. It also helps the ESM team by making user management simpler. Meaning that we don't have to create so many accounts and train them on the UI. Much like our existing Dynatrace Application performance reports, receivers of these infrastructure reports are not always technical personnel. We would defiantly link on-demand dashboards in DT alerts. I will admit that Dynatrace offers us the opportunity to perform application and infrastructure monitoring within the same tool. This puts us in the position to reconsider the need for certain reports and perhaps utilize live dashboards for infrastructure monitoring.

b) When setting up monitors in dynaTrace you can leverage the "Infrastructure" definitions in the Server Settings Dialog. You said that your envioronment is split between QA & Prod. You could define two "sites" and manage your hosts in both environments which makes it easier to define the monitor plugins when it comes to defining the list of machines to monitor -> I assume you will have different Collectors for these two environments

We have multiple collectors for these environments. In fact, I may define two additional collectors per environment for the monitoring tasks.

c) Besides using the Unix Monitor also have a look at our Host Agent. This agent automatically monitors your host system (cpu, network, disk, memory). What additional metrics are you looking for? We are looking to expand our host agent capabilities for future versions - so - it would be interesting to learn which additional system metrics you are interested in

When you reference the Host Agent, are you speaking of the Host monitoring that is available by implementing JVM/.net agents? Or, is this a separate agent that is installed on the server? Many of the servers we will monitor are not java based. Regarding metrics, a set that is heavily utilized in SV is the LPAR Host monitoring metrics. While Dynatrace has VMware Host monitoring it is lacking in the IBM area.

d) Querying dashboards from several new clients should be that big of a performance impact on the dynatrace server. Especially when these dashboards contain charts where the data comes from the performance warehouse. You should definitely keep an eye on the performance warehouse performance as you will have much more Writes to the database than before - but in general I dont see a problem. But please have a look at the Performance Warehouse Health Dashboards accessible through the Start Center

I am concerned about the use of new infrastructure dashboards. as it will increase the demand on the FE JVM. While in 6.x the performance is much better, we still see "out of server memory" errors at times when executing dashboards. Today, we have at least 25 concurrent users in productions at peak times.

Hi Ken, let me provide some insight what we have done in our envrionment. We are monitoring several thousand Linux and Windows servers across non-Prod and Prod. In our dynaTrace Test Center Edition, we do all of our non-Production server monitoring and in our dynaTrace Production Edition we are doing all of our Production server monitoring. Both environments have a ton of Collectors dedicated to agent-less/plugin monitoring. These Collectors themselves are VMs and run 10 Collector instances on them. The reason why they are running 10 instances is because dynaTrace Collectors only support 100 concurrent threads running at once so we get around this by running multiple Collectors on a single server to accomodate the thousands of servers being monitored every minute. I don't think there's a limit on how many instances you have on a server, just as long as resources on the server (CPU and Memory) can accomodate it. We just found that 10 was our sweet spot.

There are a number of AIX and Linux based plugins my team has developed, feel free to search for them under the plugin downloads. To answer some of your other questions......

How large was the migration (server and report counts) Several thousand servers. Here's a tip when adding the server names into dynaTrace under dynatrace Server settings > Infrastructure > Hosts. Adding your 700 servers will take a while adding them one by one since there's no formal "mass import". You can cheat though by editing the cmdb.config.xml on the dynaTrace Server and adding the servers that way. You have to stop the Server, modify the file, then start the server in order for it to take affect. Be sure and create a backup of the file first in case you mess it up and need to reset. If you view the xml file you can see the tags and format required so you could then script it.

How did the additional measures affect database growth? The database grew a crap ton. This will really depend on how often (i.e. every minute, etc.) and how many metrics you are collecting and then of course what your Performance Warehouse Storage Settings are for aggreating the data (high, medium, low resolutions). I can't give you a clear cut answer on what you can expect.

What challenges did you face? Mainly the 100 concurrent thread restriction, didn't realize this in the beginning. Another challenge was determining what metrics we really wanted. Moving all of this into dynaTrace allowed us to get rid of other tools that were doing this. In doing so, we took this opportunity to re-evaluate the need of some of the monitoring and were able to determine that traditional metrics we collected for various users, weren't needed anymore so we were able to "trim the fat" if you will.

Did the additional dashboard execution slow response time down? (ie slow dashboard loads or "low server memory" error while loading) We didn't really see any degredation in performance. DynaTrace continues to mature and performance improves every release. I would just say keep an eye on the out-of-the-box dashboards like Performance Write and Performance Read.

Were you able to generate nightly reports and have them complete in a timely manner? We generate a lof of reports and they all generate exactly when they are configured to. I wouldn't worry about this.

Did the nightly reports effect the completion of the cleanup task? This did not seem to have an impact on the cleanup task.

Did you have issues finding equal or similar measures/metrics in Dynatrace Unix Monitoring Plug-in that were available in the old server monitoring system. A lot of the metrics in the older monitoring tools, dynaTrace could capture the same things. If dynaTrace couldn't out-of-the-box, then we created our own custom plugins to accomplish what we needed. Which we've shared with the Community.

How did you organize the servers in Dyantrace? For example did you create system profiles for non-jvm/clr applications or perhaps create one monitoring profile and use infrastructure sites and groups to organize them by application? We have a number of System Profiles for non-JVM/CLR applications. We have one dedicated System Profile just for collecting OS level metrics (i.e. CPU, memory, disk, etc.). As I mentioned earlier, our Test Center Edition does all non-Prod monitoring and our Production Center Edition does all Production monitoring.

Hope that helps Ken. Also, take advantage of the Host Groups if possible. A simple example could be having a "Base Linux Group" (or name it whatever you want) where this group would contain your Linux servers because when you set up your monitors you can target group(s) or server(s). By targetting groups, this saves you time because when you add/remove a server from a group, you don't have to update any monitors if those monitors are targetting the group. Otherwise if your monitors were targetting specific servers, you would need to update those monitors when you add/remove servers in your environment.

There may be some scenerios where targetting specific servers makes more sense, but generally for your "default monitoring", groups will come in handy.