• Forums
    • Public Forums
      • Community Connect
      • Dynatrace
        • Dynatrace Open Q&A
      • Application Monitoring & UEM
        • AppMon & UEM Open Q&A
      • Network Application Monitoring
        • NAM Open Q&A
  • Home /
  • Public Forums /
  • Application Monitoring & UEM /
  • AppMon & UEM Open Q&A /
avatar image
Question by Ken L. · Nov 14, 2014 at 07:08 AM ·

Dynatrace Infrastructure Monitoring

All,

My team is considering replacing our existing AIX/Linux server monitoring tool with the Dynatrace Unix Monitor Plug-in. This will mean migrating 700 servers and 100 reports to dynatrace as well as an additional 30 plus users. The server count is split by QA and Production dynatrace instances. I am wondering if any of you have completed such a task. If so, can you please provide me feedback on the following:

How large was the migration (server and report counts)

How did the additional measures affect database growth?

What challenges did you face?

Did the additional dashboard execution slow response time down? (ie slow dashboard loads or "low server memory" error while loading)

Were you able to generate nightly reports and have them complete in a timely manner?

Did the nightly reports effect the completion of the cleanup task?

Did you have issues finding equal or similar measures/metrics in Dynatrace Unix Monitoring Plug-in that were available in the old server monitoring system.

How did you organize the servers in Dyantrace? For example did you create system profiles for non-jvm/clr applications or perhaps create one monitoring profile and use infrastructure sites and groups to organize them by application?

 

Thank you for your time,

Ken Lynch

 

 

Comment

People who like this

0 Show 0
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

4 Replies

  • Sort: 
  • Most voted
  • Newest
  • Oldest
avatar image

Answer by Derek A. · Nov 18, 2014 at 06:49 AM

Hope that helps Ken. Also, take advantage of the Host Groups if possible. A simple example could be having a "Base Linux Group" (or name it whatever you want) where this group would contain your Linux servers because when you set up your monitors you can target group(s) or server(s). By targetting groups, this saves you time because when you add/remove a server from a group, you don't have to update any monitors if those monitors are targetting the group. Otherwise if your monitors were targetting specific servers, you would need to update those monitors when you add/remove servers in your environment.

There may be some scenerios where targetting specific servers makes more sense, but generally for your "default monitoring", groups will come in handy.

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Derek A. · Nov 18, 2014 at 06:09 AM

Hi Ken, let me provide some insight what we have done in our envrionment. We are monitoring several thousand Linux and Windows servers across non-Prod and Prod. In our dynaTrace Test Center Edition, we do all of our non-Production server monitoring and in our dynaTrace Production Edition we are doing all of our Production server monitoring. Both environments have a ton of Collectors dedicated to agent-less/plugin monitoring. These Collectors themselves are VMs and run 10 Collector instances on them. The reason why they are running 10 instances is because dynaTrace Collectors only support 100 concurrent threads running at once so we get around this by running multiple Collectors on a single server to accomodate the thousands of servers being monitored every minute. I don't think there's a limit on how many instances you have on a server, just as long as resources on the server (CPU and Memory) can accomodate it. We just found that 10 was our sweet spot.

There are a number of AIX and Linux based plugins my team has developed, feel free to search for them under the plugin downloads. To answer some of your other questions......

How large was the migration (server and report counts) Several thousand servers. Here's a tip when adding the server names into dynaTrace under dynatrace Server settings > Infrastructure > Hosts. Adding your 700 servers will take a while adding them one by one since there's no formal "mass import". You can cheat though by editing the cmdb.config.xml on the dynaTrace Server and adding the servers that way. You have to stop the Server, modify the file, then start the server in order for it to take affect. Be sure and create a backup of the file first in case you mess it up and need to reset. If you view the xml file you can see the tags and format required so you could then script it.

How did the additional measures affect database growth? The database grew a crap ton. This will really depend on how often (i.e. every minute, etc.) and how many metrics you are collecting and then of course what your Performance Warehouse Storage Settings are for aggreating the data (high, medium, low resolutions). I can't give you a clear cut answer on what you can expect.

What challenges did you face? Mainly the 100 concurrent thread restriction, didn't realize this in the beginning. Another challenge was determining what metrics we really wanted. Moving all of this into dynaTrace allowed us to get rid of other tools that were doing this. In doing so, we took this opportunity to re-evaluate the need of some of the monitoring and were able to determine that traditional metrics we collected for various users, weren't needed anymore so we were able to "trim the fat" if you will.

Did the additional dashboard execution slow response time down? (ie slow dashboard loads or "low server memory" error while loading) We didn't really see any degredation in performance. DynaTrace continues to mature and performance improves every release. I would just say keep an eye on the out-of-the-box dashboards like Performance Write and Performance Read.

Were you able to generate nightly reports and have them complete in a timely manner? We generate a lof of reports and they all generate exactly when they are configured to. I wouldn't worry about this.

Did the nightly reports effect the completion of the cleanup task? This did not seem to have an impact on the cleanup task.

Did you have issues finding equal or similar measures/metrics in Dynatrace Unix Monitoring Plug-in that were available in the old server monitoring system. A lot of the metrics in the older monitoring tools, dynaTrace could capture the same things. If dynaTrace couldn't out-of-the-box, then we created our own custom plugins to accomplish what we needed. Which we've shared with the Community.

How did you organize the servers in Dyantrace? For example did you create system profiles for non-jvm/clr applications or perhaps create one monitoring profile and use infrastructure sites and groups to organize them by application? We have a number of System Profiles for non-JVM/CLR applications. We have one dedicated System Profile just for collecting OS level metrics (i.e. CPU, memory, disk, etc.). As I mentioned earlier, our Test Center Edition does all non-Prod monitoring and our Production Center Edition does all Production monitoring.

 

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Ken L. · Nov 18, 2014 at 06:33 AM 0
Share

Derek, thank you so much for replying to this post. I consider the information and advice you have provided very valuable. The import and collector tips will be especially time saving.  I will also checkout the custom plugin you have posted.

Kind Regards,

avatar image

Answer by Ken L. · Nov 18, 2014 at 02:35 AM

Andreas, thank you so much for responding to the post. I hope the clients you have in mind will be willing to respond to the post. I appreciate you reaching out to them.

In response to your questions:

) Can you share which tools you are using right now? Also - can you tell us why you consider using scheduled reports for infrastructure vs. looking at dashboards on-demand in case dynatrace alerted on problems? We are using Compuware Server Vantage. Scheduled reporting is a way of life for us. Users prefer these reports as opposed to using a the client software. It also helps the ESM team by making user management simpler. Meaning that we don't have to create so many accounts and train them on the UI. Much like our existing Dynatrace Application performance reports, receivers of these infrastructure reports are not always technical personnel. We would defiantly link on-demand dashboards in DT alerts. I will admit that Dynatrace offers us the opportunity to perform application and infrastructure monitoring within the same tool. This puts us in the position to reconsider the need for certain reports and perhaps utilize live dashboards for infrastructure monitoring.

b) When setting up monitors in dynaTrace you can leverage the "Infrastructure" definitions in the Server Settings Dialog. You said that your envioronment is split between QA & Prod. You could define two "sites" and manage your hosts in both environments which makes it easier to define the monitor plugins when it comes to defining the list of machines to monitor -> I assume you will have different Collectors for these two environments

We have multiple collectors for these environments. In fact, I may define two additional collectors per environment for the monitoring tasks.

c) Besides using the Unix Monitor also have a look at our Host Agent. This agent automatically monitors your host system (cpu, network, disk, memory). What additional metrics are you looking for? We are looking to expand our host agent capabilities for future versions - so - it would be interesting to learn which additional system metrics you are interested in

When you reference the Host Agent, are you speaking of the Host monitoring that is available by implementing JVM/.net agents? Or, is this a separate agent that is installed on the server? Many of the servers we will monitor are not java based. Regarding metrics, a set that is heavily utilized in SV is the LPAR Host monitoring metrics. While Dynatrace has VMware Host monitoring it is lacking in the IBM area.

d) Querying dashboards from several new clients should be that big of a performance impact on the dynatrace server. Especially when these dashboards contain charts where the data comes from the performance warehouse. You should definitely keep an eye on the performance warehouse performance as you will have much more Writes to the database than before - but in general I dont see a problem. But please have a look at the Performance Warehouse Health Dashboards accessible through the Start Center

I am concerned about the use of new infrastructure dashboards. as it will increase the demand on the FE JVM. While in 6.x the performance is much better, we still see "out of server memory" errors at times when executing dashboards. Today, we have at least 25 concurrent users in productions at peak times.

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Andreas G. · Nov 16, 2014 at 06:14 PM

Hi Ken

Great that you are consolidating your tool landscape. I have a couple of customers in mind that are using dynatrace for large scale system monitoring. I will ping them in the hope they share their experiences here on your forum post.

I just got a couple of questions and also answers to some of your points

a) Can you share which tools you are using right now? Also - can you tell us why you consider using scheduled reports for infrastructure vs. looking at dashboards on-demand in case dynatrace alerted on problems?

b) When setting up monitors in dynaTrace you can leverage the "Infrastructure" definitions in the Server Settings Dialog. You said that your envioronment is split between QA & Prod. You could define two "sites" and manage your hosts in both environments which makes it easier to define the monitor plugins when it comes to defining the list of machines to monitor -> I assume you will have different Collectors for these two envirionments

c) Besides using the Unix Monitor also have a look at our Host Agent. This agent automatically monitors your host system (cpu, network, disk, memory). What additional metrics are you looking for? We are looking to expand our host agent capabilities for future versions - so - it would be interesting to learn which additoinal system metrics you are interested in

d) Querying dashboards from several new clients should be that big of a performance impact on the dynatrace server. Especially when these dashboards contain charts where the data comes from the performance warehouse. You should definitely keep an eye on the performance warehouse performance as you will have much more Writes to the database than before - but in general I dont see a problem. But please have a look at the Performance Warehouse Health Dashboards accessible through the Start Center

I hope this helps + I hope I get some additoinal answers from some customers

Andi

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

How to get started

First steps in the forum
Read Community User Guide
Best practices of using forum

NAM 2019 SP5 is available


Check the RHEL support added in the latest NAM service pack.

Learn more

LIVE WEBINAR

"Performance Clinic - Monitoring as a Self Service with Dynatrace"


JANUARY 15, 3:00 PM GMT / 10:00 AM ET

Register here

Follow this Question

Answers Answers and Comments

1 Person is following this question.

avatar image

Forum Tags

dotnet mobile monitoring load iis 6.5 kubernetes mainframe rest api dashboard framework 7.0 appmon 7 health monitoring adk log monitoring services auto-detection uem webserver test automation license web performance monitoring ios nam probe collector migration mq web services knowledge sharing reports window java hybris javascript appmon sensors good to know extensions search 6.3+ server documentation easytravel web dashboard kibana system profile purelytics docker splunk 6.1 process groups account 7.2 rest dynatrace saas spa guardian appmon administration production user actions postgresql upgrade oneagent measures security Dynatrace Managed transactionflow technologies diagnostics user session monitoring unique users continuous delivery sharing configuration alerting NGINX splitting business transaction client 6.3 installation database scheduler apache mobileapp RUM php dashlet azure purepath agent 7.1 appmonsaas messagebroker nodejs 6.2 android sensor performance warehouse
  • Forums
  • Public Forums
    • Community Connect
    • Dynatrace
      • Dynatrace Open Q&A
    • Application Monitoring & UEM
      • AppMon & UEM Open Q&A
    • Network Application Monitoring
      • NAM Open Q&A