• Forums
    • Public Forums
      • Community Connect
      • Dynatrace
        • Dynatrace Open Q&A
      • Application Monitoring & UEM
        • AppMon & UEM Open Q&A
      • Network Application Monitoring
        • NAM Open Q&A
  • Home /
  • Public Forums /
  • Application Monitoring & UEM /
  • AppMon & UEM Open Q&A /
avatar image
Question by Frank P. · Nov 07, 2013 at 05:27 PM ·

Virtual Vs. Physical

We are building out new infrastructure to handle new dynaTrace installs we are planning. Our company has a huge push to put everything on Virtual infrastructure (VmWare, Pureflex etc.). I know dynaTrace "will work" on a virtual server but I'm concerned that time measurements calculated by the dynaTrace server will be off. Can someone explain if dynaTrace can calculate and report on exact cputime when running in a virtualized environment? I'm concern that if the physical host hosting the Virtual server is under duress, that the time calculated by dynaTrace running on that virtual server will get skewed. If dynaTrace can calculate times exactly, can you provide some detail as to how?

 

Thanks,

Frank

Comment

People who like this

0 Show 0
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

7 Replies

  • Sort: 
  • Most voted
  • Newest
  • Oldest
avatar image

Answer by Michael K. · Nov 28, 2013 at 09:15 AM

Hi Frank,

We have a general mechanism called "Agent Time" that ensures that clock drift on monitored machines doesn't impact our PurePaths. It's kind of an global time sync management. It works in physical and virtual environments and ensures that data is not essembled out of order due to "normal" clock drift issues.

As for vmware introduced timekeeping issues, on tickless systems we don't have to do anything special. A tickless system does not suffer time keeping issues, because its not based on a constant rate interrupt, even under high utilization. We typical use the OS available high precision time to measures execution and response times. In a tickless system the measurement is real time (no timekeeping issues, no apparent time). The only caveat is that steal time is of course accounted in that time and thus execution time. You can identify this by correlating guest steal times with your execution times. You should use the VMWare plugin to capture CPU ready time (vmware's name for steal time) for your vms. (as long as steal time is in normal bounds this should not matter much, if its big you will see it, think of it similar to GC suspension, its always there, but as long as its small it doesn't matter).

In a VMWare system you should install the guest tools to avoid that steal time is inadvertently accounted as CPU time by the guest, there is not such issue with Xen or other paravirtualized systems.

In Tick counting systems on VMWare we suggest using the VMWare Pseudo Counters to avoid clock issues. This timer acts similar to clocks in tickless systems in that it provides the system with a real time clock. This clock is provided by the hyperviser and does therefore not suffer time keeping issues. Just like with tickless systems steal time or other suspension events (e.g. VMotion) are included in measured times and thus you need to correlate these events to your transaction time measurements to be aware of them.

Tick Counting systems on paravirtualized systems (Xen or KVM) should not have any timekeeping issues as they use the real physical clock.

In all cases a VMotion event will not affect the clock, but of course methods and response times measured "during" a vmotion event will be much longer then they usually are even though the system didn't do anything. 

I hope this answers all you questions, if you need more information please don't hesitate to ask.
 

Best
Michael

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Frank P. · Nov 27, 2013 at 05:33 PM

Ping... I need to make a case one way or another to my management. Can someone answer this question either here or send me a private email ...

"We have both tick counting OS's and tickless counting OS's here. I would be interested in any additional detail you can provide on how you're avoiding clock drift type issues. Especially on agents and dt servers that are on highly utilized virtual hosts, hosts that may be undergoing vmotion etc"

 

Thanks,

Frank


 

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Bernd G. · Nov 21, 2013 at 10:04 AM

Also do our native dynaTrace agents have built-in special virtualization detection and high-resolution timing code that has been specially built and optimized for accurate nano-second timing in virtualized environments

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Frank P. · Nov 21, 2013 at 07:05 PM 0
Share

Bernd,

Thank you for your response. I'm very pleasantly surprised to see that you are active on the community. We have always been very impressed with the level of involvement senior management has with your customers and this is a great example of that. It's great to see that's continuing. It's also good to see that you have implemented something to counter clock time issues inherent in virtual environments.  We have both tick counting OS's and tickless counting OS's here. I would be interested in any additional detail you can provide on how you're avoiding clock drift type issues. Especially on agents and dt servers that are on highly utilized virtual hosts, hosts that may be undergoing vmotion etc.

avatar image

Answer by Andreas G. · Nov 21, 2013 at 07:44 AM

Good Morning

To add to Robs comment and answer your question: Running the dT Server on a VM wont have any impact on time calculations. We get the timings from the Agents - so - the VMs on the dT Server dont have an impact.

So - just make sure that you have enough resources allocated to this VM - use the dynaTrace Health Dashboards to monitor the dynaTrace Server / Collector Health and Performance. Follow the Deployment Guide to make sure everything is properly sized - then you should be ok

Comment

People who like this

0 Show 2 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Frank P. · Nov 21, 2013 at 06:50 PM 0
Share

Thanks for the replys Rob and Andreas,

To answer your question, I'm actually concerned about both times coming from virtualized agents and time calcuated on the server itself for health measures etc. If we are unable to dedicate CPU's to a virtualized server, would it then be considered un-wise to install dynatrace server on virtualized infrastructure?  

avatar image Rob V. Frank P. · Nov 21, 2013 at 07:10 PM 0
Share

It would be unwise to put a dT server in an active environment if you cannot commit to the CPU, RAM, and I/O requirements. As mentioned, we need to run at the speed of your app to keep up with all transactions, plus our post-processing of them. If we're starved for resources you can imagine that causing performance data loss.

avatar image

Answer by Rob V. · Nov 20, 2013 at 07:29 PM

Hi Frank,

I can't answer your question on computations as I don't have access to the code, but I want to ask a clarifying question to help anyone from the lab who steps in to answer.

When you're talking about the "time measurements calculated by the dynaTrace server..." - are you talking about its own self-health measures (CPU etc) or are there other measurements you are concerned about?

I can comment in general on hosting the dT server on virtual gear, from experience:

  • First, it cannot be the case that you over-commit resources on the physical box (for example the case where four VMs each have 4GB of RAM, but the physical host only has 12GB of RAM). The dT server requires full access to the resources that have been spec'd for it. Remember that it runs at the speed of your application (capturing every transaction) plus it requires the ability to process that data.
  • Second, there will be hypervisor-related overhead associated with running as a VM. The least overhead I've seen (testing at a large bank) was in the ballpark of 2+%. It was unclear what the exact virtualized I/O overhead was. The point is you need to take that into account when building a virtual dT server

Rob

 

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Frank P. · Nov 20, 2013 at 06:29 PM

Ping.... Andreas... or others... any comments on this?

Comment

People who like this

0 Show 0 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image

Answer by Andreas G. · Nov 08, 2013 at 02:54 PM

Hi Frank

Just to verify. Are you talking about running the dynaTrace server on a VM Infrastructure or your application (and with that our agents) on a virtualized infrastructure?

Comment

People who like this

0 Show 1 · Share
10 |2000000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

avatar image Frank P. · Nov 12, 2013 at 01:00 AM 0
Share

Hi Andreas,

I'm talking about the dynaTrace server on VM. We have agents currently running on VM infrastructure. Our company wants to virtualize everything and we have to file an exception to acquire physical infrastructure.

Thanks,

Frank

 

How to get started

First steps in the forum
Read Community User Guide
Best practices of using forum

NAM 2019 SP5 is available


Check the RHEL support added in the latest NAM service pack.

Learn more

LIVE WEBINAR

"Performance Clinic - Monitoring as a Self Service with Dynatrace"


JANUARY 15, 3:00 PM GMT / 10:00 AM ET

Register here

Follow this Question

Answers Answers and Comments

3 People are following this question.

avatar image avatar image avatar image

Forum Tags

dotnet mobile monitoring load iis 6.5 kubernetes mainframe rest api dashboard framework 7.0 appmon 7 health monitoring adk log monitoring services auto-detection uem webserver test automation license web performance monitoring ios nam probe collector migration mq web services knowledge sharing reports window java hybris javascript appmon sensors good to know extensions search 6.3+ server documentation easytravel web dashboard kibana system profile purelytics docker splunk 6.1 process groups account 7.2 rest dynatrace saas spa guardian appmon administration production user actions postgresql upgrade oneagent measures security Dynatrace Managed transactionflow technologies diagnostics user session monitoring unique users continuous delivery sharing configuration alerting NGINX splitting business transaction client 6.3 installation database scheduler apache mobileapp RUM php dashlet azure purepath agent 7.1 appmonsaas messagebroker nodejs 6.2 android sensor performance warehouse
  • Forums
  • Public Forums
    • Community Connect
    • Dynatrace
      • Dynatrace Open Q&A
    • Application Monitoring & UEM
      • AppMon & UEM Open Q&A
    • Network Application Monitoring
      • NAM Open Q&A