Virtual Vs. Physical

Hi Frank,

We have a general mechanism called "Agent Time" that ensures that clock drift on monitored machines doesn't impact our PurePaths. It's kind of an global time sync management. It works in physical and virtual environments and ensures that data is not essembled out of order due to "normal" clock drift issues.

As for vmware introduced timekeeping issues, on tickless systems we don't have to do anything special. A tickless system does not suffer time keeping issues, because its not based on a constant rate interrupt, even under high utilization. We typical use the OS available high precision time to measures execution and response times. In a tickless system the measurement is real time (no timekeeping issues, no apparent time). The only caveat is that steal time is of course accounted in that time and thus execution time. You can identify this by correlating guest steal times with your execution times. You should use the VMWare plugin to capture CPU ready time (vmware's name for steal time) for your vms. (as long as steal time is in normal bounds this should not matter much, if its big you will see it, think of it similar to GC suspension, its always there, but as long as its small it doesn't matter).

In a VMWare system you should install the guest tools to avoid that steal time is inadvertently accounted as CPU time by the guest, there is not such issue with Xen or other paravirtualized systems.

In Tick counting systems on VMWare we suggest using the VMWare Pseudo Counters to avoid clock issues. This timer acts similar to clocks in tickless systems in that it provides the system with a real time clock. This clock is provided by the hyperviser and does therefore not suffer time keeping issues. Just like with tickless systems steal time or other suspension events (e.g. VMotion) are included in measured times and thus you need to correlate these events to your transaction time measurements to be aware of them.

Tick Counting systems on paravirtualized systems (Xen or KVM) should not have any timekeeping issues as they use the real physical clock.

In all cases a VMotion event will not affect the clock, but of course methods and response times measured "during" a vmotion event will be much longer then they usually are even though the system didn't do anything. 

I hope this answers all you questions, if you need more information please don't hesitate to ask.
 

Best
Michael

Ping... I need to make a case one way or another to my management. Can someone answer this question either here or send me a private email ...

"We have both tick counting OS's and tickless counting OS's here. I would be interested in any additional detail you can provide on how you're avoiding clock drift type issues. Especially on agents and dt servers that are on highly utilized virtual hosts, hosts that may be undergoing vmotion etc"

Thanks,

Frank

Also do our native dynaTrace agents have built-in special virtualization detection and high-resolution timing code that has been specially built and optimized for accurate nano-second timing in virtualized environments

Good Morning

To add to Robs comment and answer your question: Running the dT Server on a VM wont have any impact on time calculations. We get the timings from the Agents - so - the VMs on the dT Server dont have an impact.

So - just make sure that you have enough resources allocated to this VM - use the dynaTrace Health Dashboards to monitor the dynaTrace Server / Collector Health and Performance. Follow the Deployment Guide to make sure everything is properly sized - then you should be ok

Hi Frank,

I can't answer your question on computations as I don't have access to the code, but I want to ask a clarifying question to help anyone from the lab who steps in to answer.

When you're talking about the "time measurements calculated by the dynaTrace server..." - are you talking about its own self-health measures (CPU etc) or are there other measurements you are concerned about?

I can comment in general on hosting the dT server on virtual gear, from experience:

• First, it cannot be the case that you over-commit resources on the physical box (for example the case where four VMs each have 4GB of RAM, but the physical host only has 12GB of RAM). The dT server requires full access to the resources that have been spec'd for it. Remember that it runs at the speed of your application (capturing every transaction) plus it requires the ability to process that data.
• Second, there will be hypervisor-related overhead associated with running as a VM. The least overhead I've seen (testing at a large bank) was in the ballpark of 2+%. It was unclear what the exact virtualized I/O overhead was. The point is you need to take that into account when building a virtual dT server

Rob

Ping.... Andreas... or others... any comments on this?

Hi Frank

Just to verify. Are you talking about running the dynaTrace server on a VM Infrastructure or your application (and with that our agents) on a virtualized infrastructure?