Question about v4.x agent upgrades
Hello,
We are currently upgrading to 4.1 from 3.5.2 on Solaris equipment. In doing so, we are trying to better organize our DynaTrace layout to make it easier in the future. Currently we are at ~70 JVM agents with more growth to come. Our current installation process is to install the agent on each individual server and we use a symlink to point at the active version for that application (this allows us to pre-install new versions without turning them on yet). Obviously, installing the agent 70+ times is tedious and it will be getting worse as we grow. So we are thinking about making a shared mount that we have the different DynaTrace installs in. Thus when upgrading in the future, we just need a single install to be performed.
Ex:
/dynatrace |=> dynatrace-3.5.2 |=> dynatrace-4.0 |=> dynatrace-4.1
We do this with other shared software libraries and it works well. But with Dynatrace, I have a few questions about this approach.
1) By default, DynaTrace wants to log to ?$<DT_HOME>/log directory. Can we control that path at the agent level with a parameter? Otherwise in the above scenario, all agents would try to log to the same log directory which I assume would be bad.
2) In the v4.x releases, when doing the maintenance releases, it seems that the the release is loaded to the server and that is pushed down to the client which updates the binary on the next application restart. Is that correct? If so, how would that affect things if a single JVM was restarted which updated the binary that is shared by 70 other JVMs that have not yet restarted?
Also, is there any other gotchas around this?
Finally, how are any other groups who are at 50+ agent doing their installation processing to make it as painless as possible?
We have about 500 agents deployed so far, and about 70 collectors or so. Here's how we do it, and how dynaTrace 4.1 both facilitates and makes our lives more challenging. We're running 4.1 and focus primarily on Java in the middle-tier. We use the same symlink model you are proposing. Our agent file system is read-only, so we don't preserve logs at all.
We use rsync to push out agents across our infrastructure, largely because we don't use shared drives since that introduces single points of failure. We do the same approach for our collectors. With dynaTrace 4.1, agent upgrades for Java aren't required since it auto-upgrades itself after your process is restarted. Same for the collectors. Upgrades can be done even while running processes are touching the agent code, as long as your upgrade process doesn't change the file ids. SSH commands are used to auto-start or auto-shutdown/restart the collectors, although you can also use the dynaTrace console as long as the collector process is running.
The downside, and a real problem for us, is that it auto-updates everything after an upgrade. This violates our change management processes, where we may not want everything upgraded just because the central server is updated. We''ve put in an RFE to help manage that more intelligently and risk-appropriate way.
Hi Erik,
ad 1) Changing the log folder is possible through the agent option logpath, a full list of all the options with examples can be found in the documentation at https://apmcommunity.compuware.com/community/display/DOCDT41/Java+Agent+Configuration.
ad 2) The agent is not replaced but the newest version is downloaded and stored in a seperate folder (e.g. on my Windows system this is C:\Program Files (x86)\dynaTrace\dynaTrace 4.1.0\agent\downloads where I have sub folders for the various releases/patches). When the agent starts up it connects to the server, checks for the current version and then uses the appropriate binaries (+ downloads them if necessary). It means that user that the JVM is running under needs write access to the agent folder. If that behavior is not desirable in your specific environment you can prevent this by using the dtagentcore.dll/.so files (-agentpath:/path/agent/dtagentcore.so=name=test,server=localhost). In that case you have the behavior as before and would need to manually update the agent libraries.
Best, Roman
Thank you for the replies. That clarifies a bunch of items.
So here is our plan for our migration.
1) Apps all point to DT collectors using VIPs which are currently pointing to the 3.5.2 collectors.
2) Repoint the VIPs to new 4.1 collectors.
3) Repoint the symlink on each app to point to 4.1 install and bounce app to pick up and use the 4.1 binaries.
Does anyone know if that plan should work. We know we will lose our 3.5 history but we are okay with that. Our worry is when we point the VIPs to 4.1 collectors how the 3.5.2 agents will react between that point and when the app is bounced to use the 4.1 binary.
Note, we plan to try and test this out in the next week or so. Just seeing if anyone has good/bad experience around this that we can learn from ahead of time.
Thanks in advance.
- Erik
