question

Wouter C. avatar image
Wouter C. asked ·

Prevent agents to automatically update?

Hi,

Is there a way to prevent automatic agent upgrading after a server/collector upgrade to a newer version to allow upgrade of agents in a more controlled fashion?
I'd like to migrate the agents in our different production environments from 6.0 to 6.1 gradually, so each environment seperately in its own maintenance window.
As I see it, since we only have one dynatrace server, once the server and collector are upgraded, every agent restart will provoke a version upgrade for that agent.
This is quite handy for the agents I intend to upgrade at a certain moment, but it also means that from then on, all agents in other environments that happen to restart because of whatever reason will automatically be upgraded.
As I really can't enforce a no-restart-of-application-until-APM-guy-says-so policy, this means in practice we can't start upgrading until we feel it's safe to do so for all servers at once.

10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Wouter C. avatar image
Wouter C. answered ·

Hi Günter,

 

A somewhat late answer because I've been out of office for almost 2 weeks.

We've actually succeeded in making this work the day before your last post (so March 5th) with the help of a dynatrace consultant with us on location, but hadn't found the  time to respond at that moment as I left the next day on holiday.

 

I had copied the 7000 masteragent over the original 6.0 masteragent dll like you said, but I had left the original dtiisagent7.dll webserveragent as it were, because the full path could be specified in the 'Config Native Module' pop-up of the IIS configuration so I pointed it to the 7000 dtiisagent7.dll in that menu. After saving settings I did a full machine restart (so not just services to be 100% sure the new config would be used) --> this did not work

We noticed in the agent logs that even with the new path specified in 'Config Native Module' , the webserveragent in the default lib64 library was used.

After shutting everything down, overwriting the dtiisagent7.dll in the default lib64 library with the 7000 version and restarting everything, it worked.

So to me this problem seemed to be another "quirk" (for some reason the 'Config Native Module' path was not used, but instead the default one), though in no way I can be 100% sure of this, because I really don't have the time to try to reproduce this and just overwriting everything seems to do the trick.

 

So to summarize, I now know have the full information I need to 'lock' dynatrace agents at a certain (even fixpacked) level in a windows environment and I'm really very gratefull to you for the time and energy you put into clearing this out, but in essence, I still find the way dynatrace handles this unworkable in our production environment and really hope this will be considered a major issue to be addressed in a future release.

 

kind regards,

Wouter

1 comment Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Hi again, Wouter,
thanks for bearing with me to completely nail this and for all your input. Sorry it needed iterations!
I hope to finish the knowledge base article this week. I will post the link here.

Hope to revisit this in a good handful of months. (wink)

Thanks
G.

0 Likes 0 · ·
Wouter C. avatar image
Wouter C. answered ·

Hi Günter,

Unfortunately this time no "worked as advertized" from me either.

Can you confirm you've succeeded in making this nonbootstrap procedure work with masteragent and webserveragents at a fixpack upgraded level ( = not a release level like I did)?

I tried but it really didn't work out.

I stopped the masteragent service, backed up the lib directories and replaced the 'dtwsagent.dll' with the one I could find in the corresponding (6.0.0.7000) agentres jar on the dtserver.

I restarted the masteragent service with the dtagent.ini still having the "nonbootstrap true" and I could almost immediately see the masteragent connected with the right version level (6.0.7000), but no more IIS webserveragents (as expected due to compatability issues.

I then made correct native module configs mapped to the version 6.0.0.7000 dtiisagent7.dll files and deployed them on all the connection pools.

This didn't make the webserveragents visible, even not after a full server (-> full IIS) server restart.

Re-applying the nonbootstrap procedure to "roll back" to a release level, like I described earlier, immediately showed the masteragent and the webserveragents again (at version level 6.0.0.6738).

So again, can you confirm you really managed to do this for a fixpack level and do you maybe have any ideas where things are going wrong?


best regards,

Wouter

3 comments Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Sorry for the delay, Wouter!

I actually verified with the engineer and myself, but it may have been on 6.1 and you never can be sure...

I´ll revisit it on 6.0 as soon as I can and get back to you at the usual pace.

One thing in advance: It´s nobootstrap true (without the n), but as you say the replaced dtwsagent connected with the right version I guess this is just a typo here!

Thanks
G.

0 Likes 0 · ·

One thing for you today before I need to run:
The Agents in server/lib/agentres.jar seem to be the GA version. The FixPack ones are in agentres-6.0.0.7000.

Are you positive you used the right ones?

I still need to try myself.
Thanks
G.

0 Likes 0 · ·

Hi,

I tried again and practically everything worked. (I didn'tget around to test the GA and FixPack 6.0 version Agents (master and slave) against a FixPacked Server and Collector 6.1.)

I found two quirks that might get in the way, but only in a minor way:

  1. The Agents Overview consistently says "bootstrapped" when it is NOT.
  2. I had to reopen the dashboard to show 6.0 Agents against 6.1 Server / Collector. The System Overview showed it correctly and the Collector vertical tab in Server Settings also talked about it.

So, I connected non-bootstrap 6.0 GA and 7000 master and slave Agents against Server / Collector 6.0.0.7000 and 6.1 GA and got PurePath. (Copied lib and lib64 to lib-GA and lib64-GA and copied the 7000 .DLLs over these original libs. Later renamed lib and lib64 to lib-7000 and lib64-7000 and lib-GA and lib64-GA to lib and lib64 so I could test without changing anything else.)

One thing: Don't forget to open the instrumented site in the browser or the slave Agent won't load!

Let's do a Webex if you can't get it to work! (Sorry for the delay this week! I had to set up a new notebook...)

G.

0 Likes 0 · ·
Wouter C. avatar image
Wouter C. answered ·

Hi Günter,

I just finished testing the procedures you described in our test environment.

Short version: the procedures you described seem to work fine, though not 100% with the result I intented.
I'd still really really like to see a controlled-fixpack-deployment-like feature for new release agent upgrades.

Longer version:
My findings:

When "nobootstrap" was set to true, the webserver masteragent used the default dtagent.ddl in the lib directory of the original install path.
At the same time, I had the IIS webserveragents "locked" on the current version by using the path to the latest applied 6.0 fix pack updated 'dtiisagent7.dll' version as
a 'native module' using the method you have described. [the idea being to keep all the agents on the current version while upgrading the dtserver/collector and other
environments and then to revisit the locked environments and upgrade them when convenient]
This caused (as could be expected) incompatibility problems, because of a masteragent-webserveragent mismatch (by using the nonbootstrap config option
the master agent jumped back to the vanilla 6.0 release, while the webserveragents where at the latest fix pack level).
I tried to solve this by looking for a dtagent.dll at the same level as the latest fix, but unfortunately as far as I could see, fixpacks don't seem to provide newer
versions of the dtagent.dll file.
The only thing I could think off to make everything work, was downgrading the IIS webserver agents version by letting the module point to the non-fixpack-upgraded version
of the 'dtiisagent7.dll', so letting go of the idea to keep the exact version level of agents that we're running something I really didn't want to do but right now it
seems the only way (?)
After restart of the masteragent, everything seems to work fine, with some warnings in the agents overview (what was to be expected).
The masteragent has a slightly changed agent icon and the remark "agent restart required", the IIS webserver agents "agent outdated", but by filtering on the involved
agents, I could clearly confirm purepaths details where received from the "locked" IIS webserver agents (which are at 6.x while connecting to a 6.1.0.8054 collector).

For .NET I used 'regedit' to modify the dtagent.dll to dtagentcore.dll.
I found both entries (32bit and 64bit) twice, once under 'HKEY_CLASSES_ROOT' and once under 'HKEY_LOCAL_MACHINE\SOFTWARE', but as it was nowhere mentioned in the documentation you provided, I ignored the 2 entries under 'HKEY_CLASSES_ROOT' and only changed those under 'HKEY_LOCAL_MACHINE\SOFTWARE'.
Here I had the option to use the current version (fixpack 6.0.0.7059 version of dtagentcore.dll), but I decided to use the same 6.0 release version (6.0.6738) as the master agent and IIS webserver modules to have some consistancy in the version of the agent used (all agent now have the same version level, 6.0.6738).
A short time after inactivating and activating the .NET agents using the .NET Agent Configuration tool, the agents became visible in the dynatrace client in the agent overview with the same 'agent outdated' remark, but seemingly working fine.
I did a quick check and could easily find recent purepaths sent by the modified .NET agents.

 

My conclusion: disabling bootstrap for master/webserver/.NET agents in a windows/IIS environment seems indeed feasible using the steps you described, but ...

1) because of the lack of an updated version of the webserver masteragent (dtagent.dll) after applying a fixpack, I was also forced for the IIS webserveragents to go back to the last release version to avoid compatibility issues instead of using the current fixpack version. This was not entirely what I intended and may be unaccaptable in cases where a "fixpacked" version of a webserveragent fixes a critical bug necessary for the agent to run stable in a production environment.
2) the procedure is very cumbersome and user unfriendly, not to mention prone to errors and requires an extra maintenance window (at least the first time) just to be able to roll out a new release in a controlled fashion. I still think this really should be a basic feature for a production environment-ready product, so I'd really like to see something like this in a next release.

 

Best regards,

Wouter

2 comments Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Oh my, Wouter,
this is getting rrrreally ugly.
Sorry it took a while, but I did, guess, my third round to talk to the engineer who implemented the mechanism for the controlled FixPack roll-out and then to our web server instrumentation guru,... <sigh>

Then I tried out everything myself, because only then I can – with any confidence -- say that this will work. (Too many false prophets out there. <g>)

I had hoped for a one-liner from you (Works as advertized!), but it´s again a boatload... (Well, not that bad. <s>)

I already began a Non-Bootstrapped Agent Configuration page in our staging area, but from what I learned: This thread is "advertisement" enough for this chore and I need to ask product management how public we want this to be.

Yes, FixPacks, quite frequent, need to be an major factor in calculating the trade-off!

Necessary additional kludges:

  1. Once you are on the non-bootstrap Agents, FixPacked Agents won´t propagate to the <DT_HOME>/agent/downloads/<major.minor.release.build>/native/<OS> directory.
    You can get the Agents from <DT_HOME>/server/updates/update_<major.minor.release.build>/agentres-<major.minor.release.build>.jar.
    In this file you find all .DLLs in native/<OS>, also the new dtwsagent.dll.
  2. I guess I´d backup the lib and lib64 directories and replace the .DLLs with the new / FixPack ones.
  3. I attach the .REGs which will clear up the four vs. two spots. It´s four entries, but it changes the file from dtagent.dll to dtagentcore.dll in two spots.
    Edit: the un-reg is generic, but the regs are for a default 64 bit Dynatrace 6.1 installation, so you may need to edit the path.

Last time: With any considerable amount of Agents I would not want to go through this, but I´ll bug product management accordingly. <vbg>

Thanks again
G.

0 Likes 0 · ·

The un-reg is generic, but the regs are for a default 64 bit Dynatrace 6.1 installation, so you may need to edit the path.

DotNetAgentUnregAndRegNon-Bootstrap.zip

0 Likes 0 · ·
Guenter H. avatar image
Guenter H. answered ·

"Famous last words" / Disclaimer:

You already know about the selective / controlled roll-out of Agents with FixPacks / build-level updates. This is the best of both worlds: It gives you exact control and is convenient.

Sadly this mechanism has not yet been extended to upgrades (at the major / minor version level).
For the time being, non-bootstrapping / not auto-updating Agents are the kludge to give you the same exact control over Agent upgrades.
This control comes with a cost: manual installation of Agent upgrades. If it helps you to avoid pitfalls, it is worth the cost for sure.

Cheers
G.

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Wouter C. avatar image
Wouter C. answered ·

Apperently our posts crossed each other (smile)

Seems you've covered the last missing pieces of the puzzle now.

I'm going to try this and keep you posted

 

W.

1 comment Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

X-actly

0 Likes 0 · ·
Wouter C. avatar image
Wouter C. answered ·

ok, figured out how to add a Native Module, apparently I somehow didn't notice the only possbile to do this in the 'Configure Native Module' window ( "register"), so now I can add the specific IIS agent module instead the default-bootloader–one.

I think this should cover my 1) in the previous post.

Still not sure about 2) though, the .NET agent, but maybe I'm again missing something very basic here

1 comment Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

I just edited my above post and added

If you configure a .Net app for instrumentation then in the Start menu > Dynatrace > Dt <version> > Dt .Net Agent Configuration tool it will use the non-bootstrap Agent.

That should drive home how to instrument an ADO.Net app in general and in particular. (with the modded reg.

Cheers
G.

0 Likes 0 · ·
Wouter C. avatar image
Wouter C. answered ·

Hi, somewhat further, but still stuck ...

I'm trying to figure things out here without the aid of the application/OS team, so sry if I'm clearly missing some basic knowledge here.

1) How do I make e.g. "C:\Program Files\dynaTrace\dynaTrace 6.1.0\agent\lib\dtiisagent7.dll" available to the 'Configure Native Module' menu?

The only dynatrace related dll's I can select (after having deleted the orignal dtagent.dll/64-dtagent.dll) are again the same 2 dll's: dtagent.dll/64dtagent.dll

Do I have to use the installer or something to make them available (I missing some kind of 'add native module' mechanism)?

2) I think once I've succeeded in implementing 1)  and dtagent.ini "nobootagent true" is set, there's still the .NET agent that will use the bootagent? What do I have to do to prevent this?

W.

1 comment Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Never be afraid to ask! And I know how hard it often is to get something from a different department!

I think we have a great product, but sadly we are not yet world champion in usability. Our CTO likes to say though that we do work on world dominance, also in ease of use. <g> (wink)

Sorry for the inaccurate description off the top of my head! There are two corrections:

  1. You need not remove the normally installed bootstrap Agent, simply name the non-bootstrap differently.
  2. When you select the IIS itself in the Manager (and not a site that you want to instrument), then you get Register, Edit and Remove buttons in Configure Native Module.
    Now register the 32 and 64 bit modules, but only configure (check-mark) them when you want to instrument all IIS sites.
    If you want just some, select the site, Configure Native Module in Modules and check the right one(s).

I just verified with our chief web app guy that you can configure both bitnesses for an IIS site and it will load the right bitness Agent!

So, this sums up the IIS web server and ASP.Net part.

For ADO.Net in two registry keys (32 and 64 bits) the file must be changed from dtagent.dll to dtagentcore.dll (as cryptically mentioned above – as I didn´t have any practical idea how to do it. Needed our chief .Net guy to actually work this out. Great if one has everybody handy a few doors apart. <fg> (wink) )

I will attach a registry-mod-file later today on the .Net Agent Configuration pages that handles that.

If you configure a .Net app for instrumentation then in the Start menu > Dynatrace > Dt <version> > Dt .Net Agent Configuration tool it will use the non-bootstrap Agent.

Finally I will create an extra non-bootstrap page with all the information from here in one place.

Thanks again for asking and helping to uncover all this!
G.

0 Likes 0 · ·
Wouter C. avatar image
Wouter C. answered ·

Günter,

 

I have to come back on this. I think I need some more details on how to do this if possible.

For linux/apache/php this seems pretty straightforward, the same goes for java, but on an IIS server things seem a little more complicated.

The masteragent property 'nobootagent" should be no problem, same as with a linux/apache combination, but how to change the dll used by the IIS modules and .NET agent, as there seems to be no clear way (GUI/config file) to modify this.

Are these registry settings (and which)?

Also, I can find a specific dtiisagent7.dll (to which I want to point my IIS module config once I've located it), but no clear equivalent for .NET

Can you help me any further on this, it would be very much apprectiated!

 

Wouter

1 comment Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Sure!
Fire up IIS Manager. If there are (bootstrap) Modules installed (that point to dtagent.dll) for the respective level (IIS or site), remove them and then Configure Native Module.

E.g. C:\Program Files\dynaTrace\dynaTrace 6.1.0\agent\lib\dtiisagent7.dll or lib64.

I think this should make the list complete.

As I said: I can relate and have already taken it way up. (We want to stay the best!)
Cheers and thanks for using Dynatrace!
G.

0 Likes 0 · ·
Wouter C. avatar image
Wouter C. answered ·

Hi Günter,

 

Thanks, this helps a lot, it's exactly what I was looking for!

It's indeed not a very convenient, as right now I have to prepare manually all agent configs that I don't want to risk to upgrade in an uncontrolled fashion, restart or wait till they are restarted, upgrade server/collectors to the new version, do my small scall upgrade by restarting the environment I'm focussing on, prepare the next environment by pointing the agents back to the default version etc ...but at least I have the option right now.

I'm also really interested in any updates concerning a possible implemenation of a  non-bootstrap upgrade equivalent of the "Fixpack selective roll-out" mechanism, so please keep me posted! (I also upvoted the RFE "Synced bootstrapped agent configuration" just to make this more obvious (smile))

 

Best regards,

Wouter

 

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Wouter C. avatar image
Wouter C. answered ·

Hi Guenter (Günter?) and others,

Thanks to you all for the reactions.

 

Short version how do I disable agent bootstrap, is it possible for alle types of agents in dynatrace version 6.0.0.7? Because right now I'm thinking It's  the closest thing I can find for what I'm looking for.


Long version:

My biggest concern in this case is not mostly dynatrace data loss, but rather the risk for stability of the applications itself when introducing new software (the upgraded agent) in our environments. Even after tests in each test environment, we're very well aware that we never can fully replicate all different kinds of traffic that pass through our production servers. We've had some major issues last year, so people tend to say "don't even try to look at my servers in that critical period". So dynatrace upgrades are quite impossible for certain environments in some periods. Unfortunately, these periods do not necessarily overlap for our different environments and this leaves very small (and far between) dynatrace upgrade windows which have to be used for all of our environments at once (because if we don't trigger controlled agent upgrades in a non-critical window where we have room to fix things that might happen unexpectedly, they might be triggered by a server restart in a critical period or just before and if this (however unlikely) would cause any problems,people would really really not appreciate it). So my idea was, after testing the upgrade in our test environment, to upgrade the dynatrace production server / collectors up front without necessarily taking into account all of the critical windows (which is already a bit of a stretch, because communication between the old agent and new collector might cause some unexpected problems, but you have to draw a line somwehere...) and to gradually roll out the new agent version whenever an opportunity presented itself for a specific environment. But that implies a means to control the upgrade of the agents. Hence my question.

Best regards,

Wouter

 

3 comments Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Hello Wouter,
sorry it took a couple of days, but the flu caught me on Tuesday. <sigh>

Thanks for the detailed follow-up description of your pain points. I appreciate your taking time, can relate to it and want to make sure it has the right priority here.

I immediately brought up your cause in the Monday sprint meeting in the Agent dev group where also our main product architect was present.

Naturally this isn´t new, we always strive to improve in all directions and there is something in the works, but it doesn´t hurt to bring it up again / stress it to put things in perspective. The natural way is to expand on the FixPack selective roll-out mechanism.
I promise I´ll keep a personal eye on it.

The short term solution for sure, if not the most convenient one, are the non-bootstrap Agents.
You can have any Agent non-bootstrap:

For the web server master Agent use NoBootstrap true in conf/dtwsagent.ini.

Generically speaking:

Use the respective specific .dll (Windows) or lib... .so (*NIX) instead of the dtagent.dll or libdtagent.so. (the bootstrapper)

Java and .Net dtagentcore.dll or equivalent .so.
Apache 2.4 e.g. dtapacheagent24.dll or .so
PHP 5.5 (32bit) on Windows 64 bit Dynatrace 64 bit installation e.g.
extension="C:\Program Files\dynaTrace\dynaTrace 6.1.0\agent\lib\dtphpagent55.dll"
instead of
extension="C:\Program Files\dynaTrace\dynaTrace 6.1.0\agent\lib\dtagent.dll"

We may not be exactly explicit about it in the documentation, because I guess PM would like the bootstrap Agents to be the rule. If you need advice for some Agent to non-bootstrap, please follow up.

Hope this helps in several ways!
Günter

0 Likes 0 · ·

Hi Guenter,

We are actually using a Collector Group. will the non bootstraped agent still fetch and cache the collector list for his failover or is it something that is only done by the bootstrap agent

regards

Olivier

0 Likes 0 · ·

Hi Olivier, just talked to engineering and he says that both, the bootstrapper and the core Agent fetch the Collector list, so failover should not be affected with non-bootstrap Agents.

Regards G.

0 Likes 0 · ·