Rollback after upgrade of dynatrace server/collector/agents

Adding to the great details provided by James...

With regards to upgrade, from a high level, here are the implementation steps...(back up db, session store and config files; acquire the new license and uem before hand); disconnect the performance warehouse; deactivate the license and uem volume; stop the collector service, stop the server service; Install the new version for server and collector (as you can see the old version still exists); run the migration commands to port configurations from old version to new; bring up the new server service then the collector service; import the new license; connect to performance warehouse. If an application is restarted at this time the bootstrapped agents will be updated;

From a roll back strategy consider these points...once the new server service connects to the performance warehouse it will update the schema (if any) - so from a roll back point of view have a db backup that can be restored, you also want to back up the session store for restoring it. For server and collector service you can stop the current (new) service and start the old one (at this point you will have to restore the db and session store)...for the agent you can re-install the older version.

Hope this helps.

Srikar

Great feedback!

We do a variation of  James' B - we do a side-by-side with the server and collectors. So, rollback for those is straightforward by going back to the earlier versions it appears.

That leaves the question of the agents. Are you saying that when they "phone home" they will automatically go back to the previous version? Otherwise, we have hundreds of agents (potentially) that would have to be manually rolled back. Major ugh!

Hey David, we also have a strict change process that requires a back-out plan. We looked at a few options for ours since older versions of dynaTrace are less than friendly for these types of scenarios. We actually DON'T auto-update our agents. A long time ago in v4 we had an agent cause a performance problem with one system. Since then we were gun shy of course. So, we now roll out agents per machine/application.

In both instances you can lose session data if your transaction storage is on the same drives, or is backed up.

Scenario A - Install/Back-out using backup using bootstrap agents

1. Create a snapshot or backup of the servers, collectors, etc. to fall back on.
2. Migrate profiles and other data
3. Upgrade in place
4. Rollout agents
5. <<disaster strikes>>
6. Bring back old snapshots
7. Agents will phone home and get old version 
   1. This may or may not work depending on how far you're jumping in versions.

Scenario B - Install/Back-out side by side using bootstrap agents

1. Install servers, collectors, etc. on the same hardware using different ports.
   1. In most cases we just added 100 to them.
2. Copy profiles and other data
3. Roll out agents by changing the ports in the agent confg.
   1. Can be automated by editing registry entries.
   2. Can be done over time to "test in production" before committing
4. <<disaster strikes>>
5. Undo agent changes by setting ports back to original values
6. Uninstall new version once everyone is back home safe

Scenario C - Install/Back-out side by side using non-bootstrap agents

1. Install servers, collectors, etc. on the same hardware using different ports.
   1. In most cases we just added 100 to them.
2. Copy profiles and other data
3. Roll out agents by installing new version and changing the ports in the agent confg.
   1. Can be done over time to "test in production" before committing
4. <<disaster strikes>>
5. Uninstall new agents, install old agents
6. Uninstall new version once everyone is back home safe

As far as fixpacks go I think you can select which fixpack to roll out across the system so the back up would be to select the older fixpack version as far as major releases, I'm not quite sure.