question

chintan s. avatar image
chintan s. asked ·

Disaster Recovery setup

Our client wants to setup DR for dynatrace. Is it possible? If it is, please guide us on setting it up.

appmonconfigurationproduction
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Florent D. avatar image
Florent D. answered ·

Hi

there is plenty of good information in the PDF further down the thread on the link I suggested so I'm not sure what else you are expecting.

Alternatively, you can look at our migration guide which provides all the steps for moving from one server to another. You can use the guide even if you are not upgrading from one version to another.

https://community.dynatrace.com/community/display/...

Are you trying to achieve a manual or automated fail-over?

The PDF talks more about automated steps and the migration guide is a manual procedure.

The process is almost the same really:

- Make sure you have an installation ready on your DR server but it mustn't be running when the primary is active. This is because 2 servers cannot connect to the same data warehouse at the same time.

- Migrate the configuration files listed in the doc or use DTMigration.

- Start the server

- import a license

If the IP address of your DR server is different, then you will have to re-point your collectors to the new machine. This is why the PDF mentions using a VIP (virtual IP address) to hide this away from the collectors. I don't believe that a DNS change will work as java caches in-definitively the value until the process is bounced (I haven't tried it yet with Dynatrace but I have seen it happen with other applications).

Regards

Flo

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Florent D. avatar image
Florent D. answered ·

Hi

yes it can be done with bit of work.

Have a look at this link:

https://answers.dynatrace.com/questions/151655/hig...

Regards

Flo

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Rob V. avatar image
Rob V. answered ·

Hi Chintan,

This design is getting beyond what we can reasonably do in a forum post, but let me try to answer your questions.

First: I am going to assume that your collectors are in the same locations as your app servers, and that you will use whatever strategy you are using for the app servers, for the collectors. So to answer your points above:

1) yes, it's possible to implement the layout that you have in the picture. Two data centers; one is the active primary and the other is the DR site, ready to turn on in case the primary goes down.

2) Yes, you still need the VIP for the server. Clients point to the server, so that needs to be seamless. Collectors also point to the server. Imagine you have the situation where the collectors are in a different data center with the app servers, and they don't go down but the server does and is pushed to the DR site. Everything has to be able to keep working without any configuration or naming changes. The collectors would be defined to connect to the VIP so that will be transparent for them.

3) The VIP is normally handled with DNS. Today it points at this IP/server, tomorrow it points at the DR IP/server.

4) Licenses are machine-locked, so yes you will need two licenses. There are special licensing agreements for this situation so it should not cost you any more money as long as you agree that only one site will be active and handling agents at any given time.

5) I do not have a list of files that need to sync. To be safe you could sync the entire DT_HOME and session store. The exception to that (as in point #4) is the license file. There needs to be a unique license in each location.

Note: if the collectors are in the same data center as the server, then you have some more planning and thinking to do. If you don't use collector groups. you will need a VIP per collector (since the agent refers directly to a collector by name or IP in the injection connection definition). The VIP will make sure that things keep working even when you switch to the DR data center; the VIP will just resolve via DNS to the relevant collector. If you are using collector groups: I haven't thought that through. Probably the same deal with a VIP per collector, but I haven't done a detailed analysis.

HTH,

Rob

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Rob V. avatar image
Rob V. answered ·

Hi Chintu,

There are no specific DR features or functions built into the product. If you want a detailed set of instructions for successful DR I would suggest that you search that out from a DR specialist.

My customers that have successfully implemented DR (as opposed to local HA active/passive failover) have had the following pieces in place:

  • Two separate data centers
  • A complete copy of the protected hardware in each data center. That would include the server, storage, database server for the Performance Warehouse, and any collectors that are in that same data center. Note that the collectors are designed to be physically close to the agents that they serve, so collectors may not be in the same data center as the server, storage, and PW. The DR strategy for the collectors is likely to follow closely the DR strategy for the monitored application servers
  • Approved licenses that allow for a hot second system to be running (with agents reporting to only one data center at a time)
  • Real-time disk syncing from one DC to the other for things like session store, config files, caches, etc.
  • Real-time database syncing from one DC to the other for the Performance Warehouse
  • A VIP implemented that can be switched to point from one DC to the other in the event of a required failover

As with any backup/recovery/failover scenario, testing and regular use is key to ensure that things work. You should consider routine "switches" from one DC to the other (perhaps once per month, or once per week) to make sure the process is smooth. It should not matter which DC you are running from if your DR is implemented properly.

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Praveena P. avatar image
Praveena P. answered ·

Hi Rob,

In the previous comment you told that "the collectors all will talk to the DT server (e.g. "DT_Server"), so DT_Server will need to be a VIP that can be changed if the data center goes down." How should we setup VIP and can you plaese tell us the steps to do that?

Thanks

Parveena

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Rob V. avatar image
Rob V. answered ·

Hi Chintan,

Yes, your setup is a good disaster recovery setup, having a complete replica in another data center.

You will need a VIP for all of your servers (DT server and each DT collector machine). If an agent is configured to talk to "DT_Collector_1" it will need to be able to do that no matter where the collector lives so DT_Collector_1 (and all collectors) will need to be a VIP that can be changed on the fly if the main data center goes down. Similarly, the collectors all will talk to the DT server (e.g. "DT_Server"), so DT_Server will need to be a VIP that can be changed if the data center goes down.

I am not a network guy but my understanding is that VIPs are just implemented in DNS. Talk with your network person about their best practices for doing this.

Rob

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

chintan s. avatar image
chintan s. answered ·

Hi Rob,

1) My agents, collectors & servers are in the same data centre say DC. I have also created replica environment in DR with agents, collectors & servers all placed in that same DR data centre. In this case how to implement DR using VIP?

2) As mentioned in point #3, Should VIP server be placed in DC or DR? If we place VIP server in DC then if DC goes down so VIP also goes down. please suggest.

Thanks Chintan,

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

chintan s. avatar image
chintan s. answered ·

Hi Rob,

1) We need to implement the attached architecture for DR. Is it possible to do so? Please note that both DC & DR is in separate data centre.

2) Since DR is manual intervention (it is going to be manually up when DC goes down) so how VIP will work in that case? Does VIP is required in this case?

3) Also if VIP is implemented, where to place VIP? DC or DR?

4) How licensing will work in this case? Do we need to purchase separate licenses for DC & DR?

5) What all files needs to be sync with DR? Please provide the list of files.


dr-architecture.jpg (170.5 KiB)
Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

chintan s. avatar image
chintan s. answered ·

Hi Florent,

It doesnt have helpful info regarding Disaster Recovery procedure. We would like to have the steps or procedure to follow for setting DR for dynatrace.

Regards,

Chintu

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.