Restore a cluster from backup - test before migration and as a disaster recovery scenario

[desstef]

Hello, 

We are testing backup and restore option on a new test cluster(managed).  

https://www.dynatrace.com/support/help/shortlink/managed-cluster-restore#restore

Unfortunately the instruction above doesn’t state clearly whether or not we could "lose" and uninstall all nodes(3) at once and manage to restore from that point. 

Also it doesn't state that we will need a token so we can start the restore process if all nodes are gone, just one by one.

My question is if that is even an option, or we should be thinking of other disaster recovery scenario for the case that all 3 nodes are gone? 

[Iplinsky · Accepted solution]

you need to have at least one node running to initiate the restore process. You also need to have a valid cluster token that was generated when you created the backup. If you lose all three nodes and the cluster token, you will not be able to restore the cluster from the backup.

Therefore, it is recommended that you keep a copy of the cluster token in a safe place, and also have a backup plan for restoring at least one node in case of a disaster. You can use the same backup file to restore any node in the cluster, as long as you have the cluster token.

[radek_jasinski]

On the other hand, you might think about a snapshot on VMWare. Some of my clients prefer such a workaround.

Radek

[mrc15816]

@radek_jasinski were you able to verify if the method works? My understanding is Cassandra and ElasticSearch would break if we restore from a Snapshot as the versions would become out of sync with the other nodes. We did reach out to Dynatrace and they strongly recommended against the use and our backup teams were not found of the snapshot either as the image grows.

[mrc15816]

@Iplinsky we have multi region with multiple nodes and we were able to restore at our standby site without any issues, but you can only add node by node and the more data you have the longer the process it takes to come online. We have taken a different approach where for some clusters only backup the configuration and not the metric data so the restore process would be under 4 hrs. Have you looked at Managed (Premium) where you can have two cross site regions be active? it is little too expensive you have to be Managed Online. https://www.dynatrace.com/support/help/shortlink/managed-multi-data-center

All the best!