We have a team in Dynatrace which regularly upgrades the Kubernetes version on their production and staging clusters. The process for this upgrade is to spin up new hosts with the new Kubernetes version and to destroy the old hosts with the old Kubernetes version. They did this last night and Dynatrace raised 2 problems due to the hosts being destroyed.
"Host or monitoring unavailable due to connectivity issues or server outage"
Of course, Dynatrace isn't wrong, the hosts were shutdown intentionally as part of the upgrade process. Another wrinkle is that sometimes this upgrade process is automatically triggered by Google, so we may not always be able to plan ahead.
Is there a recommended way to handle this circumstance?
Answer by Larry R. ·
It would still be a 4 hour period of blackout, but at least it's a start.
Answer by Sebastian K. ·
If you plan it best way would be maintenance window I think. This is how we handle the same issue. Problem is with upgrade made by google. Question is if it is possible to have some trigger that may send curl to Dynatrace and start maintenance window?