question

Larry R. avatar image
Larry R. asked ·

"Host or monitoring unavailable due to connectivity issues or server outage" alerts

We have a team in Dynatrace which regularly upgrades the Kubernetes version on their production and staging clusters. The process for this upgrade is to spin up new hosts with the new Kubernetes version and to destroy the old hosts with the old Kubernetes version. They did this last night and Dynatrace raised 2 problems due to the hosts being destroyed.

"Host or monitoring unavailable due to connectivity issues or server outage"

Of course, Dynatrace isn't wrong, the hosts were shutdown intentionally as part of the upgrade process. Another wrinkle is that sometimes this upgrade process is automatically triggered by Google, so we may not always be able to plan ahead.

Is there a recommended way to handle this circumstance?

Thank you!

configurationalertinghost monitoringkubernetesgoogle cloud platform
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Larry R. avatar image
Larry R. answered ·

We did find this today while looking into this problem. For those of you on GCP, this is one way around it - GCP Maintenance Window BETA

https://cloud.google.com/kubernetes-engine/docs/how-to/maintenance-window

It would still be a 4 hour period of blackout, but at least it's a start.

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Sebastian K. avatar image
Sebastian K. answered ·

If you plan it best way would be maintenance window I think. This is how we handle the same issue. Problem is with upgrade made by google. Question is if it is possible to have some trigger that may send curl to Dynatrace and start maintenance window?

Sebastian

1 comment Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Yeah we had the same thought, but as I and you pointed out it would still be an issue due to Google. I am trying to gather more information on how exactly a host is "destroyed" per say in that environment. I know that Dynatrace should recognize the difference between a clean shutdown or a dirty one with it alerting on the dirty one. It seems like there should be some way to tell the difference in this situation as well.

0 Likes 0 · ·

Space Topics

mobile monitoring dotnet synthetic monitoring reports iis chat kubernetes servicenow amazon web services mysql mainframe rest api errors cassandra dashboard oneagent sdk cmc application monitoring openkit smartscape request attributes monitoring developer community user tagging log monitoring services ufo syntheticadvisory activegate ip addresses auto-detection high five award oracle hyperion webserver uem usql iib test automation license web performance monitoring ios news migration management zones index ibm mq web services notifications sso host monitoring knowledge sharing reports browser monitors java hybris sap vmware maintenance window user action naming javascript appmon ai synthetic classic availability tipstricks automation extensions diagnostic tools session replay permissions davis assistant faq documentation problem detection http monitors server easytravel apdex aws-quickstart network docker tags and metadata cloud foundry google cloud platform synthetic monitoring process groups account usability dynatrace saas gui paas openshift key user actions administration user actions postgresql synthetic locations oneagent security Dynatrace Managed user management custom python technologies mongodb openstack user session monitoring continuous delivery citrix configuration alerting NGINX action naming linux nam installation masking error reporting database mission control jmeter recorder apache mobileapp RUM php threshold azure purepath davis scripting agent aix nodejs android