DCRUM - Unique/Weird Trend in Metrics
Hi All,
I have been implementing DCRUM for a client, and been doing discovery for the monitored traffic. While doing that, I saw a mismatch in number of requests in two backend servers which the 'application' servers talk to. To get a better picture, I created a chart with 'Number of Requests' for a server, and separated the chart by the client IP.
IPs 172.31.94.220 and 172.31.94.221 have similar load, and talk to this IP 10.4.12.28. In theory, the load should be similar, however I see that for any given 5 minute interval, there is load from either 94.220 or 94.221. I have attached the reports. One of the report, I have separated the charts, and you can see the gaps between them. In another chart, I overlaid the previous two charts with different colors, and it perfectly fills all the gaps.
Bottom line, for any given 5 minute period, DCRUM does not capture traffic from both IP's. It just does either one, and tends to be higher for 94.221.
Has anyone seen something like this before or would know why this would be happening? This behavior is consistent for other backend services too.
Please help.
Thanks,
Mohammed
Hi Mohammed
What release are you on?
How many AMD's?
How do you feed the AMD's? Is it by Netflow, SPAN port or by a passive TAP?
Do you pass the traffic through any device before arriving at the AMD's?
What decode are you using?
Hi Ulf,
We are using 12.1.1, and have total 4 AMDs. We have distributed two AMDs in each data center, as Active-Standby.
Customer is using VACL's to span the data from switches to Gigamon, and then to the AMD. The traffic is sent to the Gigamon, and they filter using VLAN's. At the Gigamon, they filter using IPv4, and only allow relevant IPs.
I see this behavior with Generic with Transaction, XML, and HTTP decodes.
I saw the zdata and vdata files, and have the IP addresses, however they do not show up in CAS.
-Mohammed
OK - Is that 1 Gigamon?
Do the traffic to the 2 hosts go to different AMD's?
I am suspecting that. The second AMD is supposed to be a Stand by one, and it was not supposed to get traffic. We did not monitor that, and it was discovered that the second AMD's link was down.
The link was turned on again, and I can see data going to it, however the network engineer says that the Stand by data from the switches is going to that port.
The answer to your question, I see traffic going to two different AMD's, but in theory it should be Active-Standby, unless there is a mis config with the data feeds.
So - without knowing all details I think you might suffer from deduplication running a bit wild.
The deduplication alters between (at it's own preference) the probes and that is why you only see one host at any given time.
You can either do some research and correct the standby mechanism or the AMD config, or log a ticket and the support will help you go through your config.
Welcome the
Dynatrace Community Forums
Check out the Forum User Guide and Forum Guidelines to learn how to get started.Community Member of the Month
October 2017
Announcing Dynatrace's Community Member of the Month for October 2017, Ram S.! Click here to read more! 