RtmMemoryPerfind
Hello,
What does the metric RtmMemoryPerfind mean exactly ?
In this view, we have the RtmMemoryPerfind with 4 saturations (but the value is 22 % only) - for each satured period, the capture flow is incomplet (it miss a lot of operation and activity)
In my case, is there a memory saturation on the RTM ? Where can I change it?
Regards,
JLL
Jean-Louis,
The rtmMemoryPerfind metric shows the percentage of total memory that can be used by the AMD (data.mem.limit + SHM size) vs the amount actually used (normal memory allocation + SHM buffer usage). According to the chart you attached, the AMD is running out of SHM space - i.e. it is hitting the SHM buffer capacity and dropping packets - that's why you see a drop in monitored traffic during that time.
There can be two reasons for that:
- The AMD drops packets due to short bursts of increased traffic volume. In this case, you can solve the problem by increasing the SHM buffer size in rtminst.
- The AMD is overloaded (analyzes too much traffic) or there is some performance problem.
Regards,
Wojtek.
Hello Wojtek,
thank you for this explanation.
We have tested these 2 configurations :
| data.mem.limit (MB) | SHM buffer size (MB) | rtmMemoryPerflnd (MAX) | Data collected during the Max period | |
|---|---|---|---|---|
| Default configuration | 103212 | 20642 | 22% | about 50% |
| Test | 51606 | 32768 | 64% | 5% |
The total memory is 146GB.
We increased the SHM buffer size during a test period. But with this new configuration, the AMD lost almost all the packets and there were only 5% of data collected in the CAS.
which value do you recommand for the SHM buffer size?
I don't think that the AMD is overloaded, there are 6 Gbps on the 10 Gb NIC received by the AMD, but there are only 300 Mbps filtered (and treated by the AMD).
It appears that the Max of the rtmMemoryPerflnd is reached with a high activity of the SOAP.
Regards,
JLL
Hi JLL,
Just curious: the documentation says that the SHM buffer size should be 20% of the data memory limit or 4096 MB, whichever is greater..and never be greater than 5120 MB. The example in the documentation sets it to 5120 MB. Any reason you have the SHM buffer size set that high?
Thanks,
Harshal.
Hi Harsal,
For this environment, the default of the SHM Buffer Size is 20642 MB (proposed automatically from the rtminst - I use the standard command of the documenation as you mentioned) and the max is 51606 MB.
I think the AMD change the default and max SHM buffer size depending of the total memory (146 MB here).
Regards,
JLL
Jean-Louis,
Increasing the SHM buffer space past some value won't have any effect on performance (it may only delay sampling a bit). The fact of the AMD being overloaded is not related to NIC capacities. The 300Mbps that is being filtered and sent to analysis equals to more or less 300k packets per second - which is roughly the top traffic volume for all decodes (you can find exact traffic processing capabilities in the documentation). The AMD is expected to drop packets and activate sampling when reaching this limit (the exact limit depends on the decodes being used and the traffic profile).
Wojtek.
Hello Wotjek,
What is the max value for the SHM buffer size you recommend to configure in the rtminst ?
For this case, I don't think that the data collected has reached the limit. The AMD is a very large AMD (32 Proc, 146 GB Ram...), and a regular AMD can collect about 900 Mbps according to the documentation. When the limite of the rtmMemoryPerfInd is reached, there are max 10 MPkts by 5 minutes - the traffix does not exceed 33k packets per second!
regards,
JLL
Jean-Louis,
Usually one should use the value recommended by rtminst. Too large values may even increase the amount of dropped traffic (a large buffer allows a large backlog of packets to be accumulated, and when sampling starts the AMD must process all those buffered packets, further reducing performance and increasing sampling).
If you are experiencing drops/sampling at a traffic volume of 30kpps you should open a Support ticket. This apparently is a problem, but it is not memory-related (at least not directly). Determining the root cause would require anaylysis of AMD performance logs.
Regards,
Wojtek.
Very interesting. Thank you for this explanation.
The call is already opened. SUPDCRUM-5320
Regards.
JLL
