How to understand Processing Status Report on CAS?
Hi There,
We often experience CAS processing delay. When I look at the Processing Status report, there is some confusion here.
Example:
From Delay Chart, I see zdata processing has 1 hour 42 min max deplay on 01/21.
From zdata processing chart, I see max processing time just 5 minutes.
We all know zdata file will be generated every 5 minute. So what's the actual delay on that date? How can I use those chart to narrow down the reason of delays.
Thanks
YC
YC,
I believe that your first chart is aggregated to 1 day resolution that causes this.
When you troubleshoot sample delay remember that:
- delay most often comes from processing time > 5 minutes,
- in most of the cases you should concentrate on zdata files,
- when troubleshooting zdata processing time check if:
- it depends on the time zdata file is downloaded from AMD (autogegotiation is recommended to be set on CAS-network-AMD communication interfaces),
- it depends (changes according to) to number of records inside the zdata
- when you exclude network problems and increase on number of records in zdata you best friend may become log\server.log,
- if you could not find there any particular reason for long processing of zdata by CAS contact support but first exclude that bottleneck if poor performance of CAS' DB using CAS Benchmark Tool.
- if you decide to contact support for a situation that CAS itself processes zdata slowly (but network and SQL are fast) it would be very helpful if you could hit http://CAS/DiagConsole#/diag/DUMP+STACK page during slow processing as many times as you can and grab ExportConfig.zip.
Also remember that you can edit default diagnostic reports/charts, modify them and save as your own.
Hi Yc
To get more insight into what the reason is you can explore the DMI and create additional graphs that can answer question such as number of files, files size, or processed records.
And as Adam says - the benchmark tool (don't run this during production hours and warn the SQL people that it will trigger alerts) will give you a better understanding whether you need to upagrade your SQL.
In release 12.3.1 it's very easy to add nodes to a farm/cluster to share the load as well so recovering from your current state doesn't have to be complicated.
Hi Adam and Ulf,
Thanks for response. Those are really helpful.
Are you sure http://<CAS>/DiagConsole#/diag/DUMP+STACK is the correct link? I tried it and got 404 error.
Regarding Processing time report, I have three questions here,
- The Average/Total and Max file processing time are same. Is that means the time process one file, or all files? How can I know the processing time for all files?
- If the above time means total processing time for all zdata files, they are all below 4 minutes, where is the delay from?
- Why this report count 2 file processed for one file from a AMD? You can see I list zdata files by file name. The column file processed show "2" for each file.
Thanks
YC
YC,
Ad.0. You should replace <CAS> with your CAS' IP or host name to make it work.
Ad.1. It's the same as I believe the resolution on your report is 1 period, averages are useful only if you aggregate bigger time range using resolutions other than 1 period,
Ad.2. It may come from the other file types processing. Can't tell more w/o server.log.
Ad.3. AMD serves one "native" zdata and the other from NFC process. You can count it as one.
Adam,
For http://<CAS>/DiagConsole#/diag/DUMP+STACK, I did use cas ip for <CAS>. It just can go <CAS IP>\diagconsole, then 404.
Thanks
YC
What is the IP and port that your CAS is running on? It should be visible in the browser when you use the CAS ...
