Detected PA Injection failure with Openshift4

sytar_alex · ‎03 Dec 2020

Hello I have a Openshift4 cluster with CRI-O containerezation. Trying to instument some java application with dynatrace-operator. I see - dynatrace-agent successfuly deploy and connected to ActiveGate, but in logs I see error:

2020-12-03 13:50:15.845 UTC [43ff880d] info    [native] Generating group id data for process group: 10232776891962716058, group name: SpringBoot App *********-* took: 4212ms. (mmap read time: 0 ms)
2020-12-03 13:50:15.925 UTC [43ff880d] warning [native] Update metrics took: 5809 ms, collecting host metrics: 1 ms, collecting disk metrics: 123 ms, collecting network metrics: 106 ms, detecting processes: 1281 ms, reporting os runtime configuration: 0 ms, updating network agent metrics correlator : 0 ms, filtering docker bridges: 0 ms, host events generation: 1 ms, grouping processes: 4218 ms, handling blacklist: 0 ms, group events generation: 1 ms, updating aggregator: 4 ms, resource exhausted events: 1 ms, updating host info: 73 ms, deployment changes storing: 0 ms, logging monitored processes: 0 ms, closing child processes: 0
2020-12-03 13:50:15.925 UTC [43ff880d] warning [native] Timer task 'updateMetrics' took 5809ms to execute (which is more than half of the timer's period)
2020-12-03 13:50:15.926 UTC [43ff880d] info    [native] Detected PA Injection failure for PGI:10232776891962716058 (0x8e02205e1f436f9a), name:/tmp/*********.jar *********-* *********, process data: Process PID:                       3524072
    Process parent PID:                3524015
    Process executable:                java
    Process path:                      /usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/bin/java
    Working directory:                 /
    Process name:                      java
    Argv zero:                         java
    Command line parameters count:     15
        -Dfile.encoding=UTF-8
        -XX:+UseContainerSupport
        -XX:InitialRAMPercentage=30
        -XX:MaxRAMPercentage=75
        -XX:MaxMetaspaceSize=256M
        -XX:ActiveProcessorCount=1
        -XX:MaxRAM=1000m
        -XX:+UnlockExperimentalVMOptions
        -XX:InitialRAMPercentage=50
        -XX:MaxRAMPercentage=70
        -XX:G1HeapRegionSize=2M
        -XX:G1NewSizePercent=30
        -XX:G1MaxNewSizePercent=70
        -jar
        /tmp/*********.jar
    Physical path:                     /proc/3524072/root/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/bin/java
    Physical working directory:        /proc/3524072/root/
    Working set size:                  354381824
    Virtual memory peak size:          3874779136
    Previous virtual memory peak size: 3874779136
    CPU utilization:                   6.28976
    last CPU val:                      173
    Process virtual machine:           Java
    Java mainClass:                    /tmp/*********.jar
    Process loaded libraries (22) :    /tmp/hsperfdata_1000920000/15|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/bin/java|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/jli/libjli.so|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/libjava.so|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/libjimage.so|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/libnet.so|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/libnio.so|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/libverify.so|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/libzip.so|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/modules|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/server/classes.jsa|/usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/lib/server/libjvm.so|/usr/lib64/ld-2.17.so|/usr/lib64/libc-2.17.so|/usr/lib64/libdl-2.17.so|/usr/lib64/libgcc_s-4.8.5-20150702.so.1|/usr/lib64/libm-2.17.so|/usr/lib64/libnss_files-2.17.so|/usr/lib64/libpthread-2.17.so|/usr/lib64/librt-2.17.so|/usr/lib64/libstdc++.so.6.0.19|/usr/lib64/libz.so.1.2.7
    Observed measurement cycles:       1
    Group id params:                   7
        JAVA_JAR_FILE | *********.jar
        SPRINGBOOT_STARTUP_CLASS | App
        EXE_NAME | java
        KUBERNETES_CONTAINERNAME | *********
        KUBERNETES_BASEPODNAME | *********-*
        KUBERNETES_NAMESPACE | *********
        HOST_GROUP | OpenShift4
    Node id params:                    5
        JAVA_JAR_PATH | /tmp/*********.jar
        EXE_PATH | /usr/lib/jvm/java-*-openjdk-*-*.el*_*.x*_*/bin/java
        KUBERNETES_FULLPODNAME | *********-213-5jxrv
        KUBERNETES_PODUID | 14135a1b-5b89-486e-8294-a0b357d5492b
        COMMAND_LINE_ARGS | /usr/lib/jvm/java-11-openjdk-11.0.3.7-0.el7_6.x86_64/bin/java -Dfile.encoding=UTF-8 -XX:+UseContainerSupport -XX:InitialRAMPercentage=30 -XX:MaxRAMPercentage=75 -XX:MaxMetaspaceSize=256M -XX:ActiveProcessorCount=1 -XX:MaxRAM=1000m -XX:+UnlockExperimentalVMOptions -XX:InitialRAMPercentage=50 -XX:MaxRAMPercentage=70 -XX:G1HeapRegionSize=2M -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=70 -jar /tmp/*********.jar
    Container Data:
        Container ID:                           0a6cf3a912ea48be485d35ef78936d76c34ec25f00ef87434805282646d55cd9
        PID:                                    3524015
        Created:                                2020-12-03T13:49:48.900331312Z
        Image ID:                               registry.localhost.ru/dev/*********/*********/*********@sha256:b025943b0f5faed7eb5e2677c18beff2051c2824df11c2f8ca4415d986aeb7d0
        Image name:                             registry.localhost.ru/dev/*********/*********/*********@sha256:b025943b0f5faed7eb5e2677c18beff2051c2824df11c2f8ca4415d986aeb7d0
        Orchestration layer data:               KubernetesData:
            Container name:                        *********
            Base POD name:                         *********-213
            POD name:                              *********-213-5jxrv
            POD UID:                               14135a1b-5b89-486e-8294-a0b357d5492b
            K8s namespace:                         *********
            K8s pod Ip address:                    10.128.125.14
        HI Level Container Runtime:             crio


 , failure reason:Container injection failed
2020-12-03 13:50:20.114 UTC [43ff880d] info    [native] Received Throttler non active: 0
2020-12-03 13:50:21.636 UTC [43ff880d] info    [native] Update metrics took: 1522 ms, collecting host metrics: 1 ms, collecting disk metrics: 122 ms, collecting network metrics: 102 ms, detecting processes: 1284 ms, reporting os runtime configuration: 1 ms, updating network agent metrics correlator : 0 ms, filtering docker bridges: 0 ms, host events generation: 0 ms, grouping processes: 4 ms, handling blacklist: 0 ms, group events generation: 1 ms, updating aggregator: 4 ms, resource exhausted events: 0 ms, updating host info: 3 ms, deployment changes storing: 0 ms, logging monitored processes: 0 ms, closing child processes: 0

How to troubleshoot this problem?

Thanks

ChadTurner · ‎31 Dec 2020

@Alexander S. were you able to get this error resolved? if so what did you need to do to correct the issue?

-Chad