question

Dominik P. avatar image
Dominik P. asked ·

Sending PureLytics stream to Splunk

Following up on last weeks Performance clinic about the PureLytics stream and the heatmap, I would like to share two ways how to connect the PureLytics stream to Splunk.

1. Integration with Logstash

Splunk is perfect for watching log files. Therefore, one very easy approach is to set up a component that receives the PureLytics data (e.g. Logstash) and dumps it into a file and have Splunk pick it up from there.

Logstash configuration

input {
  http {
    host => "127.0.0.1"
    port => 8080
  }
}
output {
  file {
      path => "/opt/splunk/purelytics/purelytics.log"
      codec => line { format => "%{message}" }
  }
}

Splunk inputs.conf (forwarders)

[monitor:///opt/splunk/purelytics/purelytics.log]
disabled = false
index = dynatrace
sourcetype = Dynatrace

Splunk props.conf (indexers)

[Dynatrace]
BREAK_ONLY_BEFORE =
DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 13
NO_BINARY_CHECK = true
TIME_FORMAT = %s%3N
TIME_PREFIX = "startTime":
disabled = false
kv_mode = json
TRANSFORMS-nullqueue = startswithindex
SHOULD_LINEMERGE = false

Splunk transforms.conf (indexers)

[startswithindex]
REGEX=^{"index":
DEST_KEY=queue
FORMAT=nullQueue

2. Integration with Splunk HEC

Splunk also offers a component called HTTP Event Collector that can receive the JSON data directly. The only requirement is to add the authentication token to the generic HTTP endpoint URL that gets configured in AppMon (https://localhost:8088/services/collector?token=46...). See http-event-collector-in-splunk-more-super-powers.pdf for details.

Splunk HEC inputs.conf

[http]
disabled = 0
enableSSL = 0
 
[http://purelytics]
disabled = 0
index = dynatrace
indexes = dynatrace
sourcetype = Dynatrace
token = xxx

Prior to Splunk 6.5 a specific Authentication header had to be set. The PureLytics stream only supports Basic Authentication, therefore as a workaround an Nginx proxy server that forwards the data can be used to add the authentication header.

Nginx config

server {
    listen 8081 default_server;
    server_name localhost;
    location / {
        access_log off;
        # only accept HTTP POST
        if ($request_method != POST) {
            return 403;
        }
        # pass the auth token in the header
        proxy_set_header Authorization "Splunk xxx";
        proxy_set_header X-Splunk-Request-Channel "xxx";
        proxy_pass http://localhost:8088/services/collector/raw;

Note:

Using AppMon 6.5 or later will allow you to configure the PureLytics stream to send the data to a generic HTTP endpoint. This will result in the data being sent directly to the provided URL (including query arguments) rather than appending “/_bulk” to access the Elastic Search bulk interface. Also the AppMon server will only validate if the POST requests return with a valid 200 code rather than parsing the response and expecting a valid Elastic Search response.

purelyticssplunk
splunk-logstash.png (15.1 KiB)
splunk-hec.png (8.4 KiB)
8 comments
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Attempting to use Purelytics Stream with Splunk HEC as described above (#2).

The connection to HEC using the Business Transactions Feed works, but the 'Purelytics Stream' throws an HTTP 405 error (using Generic HTTP Post option).

I'm using the http://splunk-server:8088/services/collector/raw?channel=xxxx as the URL with basic authentication via the HEC token as the password value. This works from curl & the Bus. Trans. Feed, but not Purelytics Stream.

What am I missing?

0 Likes 0 · ·

you cannot use basic authentication here - you have to provide the authorization token as query argument in the form "token=1234...".

This is only supported in Splunk 6.5 or later. If you are using an earlier version, you should set up a proxy server that adds the authorization header.

0 Likes 0 · ·

Thanks for the response. Actually, with Splunk you can use Basic Authentication by passing the token value in the password field. This is the method that is working for me with the Business Transaction Feed, it just doesn't work with the PureLytics Stream.

0 Likes 0 · ·

Just an update for future site visitors. I've opened a tech support case on this one to support the same methods of access for both BT Feeds and PureLytics Stream.

0 Likes 0 · ·

Received word back from Dynatrace support. This is working as designed. PureLytics Stream appends a "/_bulk" to whatever URL you specify in AppMon.

The support engineer filed an enhancement request to make "Generic HTTP Post" actually be generic :(.

0 Likes 0 · ·

I'm using Splunk 6.5.2 which now supports Basic Authentication to the HTTP Event Collector (details here: http://dev.splunk.com/view/event-collector/SP-CAAAE7F#usinghttpeventcollector ).

It works fine from the command line using:

curl -k -u "x:B8413BA0-00A0-4CA4-8B7C-CC61F5EE5191" http://192.168.0.105:8088/services/collector -d '{"sourcetype": "mysourcetype", "event":"Hello, World!"}'

When I try to setup a PureStream Generic HTTP Post using Basic Auth and the same token, same URL, I receive an HTTP 405 Error in Dynatrace.

@Dominik P. - any suggestions?

0 Likes 0 · ·

Hey Tom were you ever able to get this working? I'm having the same issue as you here.

0 Likes 0 · ·

@Andrew C. - sadly I did not. @Dominik P. - any suggestions?

0 Likes 0 · ·
Andreas G. avatar image
Andreas G. answered ·

THANK YOU Dominik for such a great writeup!! This is very helpful for folks that want to get started with leveraging Dynatrace data in Splunk.

Andi

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Marion R. avatar image
Marion R. answered ·

What is the resolution to the 405 response code

1 comment Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

It seems that this occurs when using basic authentication. You can try to provide the token as a query argument instead of basic authentication.

0 Likes 0 · ·
Emerson M. avatar image
Emerson M. answered ·

Hi,

I have been using feature provided by Appmon to connect to splunk through Flume since a while and so far, it has been working very good.. however, I would like to understand if PureLytics stream has more benefits than using Flume to stream the data to splunk

Thanks

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Dominik P. avatar image
Dominik P. answered ·

You should be able to simply add the token as a query argument (like described by splunk here on page 25: https://conf.splunk.com/files/2016/slides/http-ev...

Best regards,

Dominik

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Jonathan L. avatar image
Jonathan L. answered ·

Hi,

In AppMon 7.0, with the new Purelytics stream supporting "Generic HTTP Post", will it be possible to make use of this function to send data straight to Splunk?

I have a customer who is having problems trying to get connectivity to the Splunk server as a token is required to be passed in the URL. What is the format of the URL that needs to be passed in the screenshot attached?

Also, this should not require a webserver as an intermediary to consume the HTTP requests right?

Best Regards,

Jonathan Lim


Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Ari P. avatar image
Ari P. answered ·

Hello @Dominik P,

We were able to follow these approaches and set up the Splunk Feed (we don't use a HEC here but have an equivalent which works).

However, when we feed the data into Splunk, it is not able to extract all the fields. We think its the multiple separate 'blobs' that are next each other (location, client details for example) so its unable to parse the JSON.

This is what our entries in Splunk look like ->

We were hoping it would be able to parse the JSON and index the fields (kinda like in this doc -> https://community.dynatrace.com/community/download...) but doesn't look to be the case.

Any advice on ways around this?

Thanks,

Ari


purelyticsfeed.png (206.3 KiB)
4 comments Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Hi Ari,


It looks like this is a Splunk parsing issue. Whatever is receiving your Splunk feed is prepending timestamp and source IP before each json event.
"2017-04-20 12:38:24.19561300+0000 10.239.144.147 "

When data arrives at the Splunk indexer, the indexer needs to see JSON as valid JSON, from "{" to "}". As-is, it looks like there is extra text beforehand, and so Splunk will just use that as a timestamp and treat the rest of the JSON as a simple string of text.

If you can't stop your software (or whatever the middleman is before Splunk) from adding this prepended data, you can get around this by adding a SEDCMD option to your props.conf. SEDCMD can remove/replace text before the text gets ingested. Then when the indexer sees valid JSON from "{" to "}", Splunk will properly extract the JSON fields for you.

You will need to remove all of that prepended timestamp and source IP using regex:

SEDCMD-removeprepend = s/\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.[^\s]+\s\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s//g

You can also find a lot of good resources on the Splunk communities about how to work around this kind of issue.

0 Likes 0 · ·

Hi Aaron,

Thanks for your response. That was actually our first thought as well so the Splunk guys did an exclude for that prepending timestamp and source IP but it still didn't work.

I think the issue is that we have two closed JSON events right next to each other and that's what is screwing it up. Specifically the index event. You'll notice that before the event that starts with "serverID", there an event closing before it (the one with the indices per time frame).

When I run that text through a JSON extractor, its stops extracting after the first event (the one with the index). If I delete that event, and use the extractor again, it seems to be work great and extract all the events (like in the documentation).

I know that index event is created for every month but we don't really plan on using it for our Splunk analysis. Is there any way for us to exclude that event from the stream itself? If not, any other ways around it you can suggest?

In case you want to try that out the bulk feed, here is one of the feeds -> splunk-purelytics-feed.txt

Thanks for your time,

Ari

0 Likes 0 · ·

Hi Ari,

I recall running into this before with the "index" events coming through but their presence didn't obstruct the creation of JSON events. My intuition says there is probably something simple which isn't configured properly or something extra which is gumming up the works.

That being said, I think you can just keep parsing the data until you get a good "parse template" through the combination of SEDCMD, LINE_BREAKER, etc. Someone else might have a more elegant solution which doesn't involve this much text manipulation. But here is a possible solution.


Using your last attachment as my "raw data", I came up with a sourcetype which successfully parsed your data file into some good-looking JSON events. This was performed on a 6.5 Splunk test machine.

The first important line is a SEDCMD which removed the index JSON events.
Next, it wasn't line-breaking properly at each "serverID" entry, so a LINE_BREAKER option fixed that.

Hopefully that helps!

[purelytics3]
INDEXED_EXTRACTIONS = none
NO_BINARY_CHECK = true
category = Application
pulldown_type = 1
description = Sourcetype to format incoming Purelytics data into JSON key-value pairs
disabled = false
MAX_TIMESTAMP_LOOKAHEAD = 13
SHOULD_LINEMERGE = false
LINE_BREAKER = }(){\"serverID
TIME_FORMAT = %s%3N
TIME_PREFIX = "startTime":
SEDCMD-removeindex = s/{\"index\":{\"_index\".+?}}//g

0 Likes 0 · ·

Hi Aaron,

This is some really cool stuff! I just showed this to the Splunk guys at my account and they are now working on trying to get this implemented. Will let you know if we run into any issues.

Thanks again, really appreciate it!

0 Likes 0 · ·
Tamir R. avatar image
Tamir R. answered ·

Hi,

Did anyone succeed using nginx server option?

I'm using splunk enterprise 6.5.1.

Regards,

Tamir

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

aftab a. avatar image
aftab a. answered ·

Thanks @Tom M. I will go with ngix server option and then switch once this feature is available in splunk enterprise

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

Tom M. avatar image
Tom M. answered ·

@Aftab a. - This was a "future feature" discussed at theSplunk .Conf2017 conference. It is now available, but only in Splunk Cloud releases today. Here's the link to the what's new notice: http://dev.splunk.com/view/event-collector/SP-CAAA...

Note the comment at the top of the page:
"In the current releases of Splunk Cloud and Splunk Light Cloud, the following feature is new: ..."

Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

aftab a. avatar image
aftab a. answered ·

has anyone tried https://localhost:8088/services/collector?token=46... in splunk 6.5. I am getting 404 for this URL.

I tried posting to https://localhost:8088/services/collector/event?token=46931F1C-352C-4DF6-820C-F2689CF88494 but I keep getting "Token is required" error.

1 comment Share
10 |2000000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 50.0 MiB each and 250.0 MiB total.

From http://dev.splunk.com/view/event-collector/SP-CAAAE8Y the Set allowQueryStringAuth to true was needed

0 Likes 0 · ·