Unable to categorise tests as "Performance Tests" using .Net library
Hello,
I would like to know how can I categorise my unit test data as a "Performance Test" when I send it to the Dynatrace. I am currently using .Net library and nowadays my tests are only appearing in the "unit tests" section; I tried to look it up on the library documentation, though, I was not able to find this information. As far as I am concerned, Java library enables you to categorise your tests, but .Net library does not.
could someone help me ?
thanks!
@Wiktor Bachnik Where in our visual studio project do we add the call to StartTest task?
Hello Daniel,
This call is meant to be used in your build scripts. Currently we support MSBuild and NAnt with our plugins. If you're using other build automation tool or want to register the test run manually, you should use our REST API.
Best regards, Wiktor Bachnik
@Wiktor Bachnik @Daniel Pohanka
I am working with Daniel on Categorizing my tests as performance. We have successfully
- Added the DLLs to the Test Project
- Updated the Test Project file to include the StartTestTask
- We see the StartTestTask run and complete during a build.
- We get errors if we leave out parameters such as VersionMajor. This is verification that the StartTestTask is running appropriately.
However, dynaTrace does not pick up any of the parameters we set including Category. The tests still show up as Unit Tests with an autogenerated version number. Below is what we have included in the Project File.
<UsingTask AssemblyFile="dynaTrace.MSBuild.Task.dll" TaskName="StartTestTask" />
<Target Name="BeforeBuild">
<StartTestTask VersionMajor="1" VersionMinor="1" VersionRevision="0" VersionMilestone="Final" VersionBuild="1" Marker="Final Build" Category="performance" Platform="Windows 8.1" SystemProfile="LocalProfile" CustomProperty="$(CustomProperties)" > </StartTestTask>
</Target>
Hello Bill.
Do you see a line in your log file after running the StartTestTask that says:
StartTestTask: Setting property DT_TESTRUN_ID to value {UUID}
where {UUID} is some actual identifier?
If so, what do you see when you open the following URL in your browser:
http ://{YOUR_DYNATRACE_SERVER}:{PORT}/rest/management/profiles/{SYSTEM_PROFILE}/testruns/{UUID}
Where {YOUR_DYNATRACE_SERVER}, {PORT}, {SYSTEM_PROFILE} and {UUID} are replaced with proper values.
Do you see the same details of the test run you used when registering it? Is the category matching?
I suspect the test runner is executed from another process that doesn't inherit the DT_TESTRUN_ID environment variable set by the StartTestTask, that's why you see your tests in the default category (unit) with autogenerated version numbers.
Regards, Wiktor Bachnik
Hello Luiz,
I think I found the source of the problem. It looks like Dynatrace installation puts DynaTrace.Command20.dll library into Global Assembly Cache (GAC), and version that's being put there is wrong. It doesn't contain the IDTCommand61 interface and its implementation (verified that with ILSpy). The DynaTrace.Command20.dll included in the dynatrace-msbuild.plugin-6.1.0.7880.zip archive offers correct implementation.
Solution: Please try either replacing the DynaTrace.Command20.dll in GAC or removing it and using the local copy from our archive in your project.
Please let us know if that solved your problem.
Additional note: Remember that in order for the tests to be correctly reported, DT_TESTRUN_ID environment variable in the process executing your tests must be set to contain the test run UUID generated by the startTest() method. This is done automatically by the MSBuild and NAnt libraries, but you need to remember about it when using the command library directly.
Best regards,
Wiktor Bachnik
Hi Wiktor,
First of all, I appreciate your help.
Sencondly, I have tried the actions suggested by you, though It did not work for me.
The thing is, I had to remove all the previous versions of Dynatrace (Agent and Client) from my machine in order to get all DLLs from version below 6.1.0.7880 vanished from the GAC.
After that, I have got only the new version of the DLL installed in the GAC (Version 6.1.0.7880 ), however, I still receiving the same error message.
Could you help me ?
Hi Luiz,
One important question: did you remove the original DynaTrace.Command20.dll with 6.1.0.7880 version from the GAC as well?
The problem is that the dll distributed with Dynatrace 6.1 installation has exactly the same version string (6.1.0.7880) as the one we provide in the download of the MSBuild plugin, but it's missing the implementation of the required methods. They will look the same in the GAC dll list.
You'd need to remove the dll installed by Dynatrace first and then add the dll found in msbuild.plugin-6.1.0.7880.zip zip file.
In order to confirm that the dll is correct, you can use the ILSpy too (http://ilspy.net), please see how it looks like in my system:
Best regards,
Wiktor Bachnik
Great! I'm glad I could help you to solve the problem.
Best regards,
Wiktor Bachnik
Hi Wiktor, I am currently able to use the new msbuild library in my projects (6.1), though I am not seeing my unit test categorised as a "performance test" in Dynatrace Client. Perhaps, due to the upgrade of MSBuild library or because my updagrade from Visual Studio 2010 to 2013 (Which forces you to use another MSBuild.exe and QTAgent.exe) it is not working anymore. I still can send purepath data to the Dynatrace server, but the unit test data is missing to my view in client.
Just to clarify, I am showing the enviroment where things were used to work and now.
Enviroment Before(Where it was working) :
- Visual Studio 2010 (VS2010)
- MSTest.exe from VS2010 attached in Dynatrace Server Agent (5.6)
- QTAgent.exe from VS2010 attached in Dynatrace Server Agent (5.6)
- QTAgent32.exe from VS2010 attached in Dynatrace Server Agent (5.6)
- MSBuild Tasks for Dynatrace Version 5.6 (Test Automation and MSBuild)
- Dynatrace .NET Agent Configuration Version 5.6
- Dynatrace Client Version 5.6
Enviroment Now(it is not working) :
- Visual Studio 2013 (VS2013)
- MSTest.exe from VS2013 attached in Dynatrace Server Agent (6.1)
- QTAgent.exe from VS2013 attached in Dynatrace Server Agent (6.1)
- QTAgent32.exe from VS2013 attached in Dynatrace Server Agent (6.1)
- MSBuild Tasks for Dynatrace Version 6.1 (Test Automation and MSBuild)
- Dynatrace .NET Agent Configuration Version 6.1
- Dynatrace Client Version 6.1
Hi Luiz,
I'm sorry for the late reply.
Please check sensor configuration for the appropriate Agent group in your system profile settings. In the "Sensor Placement" configuration the ".NET Tests" sensor should be enabled. In the "Sensor Configuration" tab, for the ".NET Tests" sensor the "Capture" column value should be set to "active and start PurePaths".
Please confirm that you're using the settings I mentioned. If you still can't see your tests after those changes, please let me know if you can see the PurePaths matching the tests you're running in the "PurePaths" dashet (System Profile->Diagnose Performance->PurePaths).
Best regards,
Wiktor Bachnik
Hi Wiktor, I reached out the person who is responsible for this kind of configuration and he said it is all configured properly. Take a look at the images. Is something missing either way ?
Hi Luiz,
That looks good. Can you please check for the PurePaths in the PurePaths dashlet? Do you see anything there?
Also, if possible, could you provide the log file for the agent configured for MSTest?
Best regards,
Wiktor Bachnik
Hi, I can see the purepaths in Dynatrace Client. Where can i get the logs ? could you help me ?
Hi Luiz,
You can set agent logging path for each .NET agent configuration in the "dynaTrace .NET Agent Configuration" tool.
What testing library you're using to write your tests? VisualStudio Unit Testing? NUnit? Something else?
Also, are you using continuous session recording or are you recording each session separately?
Best regards,
Wiktor Bachnik
Hi,
I have written the unit tests on Visual Studio and they are processed by MSTest.exe (the same process used inside Visual Studio).
I am using continuous session recording, I mean, I start the recording, set up the information of the tests (like the category etc), run all the unit tests and after that stop the recording.
I might activate the logs today and let you know the results.
Thank you!
Hi Wiktor, another question...which would be the best configuration to use in Log ? I am currently using log level "info", but there are plenty of them.
Hi Luiz,
I'd set logging level to "Finest" - it gives most details.
I tried to recreate the problem today, but it seems it works correctly in my installation of Dynatrace 6.1. I instrumented MSTest and QTAgent32 processes and made sure that their agent names are used in agent mapping in my system profile, I also enabled the ".NET Tests" sensor for them. It seems you're at that stage as well since you can see the PurePaths correctly.
In order to have my tests reported in the "Performance" category, I registered a test run using REST API and got its UUID. Then I set the DT_TESTRUN_ID environment variable and executed MSTest. After a while, tests appeared in proper category. Here's how I did that in command line:
set DT_TESTRUN_ID=69d3284e-c3b5-4a0c-b938-0ba9603d45c1 mstest /testcontainer:TestProject1.dll
This can be easily automated using MSBuild, that would execute "StartTest" task before starting MSTest execution. Our plugin sets the DT_TESTRUN_ID environment variable in the context if MSBuild script so it gets automatically populated to MSTest.
So maybe the problem is that you're not passing the test run UUID in the environment variable? Please try doing that manually as in my example and let me know if that helped.
Best regards,
Wiktor Bachnik
Hi Wiktor, can you tell me the address for "start test" restful services ? I am using soap UI for tests purposes but I could not find the "start test" address in the APM Website.
Thank you!
Hi Luiz,
I'm very sorry for the late reply, I got sick and was not in the office last few days.
Our REST API endpoints are documented at: REST Interfaces for Test Automation
You're interested in sending a POST request to the /testruns endpoint to register your test run. Please note that our MSBuild plugin (and the DTCommand library we talked about previously) also uses this endpoint behind the scenes, it may be more convenient to you to use it instead of doing that manually.
Please let me know if you managed to register your test run successfully. If you still have problems, maybe I could set up a WebEx to assist you remotely.
Best regards,
Wiktor Bachnik
Another doubt, I have turned the log on as you told me. Which information do you think is important to observe in the logs to know if it is everything ok ?
The most important thing is checking if the agent reports receiving the test run identifier you set in the DT_TESTRUN_ID environment variable.
If in doubt, you can upload your log here.
Best regards,
Wiktor Bachnik
I did not find this information here in the logs, I am thinking about attaching them here. Just to attach the right files, which Logs do you want to take a look, the logs generated by MSTest.exe or by QTAgent.exe ?
In my environment, logs for QTAgent were relevant.
Regards,
Wiktor Bachnik
Hi Wiktor, very good news!
It is working now, I believe it was the lack of right QTAgent.exe (since vs 2013 has some of them in different versions) that were the problem. Now I can see my unit tests on "Performance Tests" section. Thank you for your help!
Hi, you are totally right, It is related to the MSBuild library.
I took a look at the latest version(6.1) of the library and I saw that now I have an option to set the category of the tests.
I might test this today!!
Thank you very much!!
Hi,
I assume you're referring to the MSBuild library (Test Automation and MSBuild)? The StartTest task should actually support a Category field where you can register a test run as a performance test. Which version of the library are you using?
-Wolfgang
Hi Wolfgang, I have tested it yesterday and I have got some problems while trying to use the new task called "StartTest".
I am currently receiving this error message :
Could not load type 'dynaTrace.Command.IDTCommand61' from assembly 'DynaTrace.Command20, Version=6.1.0.7880, Culture=neutral, PublicKeyToken=1e72a45839f59aeb'.
I am afraid I have to refer another dll beyond the two that comes with the zip file. What do you think ?
