Possible bug in GenerateAllureReport@2 Task

[Xetoris]

I've been trying to debug why our Allure Report extension in ADO has started showing an error screen saying there was no report published. We use this at my work and noticed sometime in the last several weeks (sorry I don't have an exact date), the report stopped showing up.

I did some research on it today and noticed a couple of similar sounding threads, most notably this one:
https://github.com/orgs/allure-framework/discussions/3483.

We seem to be having similar symptoms. I unfortunately can't easily generate an example repo w/ ADO integrated (and I'm not allowed to do anything from the company end). So let me describe the symptom as best as possible:

• We use ADO for our build pipeline.
• We have installed the Allure Report extension. I had our Ops team confirm its on the latest version.
• We are using Playwright w/ NodeJS. We updated to one of the latest Playwright builds (1.62 something) and the latest version of the Allure reporter package.
• The Playwright configuration doesn't specify much other than just enabling the reporter.
• I confirmed there is no Allure RC file in the repository.
• We run the build successfully and have confirmed the reporter is putting out JSON files.
• The PublishAllureReport@2 logs don't show much at all. It looks almost empty except some generic "Choosing Node20" and "Finished" type logs. Nothing talking about finding results or generating reports. No errors or warning logs.
• The Allure Report tab on the build info screen loads and shows an error.

Here is a screenshot of the error:

Now I did some debugging based on the suggestions I saw on the similar threads.

I included the task mentioned in #3483 about logging matching json result files to the console (and Build Logs). This output the following:

/agent/_work/1/s
/agent/_work/1/s/allure-results/1f7f4aba-8c67-41e1-bc09-1d233a88c9e3-result.json
/agent/_work/1/s/allure-results/3354409c-6444-4e0f-bcfa-c3487bca9602-result.json
/agent/_work/1/s/allure-results/6ca223cd-8b7c-4d7a-a98e-cc3a7ae59a06-result.json
/agent/_work/1/s/allure-results/64b229df-92f2-4be4-9868-50d9366713cb-result.json
/agent/_work/1/s/allure-results/a9ec655b-1439-4108-a5b2-9ddcf385e0e9-result.json
/agent/_work/1/s/allure-results/74052018-9641-4c82-9297-c250e7bf2053-result.json
/agent/_work/1/s/allure-results/c01ad350-f674-4e94-9fc3-edc82e858f94-result.json
/agent/_work/1/s/allure-results/cdead555-6a74-47b8-b967-5eeae9c3d608-result.json
/agent/_work/1/s/allure-results/a38b6e63-41bc-4988-98fa-05f6090be942-result.json
/agent/_work/1/s/allure-results/d442d41f-7ead-4930-8c52-25a3ee83588d-result.json

I also confirmed that our inputs to PublishAllureReport@2 task specify the full path to the /allure-results/ directory, not a relative path.

Neither of these seemed to have an effect. So on another thread I saw to try running the build with "System Diagnostics" enabled. I did that and notice there was extra output on the PublishAllureReport@2 task. These two lines caught my eye:

##[debug]testResultsDir=/agent/_work/1/s/allure-results

##[debug]reportDir=/agent/_work/1/s 

Now "testResultsDir" lines up with the value I'm setting for the parameter in our pipeline yaml. But we don't specify "reportDir" at all. I couldn't even find a reference to it anywhere in our repository. It's weird that that's populated. It might also explain why the task doesn't find our json files. According to the Allure docs, that parameter having a value causes testResultsDir to be ignored.

But how is it getting set? I don't know, so I tried doing another run but this time I included another parameter: reportDir: '' in an attempt to force the value to blank out. But the logs showed the same two values.

I think that reportDir parameter having a value at task runtime, even though I don't specify it anywhere, is the cause of our symptom. Is there any way to validate that the GenerateAllureReport@2 task isn't somehow defaulting that value?

I apologize that I can't provide you full logs or an example build. Please let me know if there are any other useful details I can provide. Happy to give everything I can.

Thanks for your help!

===

Addendum (per @lMysticl 's suggestions)

I captured some more information that could help provide more insight.

Normal Run Output

Here is an example from what I'm calling the "normal run", everything wired as I normally would. I did add the result file matching query and System Diagnostics for this run:

Pipeline yaml

  - bash: |
      pwd
      find "$(System.DefaultWorkingDirectory)" -maxdepth 4 -type f \
        \( -name '*-result.json' -o -name '*-container.json' \) -print
    displayName: Locate Allure result files
    condition: always()

  - task: PublishAllureReport@2
    displayName: Publish Allure Report
    condition: always()
    continueOnError: true
    inputs:
      testResultsDir: '$(System.DefaultWorkingDirectory)/allure-results'

File Matching Task Output:

/agent/_work/3/s
/agent/_work/3/s/allure-results/397a0479-c937-4959-8f42-2770e83f8ff6-result.json
/agent/_work/3/s/allure-results/4c07cf5f-1eb9-4028-b93f-f75f3cc13c22-result.json
/agent/_work/3/s/allure-results/2e472dd9-1f57-4c86-940e-cce06d3f255c-result.json
/agent/_work/3/s/allure-results/4c3f3a41-5ff1-422b-9979-1340d30b85e6-result.json
/agent/_work/3/s/allure-results/b3455df8-7c89-4c7b-83e8-2b753c4a472b-result.json
/agent/_work/3/s/allure-results/67d9b1e1-9c01-4b96-a052-1ebb09b4ec78-result.json
/agent/_work/3/s/allure-results/1a03529d-ca5a-474d-9ff1-0e44b5d90f76-result.json
/agent/_work/3/s/allure-results/d6ef7084-fb3f-43fe-9fd5-0fd0b80c11f6-result.json
/agent/_work/3/s/allure-results/bf649b69-d695-47f4-bd87-0e5aae1606a9-result.json
/agent/_work/3/s/allure-results/c7eae63f-d628-4206-bf16-c0ae482e389d-result.json

Allure Publish Task

##[debug]Evaluating condition for step: 'Publish Allure Report'
##[debug]Evaluating: always()
##[debug]Evaluating always:
##[debug]Result: True
Starting: Publish Allure Report
==============================================================================
Task         : Publish Allure Report
Description  : Task for generate and publish Allure Report
Version      : 2.2.5
Author       : Qameta Software Inc.
Help         : Generate and publish Allure Report
==============================================================================
##[debug]Using enhanced node selection path for handler node resolution.
##[debug][Host] Starting node version selection
##[debug][Host] Handler type: Node20_1HandlerData
##[debug][Host] Host glibc compatibility - Node24: True, Node20: True
##[debug][Host] Checking strategy: CustomNodeStrategy
##[debug][CustomNodeStrategy] No custom node path found
##[debug][Host] Strategy 'CustomNodeStrategy' cannot handle this context
##[debug][Host] Checking strategy: Node24Strategy
##[debug][Node24Strategy] EffectiveMaxVersion=20 < 24, skipping
##[debug][Host] Strategy 'Node24Strategy' cannot handle this context
##[debug][Host] Checking strategy: Node20Strategy
[Host] Selected Node version: Node20 (Strategy: Node20Strategy)
##[debug][Host] Node path: /agent/externals/node20_1/bin/node
##[debug][Host] Reason: Selected for Node20 task handler
##[debug]Using node path: /agent/externals/node20_1/bin/node
##[debug]system.debug=True
##[debug]DistributedTask.Tasks.Node.SkipDebugLogsWhenDebugModeOff=True
##[debug]agent.TempDirectory=/agent/_work/_temp
##[debug]loading inputs and endpoints
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loading INPUT_REPORTDIR
##[debug]loading INPUT_TESTRESULTSDIR
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loaded 9
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]allureVersion=undefined
##[debug]reportName=undefined
##[debug]allureDownloadUrl=undefined
##[debug]testResultsDir=/agent/_work/3/s/allure-results
##[debug]reportDir=/agent/_work/3/s
##[debug]task result: Succeeded
##[debug]Processed: ##vso[task.complete result=Succeeded;done=true;]Published allure3 report from the provided allure report directory
Finishing: Publish Allure Report

Attempting to blank the reportDir

This run is essentially the same, but the yaml had one extra parameter.

Pipeline Yaml

  - task: PublishAllureReport@2
    displayName: Publish Allure Report
    condition: always()
    continueOnError: true
    inputs:
      testResultsDir: '$(System.DefaultWorkingDirectory)/allure-results'
      reportDir: ''

Everything else had the same output as the previous example (omitting for brevity)

Use v2 report example

This example, I specified the allure version set to 2.36.0 (v2) in the yaml file. This version works and publishes the v2 report fine. The file matching component is the same, so omitting that piece. I also turned off System Diagnostics for this run.

Pipeline Yaml

  - task: PublishAllureReport@2
    displayName: Publish Allure Report
    condition: always()
    continueOnError: true
    inputs:
      testResultsDir: '$(System.DefaultWorkingDirectory)/allure-results'
      allureVersion: '2.36.0'

Allure Publish Task Output:

Starting: Publish Allure Report
==============================================================================
Task         : Publish Allure Report
Description  : Task for generate and publish Allure Report
Version      : 2.2.5
Author       : Qameta Software Inc.
Help         : Generate and publish Allure Report
==============================================================================

[Host] Selected Node version: Node20 (Strategy: Node20Strategy)
Found 20 test results files.
Using bundled Allure Report (2.36.0)
Installing Allure CLI from bundled binary
/tmp/allure-cli-QKHCM6/allure-2.36.0/bin/allure "generate" "--single-file" "--name" "Allure Report" "--clean" "-o" "/tmp/allure-report-sKLeue" "/agent/_work/2/s/allure-results"
Report successfully generated to /tmp/allure-report-sKLeue
Finishing: Publish Allure Report

I hope these additions help! Thanks for the great suggestions and let me know if anyone else has any other thoughts.

[lMysticl]

The reportDir debug line is suspicious, but by itself it does not prove that the task selected the “publish an existing report” path; it may be a normalized internal path.

The documented contract is: omit reportDir when the task should generate a report from testResultsDir. Supplying reportDir tells the task to publish an already generated Allure 3 report and ignore testResultsDir. I would remove that input completely rather than pass an empty string.

The quickest A/B check is to force the bundled Allure 2 version:

- task: PublishAllureReport@2
  displayName: Publish Allure Report
  condition: succeededOrFailed()
  inputs:
    testResultsDir: '$(System.DefaultWorkingDirectory)/allure-results'
    allureVersion: '2.36.0'

If this publishes successfully, the results path and Playwright output are valid; the problem is in the default Allure 3/report-publishing path. The same “task succeeds with almost no output but no report” symptom was reported in #3200, where forcing 2.36.0 was the working discriminator and workaround.

For an extension bug report, include the exact task version printed in the task header and the two debug lines. Based on the documented inputs, a workspace-valued reportDir that you did not supply should be treated as an extension issue unless the logs show that it is only an internal output directory.

Reference: Allure Report documentation for Azure DevOps.

[Xetoris]

Hey Mystic!

Great suggestions. I'm going to modify my original posts with some additional data to try and provide the evidence you suggested. Please re-review and let me know if you can think of anything else that might help.