Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Branch not reported correctly on coveralls.io #1697

Open
bitfl0wer opened this issue Feb 23, 2025 · 11 comments
Open

Branch not reported correctly on coveralls.io #1697

bitfl0wer opened this issue Feb 23, 2025 · 11 comments
Assignees

Comments

@bitfl0wer
Copy link

I am using tarpaulin with coveralls.io. coveralls supports viewing coverage on branches. However, coverage data generated from runs on branches does not get recognized correctly:

Screenshot of a CI (Continuous Integration) dashboard titled “RECENT BUILDS.” The table displays eight rows, labeled #8 through #1, with column headers for “BUILDS,” “BRANCH,” “COMMIT MESSAGE,” “TYPE,” “RAN,” “COMMITTER,” “VIA,” and “COVERAGE.” Each row shows a build number, the corresponding Git branch (e.g., “main”), a short commit message (including one reading “use LLM and no-dead-code as recommended by tarpaulin for linux” and another reading “[no commit message]”), the build trigger type (“push”), the date and time the build ran, the committer (displayed as “bottower”), how it was triggered (“via bottower”), and a coverage percentage in the last column (ranging around the 66–76% range in red and orange bars). Notably, the coverage values appear inconsistent for branch builds, suggesting that coverage runs on non-main branches are not being recognized correctly (some entries are missing or remain at an unexpected value), whereas main branch builds do show an updated coverage percentage.

GitHub actions workflow

runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Swatinem/rust-cache@v2
        with:
          cache-all-crates: "true"
          prefix-key: "coverage"
      - name: Build, Publish Coverage
        run: |
          if [ -n "${{ secrets.COVERALLS_REPO_TOKEN }}" ]; then
              curl -L --proto '=https' --tlsv1.3 -sSf https://raw.githubusercontent.com/cargo-bins/cargo-binstall/main/install-from-binstall-release.sh | bash
              cargo binstall --no-confirm cargo-tarpaulin --force
              cargo tarpaulin --no-dead-code --engine llvm --all-features --tests --verbose --skip-clean --coveralls ${{ secrets.COVERALLS_REPO_TOKEN }} --timeout 120
          else
              echo "Code Coverage step is skipped on PRs from forks."
          fi

Expected outcome

In this scenario, I'd expect the branch name and commit message data to be submitted to coveralls correctly. Am I doing something wrong with my command invocation?

@xd009642
Copy link
Owner

It looks fine on your end so it's probably me. I'll check the coveralls API tomorrow and make sure I'm not missing out on any of the optional fields I can set in the request!

@bitfl0wer
Copy link
Author

Interestingly, I just created a new branch and committed it. That seems to have been picked up as intended.

Screenshot of a CI pipeline dashboard titled “RECENT BUILDS,” now displaying nine rows labeled #10 through #2 (with each row showing build number, branch name, commit message, build trigger type, timestamp, committer, and coverage). The top row, build #10 on the “coverage” branch, has a commit message “try excluding trace/ calls to increase coverage,” run via push on 23 Feb 2025 at 02:49 PM CEST by user “bottower,” and shows a coverage percentage of 72.00%. Below it, builds #9 through #2 list either “[no commit message]” or specific messages (like “use LLM and no-dead-code as recommended by tarpaulin for linux”), all with coverage values in the 66–72% range. Unlike the previous screenshot, the coverage data now appears consistently recognized across all branches, suggesting that the earlier coverage-detection issue has been resolved.

Making a second commit on this new branch also works. Perhaps, it is important to point out, that the "troubled" branch is called "dev/v0.10". Maybe that's part of the issue. Hope this helps somewhat :)

@xd009642
Copy link
Owner

Yeah I've recreated the same behaviour, and managed to find an open source project which has branches with / in and those branches show up so I know it's not a limitation in the coveralls API. I'll dig in deeper and update when I've made some progress

@bitfl0wer
Copy link
Author

bitfl0wer commented Feb 24, 2025

Here's my coveralls project, if that helps: https://coveralls.io/github/polyphony-chat/polyproto-rs

Thank you for looking into it so timely!

@xd009642
Copy link
Owner

Got a branch where I'm trying out some stuff #1699

@xd009642
Copy link
Owner

xd009642 commented Mar 4, 2025

https://coveralls.io/github/xd009642/rust-sandbox?branch=feat/do-slashs-pls well I have a job with slashes in the name! One thing to note, if it's in a PR GHA often runs against the master state of the CI config (I think partially as a security measure) and for these instances the branch name is incorrectly "master". That's what libgit2 reports so likely due to some nonsense in how github actions work... But when it's set to branches: "**" on push I got the slash properly

This has lead to some improved error reporting and logging for coveralls and I now set a bit more of the data I can set on a request to let it know about the repo so maybe I did something or maybe it's github shenanigans. But as I see the correct branch name in your actions tab on the branch for past workflows it should hopefully work for you with my changes.

Just going to tidy up and publish!

@xd009642
Copy link
Owner

xd009642 commented Mar 4, 2025

Okay the publish machine is running so it'll be on crates.io whenever the CI finishes (normally like 15 minutes at most). If you can let me know if it improves anything your end 🙏

@bitfl0wer
Copy link
Author

bitfl0wer commented Mar 4, 2025

Thank you for your hard work!

I have tested your changes by running with version v0.32.2. You can find the full workflow run here, if you'd like. Unfortunately, it seems like it looks just the same on coveralls:

Image :(

@bitfl0wer
Copy link
Author

2025-03-05T15:27:46.347013Z  WARN cargo_tarpaulin::report::coveralls: Failed to collect git info: failed to get branch name: reference 'HEAD' is neither a local nor a remote branch.; class=Invalid (3)
2025-03-05T15:27:46.347028Z  INFO cargo_tarpaulin::report::coveralls: Sending coverage data to coveralls.io
Error: "Failed to report coverage! Error: Coveralls send failed. http error: error sending request for url (https://coveralls.io/api/v1/jobs)"
Error: Process completed with exit code 1.

This just in: CI gives an indicator for why things might not work. Also, the CI pipeline fails now :P

@xd009642
Copy link
Owner

xd009642 commented Mar 5, 2025

aha me not ignoring coveralls errors has had unintended debugging benefits...

Okay this might be based on how GHA does things... Maybe I need to use one of their env vars to get the branch name instead of libgit.

It looks like this is a bit of an annoying github issue https://github.com/orgs/community/discussions/5251 I'm definitely not happy about how sometimes it reports the branch name correctly and sometimes it seems I just get rubbish 💀

@xd009642
Copy link
Owner

xd009642 commented Mar 5, 2025

Side note I'm travelling for the next 1-2 weeks so between that and remote work I might not be as quick on this as I was before - but I'll try and find some time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants