Codecov

Code coverage done right.®

Welcome to Codecov Documentation. You'll find comprehensive guides and documentation to help you start working with Codecov as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started

Comparing Commits

Increase in coverage from 50% to 52% is 2 percentage points (p.p.) or (52 - 50) / 50 = 0.04 (4%) but not 2%
https://en.wikipedia.org/wiki/Percentage_point

Choosing a parent commit

Codecov must choose a parent commit when comparing coverage reports. Having accurate coverage data is crucial. Therefore, Codecov only chooses commits that have successful CI builds. To illustrate this let's review the following commits:

Note: A is the oldest commit and D is the newest.

Order of events

  • Commit A was uploaded. The CI passed for this commit. Coverage did not change.
  • Commit B was uploaded. The CI failed.
  • Commit C was uploaded. The CI failed.
  • Commit D was uploaded. The CI passed and coverage increased.

Codecov will compare A against D because commits B and C failed in CI, and likely have invalid or incomplete coverage data. This results in a +2 p.p. change in coverage.

Commits with Failed CI

When coverage runs on your build, it is essential that Codecov recognizes that CI has passed or failed. If CI fails, one or more of the following assumptions can be made:

  1. Not all tests were executed; therefore coverage is incomplete.
  2. Exceptions may call new execution paths, resulting in different coverage metrics.
  3. A failed test could produce different coverage than the same test ran successfully.

Rebase vs Merge Target

There are two techniques to updating pull requests: rebasing and merging target.

Both techniques are respected by Codecov. The pull request base will be updated from a to b, and Codecov will use reports from b when comparing against pull request head (c).

Rebasing

# git diagram - before
```
master a . . b
pull    \ . . c
```

# rebase action
git rebase master

# git diagram - after
```
master a . . b
pull          \ . . c
```

Merging Target

# git diagram - before
```
master a . . b
pull    \ . . c
```

# merging target action
git merge master

# git diagram - after
```
master a . . b
               \
pull    \ . . . mc c
```

Comparing pull request base reports

Issue

The base commit of the pull request did not upload coverage resulting in Codecov not able to compare reports.

master . pseudo . . base
pull                  \ . . head
  • pseudo did upload coverage
  • base did not upload coverage
  • head did upload coverage

Examples of the use cases:

  • Base commit skipped CI via [ci skip], use the parent commit as the pseudo commit below.
  • Base commit was a merge commit Merge abc into xyz, use xyz as the pseudo commit below.

Resolution

Codecov will seek a parent commit and offset the report to get an approximate base report.

# first get the git diff of pseudo...base
pseudo_diff_base = diff(pseudo...base)

# check if you allow offsets (default False)
if yaml["codecov.allow_coverage_offsets"] is False
  if pseudo_diff_base.adjusts_tracked_lines(pseudo.report)
    exit "Missing report base"

# adjust the report with changes in pseudo_diff_base
approx_base_report = pseudo.report.adjust_forward(pseudo_diff_base)

# get the pull diff
pull_diff = diff(base...head)

use compare(approx_base_report...head.report) in pull comment/statuses/etc.

Updated 6 months ago


Comparing Commits


Increase in coverage from 50% to 52% is 2 percentage points (p.p.) or (52 - 50) / 50 = 0.04 (4%) but not 2%
https://en.wikipedia.org/wiki/Percentage_point

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.