Codecov must choose a parent commit when comparing coverage reports. Having accurate coverage data is crucial. Therefore, Codecov only chooses commits that have successful CI builds. To illustrate this let's review the following commits:
Awas uploaded. The CI passed for this commit. Coverage did not change.
Bwas uploaded. The CI failed.
Cwas uploaded. The CI failed.
Dwas uploaded. The CI passed and coverage increased.
Codecov will compare
D because commits
C failed in CI, and likely have invalid or incomplete coverage data. This results in a
+2 p.p. change in coverage.
When coverage runs on your build, it is essential that Codecov recognizes that CI has passed or failed. If CI fails, one or more of the following assumptions can be made:
- Not all tests were executed; therefore coverage is incomplete.
- Exceptions may call new execution paths, resulting in different coverage metrics.
- A failed test could produce different coverage than the same test ran successfully.
There are two techniques to updating pull requests: rebasing and merging target.
Both techniques are respected by Codecov. The pull request base will be updated from
b, and Codecov will use reports from
b when comparing against pull request head (
# git diagram - before ``` master a . . b pull \ . . c ``` # rebase action git rebase master # git diagram - after ``` master a . . b pull \ . . c ```
# git diagram - before ``` master a . . b pull \ . . c ``` # merging target action git merge master # git diagram - after ``` master a . . b \ pull \ . . . mc c ```
When Codecov generates a comparison for a pull-request, but the pull-request’s base in git does not have coverage information, Codecov will try to find an appropriate substitute (a “pseudo-base”). This is referred to as pseudo-comparison. Pseudo-comparison can be turned off via the Codecov YAML option:
codecov: allow_pseudo_compare: False
The default is True. When pseudo-comparison is disallowed, and the base commit of a PR has no coverage data, Codecov will render an error in the UI.
Sometimes the pseudo-base chosen by Codecov will not be recent enough to directly substitute for the base commit of a pull-request. In this case, Codecov will try to use the diff between the pseudo-base and the true base to adjust the coverage information of the pseudo-base and account for recent changes. This feature of applying “coverage offsets” to the pseudo-base’s coverage report can also be configured by the Codecov YAML:
codecov: allow_coverage_offsets: True
The default is False. If Codecov is configured to pseudo-compare when coverage is missing for a base commit, but not to apply coverage offsets when the pseudo-base is too old for direct substitution for the base commit, Codecov will render an error in the UI.
The base commit of the pull request did not upload coverage resulting in Codecov not able to compare reports.
master . pseudo . . base pull \ . . head
pseudodid upload coverage
basedid not upload coverage
headdid upload coverage
Examples of the use cases:
- Base commit skipped CI via
[ci skip], use the parent commit as the pseudo commit below.
- Base commit was a merge commit
Merge abc into xyz, use
xyzas the pseudo commit below.
Codecov will seek a parent commit and offset the report to get an approximate base report.
# first get the git diff of pseudo...base pseudo_diff_base = diff(pseudo...base) # check if you allow offsets (default False) if yaml["codecov.allow_coverage_offsets"] is False if pseudo_diff_base.adjusts_tracked_lines(pseudo.report) exit "Missing report base" # adjust the report with changes in pseudo_diff_base approx_base_report = pseudo.report.adjust_forward(pseudo_diff_base) # get the pull diff pull_diff = diff(base...head) use compare(approx_base_report...head.report) in pull comment/statuses/etc.
Updated about a month ago