Codecov

Code coverage done right.®

Welcome to Codecov Documentation. You'll find comprehensive guides and documentation to help you start working with Codecov as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started

Commit Status

Useful for blocking Pull Requests that don't meet a particular coverage threshold.

Project Status

The codecov/project status measures overall project coverage and compares it against the base of the pull request or parent commit.

coverage:
  status:
    project:
      default:
        # basic
        target: auto
        threshold: 0%
        base: auto 
        flags: 
          - unit
        paths: 
          - "src"
       # advanced
        branches: 
          - master
        if_not_found: success
        if_ci_failed: error
        informational: false
        only_pulls: false

Basic Configuration

target

auto | number
Choose a minimum coverage ratio that the commit must meet to be considered a success.

  • auto will use the coverage from the base commit (pull request base or parent commit) coverage to compare against.
  • number you can specify a target of an exact coverage number such as 75% or 100% (string, int, or float accepted).

threshold

number
Allow the coverage to drop by X%, and posting a success status.

base

[Deprecated as of July 2020 -- Codecov's product roadmap includes a novel way to manually pick base of pull requests]

flags

Flags are a list of user defined Flags, and the impact on their coverage. You can specify an array of flags as follows:

coverage:
  status:
    project:
      default:
        #...
        flags:
          - flag1
          - flag2
          - ...

and the combined coverage from those flags will be reported. Before using flags, it is highly recommended to read more about their use.

paths

Similar to flags, an array of paths and/or regular expressions can be provided and the status will report the combined coverage for the files that match the path name / regular expression.

Advanced Configuration

The following settings almost never need to be modified as the Codecov wide defaults should be sufficient for nearly all use cases. Nonetheless they are defined here for completeness.

branches

The branches that, when used, will trigger this status.

if_not_found

Settings are 'success' and 'failure' the default is 'failure'.

  • failure: the status will fail if there is no report for the head
  • success: the status will pass if there is no report for the head. Use this on commits / PRs where you won't be uploading coverage but still want codecov status checks to pass.

informational

Use Codecov in informational mode. Default is false. If true is specified the resulting status will pass no matter what the coverage is or what other settings are specified. Informational mode is great to use if you want to expose codecov information to other developers in your pull request without necessarily gating PRs on that information.

only_pulls

Only post a status to pull requests, defaults to false. If true no status will be posted for commits not on a pull request

if_ci_failed

Options are:

  • error: Will set the status to success only if the CI is successful
  • success: Will set the status to success even if the CI fails

Excluding tests (Example)

Below, is an example of using multiple project statuses that measure different aspects of your project.

coverage:
  status:
    project:
      default: false  # disable the default status that measures entire project
      tests:  # declare a new status context "tests"
        target: 100%  # we always want 100% coverage here
        paths: "tests/"  # only include coverage in "tests/" folder
      app:  # declare a new status context "app"
        paths: "!tests/"  # remove all files in "tests/"

Now you will see two unique status contexts from Codecov: codecov/project/tests and codecov/project/app.

Splitting up projects (Example)

You may have a code base that has multiple application components that you would like to monitor independently. Codecov provides a very simple way create statuses for each component.

coverage:
  status:
    project:
      users:
        paths:
          - tests/users
          - app/components/user*
      products:
        paths:
          - tests/products
          - app/components/product*

As illustrated above, you can set project statuses filtering out specific components of the application, and get 3 unique statuses monitoring each component.

Patch Status

The codecov/patch status only measures lines adjusted in the pull request or single commit, if the commit is not in a pull request. This status provides an indication on how well the pull request is tested.

coverage:
  status:
    patch:
      default:
        # basic
        target: auto
        threshold: 0%
        base: auto 
        # advanced
        branches: 
          - master
        if_no_uploads: error
        if_not_found: success
        if_ci_failed: error
        only_pulls: false
        flags: 
          - "unit"
        paths: 
          - "src"

To illustrate the usage of the patch status, let's go through this exercise.

def divide(x, y):
+     if y <= 0:
+         raise ValueError("y must be greater than 0")
      return x * y

The resulting codecov/patch status of this commit would be 0% covered because no tests are created for this method. Even though the project coverage is 72% (the entire code base not shown), this patch status will only measure lines added.

To make another commit on this pull request, adding tests, proceed as follows.

+ def test_divide_by_1(self):
+     assert divide(10, 1) == 10

Running the tests will result in a patch coverage of 50% covered because we have not yet tested the behavior of dividing by zero. Let's add another test.

def test_divide_by_1(self):
     assert divide(10, 1) == 10

+ def test_divide_by_zero(self):
+     with self.assertRaises(ValueError)
+         divide(1, 0)
+

Now Codecov will report a codecov/patch status of 100% covered for this pull request. This indicates that the pull request adjusted code is properly executed by tests.

Disabling a status

You may choose to disable the default statuses Codecov posts by using the following yaml configuration.

📘

Enabling a status

Note that you can not enable a status by setting it to yes, you need to define it by using the structure at https://docs.codecov.io/docs/commit-status#section-project-status

coverage:
  status:
    project: off
    patch: off

Changes Status

Codecov will detect changes in coverage that are NOT included in the commit/pull diff, and report these changes as a commit status.

Let's take this example to illustrate what an Unexpected Coverage Changes would look like.

First commit. 100% coverage.

As shown above, we have 100% coverage. Now let's make a change to this code base.

Second commit's diff.

Our CI will run and result in the following:

Second commit's coverage.

Lines 1 and 2 are considered "changes" in Codecov. This status would detect these changes and report them to the commit status.

Updated 17 days ago

Commit Status


Useful for blocking Pull Requests that don't meet a particular coverage threshold.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.