Codecov

Code coverage done right.®

Welcome to Codecov Documentation. You'll find comprehensive guides and documentation to help you start working with Codecov as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started

Carryforward Flags

Reference of past coverage for tests that are not run on current commit.

👍

Carryforward Flags are Best Used for Monorepos and Iterative Testing Setups

Carryforward Flags are designed for those projects that do not upload total coverage for every commit (e.g., monorepos with multiple applications/languages, iterative/partial/delta testing setups, etc).

Before Getting Started

To benefit from Carryforward Flags, make sure you are set up with Codecov's Flag feature, to tag different sessions/builds.

Basic Flags documentation, including:

  1. Set Flags in your YAML
  2. Passing -F parameter in your upload

A straightforward example repository using carryforward flags can be seen here:

GitHub -- https://github.com/codecov/cf-flags-demo
Codecov -- https://codecov.io/gh/codecov/cf-flags-demo

How Carryforward Flags Work: A Motivating Example

Before discussing how to use carryforward flags, it's important to discuss how they work. That is best demonstrated with the following diagram:

An example of two commits using flags, carryforward flags, and no flags.

The following codecov.yml is also supplied for the above example:

flags:
  ui:
    paths:
      - ui_1.py
      - ui_2.py
    carryforward: true
  unit:
    paths:
      - unit_1.py
      - unit_2.py
    carryforward: true
  enterprise:
    paths:
      - ent_1.py
      - ent_2.py
    carryforward: false 
   # If no Carryfoward flag specified in YAML, the
   # default configuration is false.

In this example, the coverage for ui_1.py and ui_2.py was taken directly from the coverage report uploaded for Commit 1. The ui_cov.xml coverage report was uploaded using the following syntax:

bash <(curl https://codecov.io/bash) -f ui_cov.xml -F ui

This upload command specifies both the path to the coverage file for files under the ui flag and the appropriate flag name (ui in this case). Once uploaded, coverage was calculated to the ui_1.py and ui_2.py files in the same manner as any report upload.

Since no report is uploaded for the unit flag, which has a setting of carryforward: true , Codecov reaches back to Commit 0 to carry forward the coverage for all files covered by the unit flag. As a result, those files have coverage information for Commit 1 that is equivalent to Commit 0.

Finally, since the enterprise flag has a setting of carryforward: false no coverage is carried forward for any file covered by the enterprise flag. Since no coverage report was uploaded for the enterprise flag for Commit 1, the ent_1.py and ent_2.py files have 0% coverage for Commit 1.

🚧

You Must Upload using -F

To properly leverage flags, and thus, carryforward flags, you must ensure you're uploading coverage reports with the appropriate flag name.

Carryforward Flags in the Codecov UI

Once your team starts using carryforward flags they will appear in the UI just like normal flags, with the addition of distinctive iconography to set them apart from Codecov's standard flags. For example, carryforward flags can be toggled when viewing source:

In this example flagone is carried forward and flagtwo and flagthree are not.

Additionally, any coverage that is carried forward will be shown in the Build tab of a the commit that carried the coverage forward. The commit from which the coverage was carried forward is also referenced wherever carryforward flags are shown, like so:

Advanced: Configuring Carryforward Flags in the Code Host

Carryforward Flags in the Pull Request Commit Status

📘

Carryforward Flags pass status checks by default

If you are using Carryforward Flags and surfacing commit statuses via Codecov, by default, these commit statuses will always pass.

This is to prevent tests not run on the current commit from blocking the current commit's status checks.

You can configure this behavior below.

There are two layers at which you can configure carryforward flags in the commit status.

  1. default_rules for the whole repo

  2. "Per flag" under any specific flag, which supersedes the default_rules

The possible settings for the Carryforward Flags Commit Status are:

  • pass [Default]: the status check will pass automatically if all flag coverage was carried forward

  • include: we’ll evaluate and send this status check as we normally do if all flag coverage was carried forward

  • exclude: we won’t send this status check at all if all flag coverage was carried forward. This is a useful feature if you only want to see statuses from the current commit.

Warning: If you require status checks to pass in your git provider to merge, and exclude those same carried-forward statuses in UI, be wary of a carried forward status blocking your pull request from merging.

Commit Status with carryforward behavior set to pass on the "#index"
flag

Example YAML configuration

coverage:

  status:
  # configuration happens at the status layer, not the "flags"
  # layer below

    default_rules:
    #default rules for the whole repo

      carryforward_behavior: "exclude"
      # options: "exclude", "pass", "include";
      # defaults to “pass"

    project:

      moduleA:

        flags:

          - moduleA

        carryforward_behavior: "include"
                # Because flag-specific behavior is set,
        # it supersedes default_rules above

      moduleB:

        flags:

          - moduleB-1

          - moduleB-2
          
        # Because no carryforward flag-specific behavior is set, 
        # Default_rules above is used


flags:

  moduleA: 

    paths: 

      - src/moduleA/code.py 

    carryforward: true 

  moduleB-1: 

    paths: 

      - src/moduleB/code1.py 

    carryforward: true

  moduleB-2:

    paths:

      - src/moduleB/code2.py

    carryforward: true

Carryforward Flags in the Pull Request Comment

📘

Carryforward Flags do not show in the pull request comment by default

By default, any flags that are carried forward will not show up in the pull request / merge request in your code host (Github, Gitlab, or Bitbucket) in the status checks and/or the pull request comments.

Instead, only flags that have coverage uploaded on the most recent commit will show in the PR Comments.

By default flags that are carried forward will not appear in the Pull Request Comment.

If you would like Carryforward Flags to appear in PR comments, use the following YAML configuration under show_carryforward_flags.

The possible settings for:show_carryforward_flags are:

  • false [default]: flags with carried forward coverage will not show up on the flags table

  • true: if any flag coverage was carried forward, the flags table will have an additional column labeled “Carriedforward” indicating which flags had coverage carried forward

PR comment with show_carryforward_flags set to true

Example YAML:

comment: 

  layout: "reach, diff, flags, files” 

  behavior: default require_changes: false 
  # if true: only post the comment if coverage changes 

  require_base: no # [yes :: must have a base report to post] 

  require_head: yes # [yes :: must have a head report to post] 

  branches: # branch names that can post comment 

    - "master"

  show_carryforward_flags: false

Updated a day ago

Carryforward Flags


Reference of past coverage for tests that are not run on current commit.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.