Flags allow you to isolate coverage reports for different tests. This is particularly helpful if you have multiple types of tests (e.g., unit, integration, frontend, backend, etc), or you're employing a monorepo setup where you'd like to encapsulate each project's test coverage independently.

As a minimal example, consider a project called Monorepo that has the following folder structure:

Monorepo
    /projectA/src
    /projectB/src

To generate a flag and individual coverage gates per project in the mono repo the codecov.yml can be structured as follows:

coverage:
  status:
    project:
      default:
        target: 90%
        threshold: null
      projectA:
        flags: projectA
        target: 90%
      projectB:
        target: 50%
        flags: projectB
flags:
  projectA:  
    paths:
      - projectA/src
  projectB:
    paths:
      - projectB/src

This yaml configuration will fail a Pull Request if...

• ...the entire Monorepo project is below 90% covered (this is the default specification of the yaml above)
• ...the projectA coverage is less than 90%
• ...the projectB coverage is less than 50%

It is important to note that for flagging to work, separate coverage reports must be uploaded and flagged per project in the mono repo. For example, in a CI run for the MonoRepo project example, assume that coverage for projectA is written to

projectA/output/coverage.xml

Then that report would be uploaded as follows:

bash <(curl https://codecov.io/bash) -t <token> -f projectA/output/coverage.xml -F projectA

The projectA flag can be used for any number of reports (unit tests, integration tests, etc) by flagging each report with projectA. The reports will be merged and contribute the project's total coverage and the total coverage of the files covered under the projectA flag.

However, specifying multiple flags for a single report upload can result in erroneous coverage unless the contents of that report fully encompass each flag. For example, one report that contains coverage information for projectA and project B, can be uploaded in a manner that is valid but technically incorrect:

bash <(curl https://codecov.io/bash) -t <token> -f coverage.xml -F projectA -F projectB

This will apply the entire coverage of the report to both flags, resulting in incorrect coverage. Specifically, the projectA and projectB flags will display the coverage of the entire uploaded report, not just the subset of the report that happens to cover the files under their purview.

For coverage information to be properly tracked per flag, the flags must be applied at upload time.

To apply a flag simply submit coverage reports with the -F flagname included in the upload command; Codecov will do the rest.

The following example illustrates how you may run specific test groups and upload coverage, flagging each coverage group. This strategy for uploading is preferred as it cleanly associates a single flag with a single report.

# example running unittests only
py.test --cov=./ -k tests/unittests/
bash <(curl -s https://codecov.io/bash) -c -F unittests

# example running integration tests only
py.test --cov=./ -k tests/integration/
bash <(curl -s https://codecov.io/bash) -c -F integration

# example running ui tests only
npm test
bash <(curl -s https://codecov.io/bash) -c -F ui

Create custom notifications

Flags can be used to create custom notifications to your repository provider. This makes it easy to see per-flag converage information alongside pull requests.

You can specify Flags in your Codecov Yaml for statuses and all notifications.

coverage:
  status:
    project:
      default: off
      frontend:
        flags: frontend
      backend:
        target: 50%
        flags: backend
      api:
        target: 89%
        flags: api

flags:
  # filter the folder(s) you wish to measure by that flag
  backend:
    # only include files in the backend folder
    paths:
      - app/backend/
  frontend:
    paths:
      - app/frontend/
  api:
    paths:
      - app/api/
  tests:
    paths:
      - tests/

Hide Builds (e.g., nightly builds)

Codecov provides a strategy to isolate specific builds from the master report while maintaining the report's integrity. When reports are not joined into the master report, they will be ignored for comparison, although they will remain accessible for source overlay, api, badges, and graphing.

A nightly build is an excellent example of this feature. What follows is a yaml configuration for a nightly build.

flags:
  nightly:
    joined: false

Now that we have configured the flag nightly to not join into the master report, let's upload a report flagged as nightly.

bash <(curl -s https://codecov.io/bash) -F nightly

By adding -F nightly we mark all the coverage report data for this build as nightly coverage data.