Flags allow you to isolate coverage reports for different tests. This is particularly helpful if you have multiple types of tests (e.g., unit, integration, frontend, backend, etc), or you're employing a monorepo setup where you'd like to encapsulate each project's test coverage independently.
Flags in the Codecov UI
Flags feature on the Codecov Compare page
Codecov has suspended flags for Codecov.io users on the Codecov Compare page.
Flags were causing the compare page to consistently timeout for a minority of users. The Codecov team is investigating refactoring flags to make the compare page more usable, even with many flags turned on.
The compare page will still work for comparing changed code
The flags feature will still be viewable on each individual commit view
Flags will still be functional on pull and merge request status checks
We will continue to iterate on this feature and push out updates here.
Flags Example
As a minimal example, consider a project called Monorepo that has the following folder structure:
Monorepo
/moduleA
/moduleB
To generate a flag and individual coverage gates per project in the mono repo the codecov.yml can be structured as follows:
coverage:
status:
project:
default:
target: 90%
moduleA:
target: 90%
flags:
- moduleA
moduleB:
target: 50%
flags:
- moduleB
comment:
layout: "reach, diff, flags, files"
behavior: default
require_changes: false
require_base: yes
require_head: yes
branches: null
flags:
moduleA:
paths:
- src/moduleA/code.py
carryforward: false
moduleB:
paths:
- src/moduleB/code2.py
carryforward: true
This yaml configuration will fail a Pull Request if...
⢠...the entire Monorepo project is below 90% covered (this is the default
specification of the yaml above)
⢠...the moduleA coverage is less than 90%
⢠...the moduleB coverage is less than 50%
It is important to note that for flagging to work, separate coverage reports must be uploaded and flagged per project in the mono repo. For example, in a CI run for the MonoRepo project example, assume that coverage for moduleA is written to
moduleA/output/coverage.xml
Then that report would be uploaded as follows:
bash <(curl https://codecov.io/bash) -t <token> -f moduleA/output/coverage.xml -F moduleA
You Must Upload using -F
To properly leverage flags and you must ensure you are uploading coverage reports with the appropriate flag name.
The moduleA flag can be used for any number of reports (unit tests, integration tests, etc) by flagging each report with moduleA
. The reports will be merged and contribute the project's total coverage and the total coverage of the files covered under the moduleA
flag.
However, specifying multiple flags for a single report upload can result in erroneous coverage unless the contents of that report fully encompass each flag. For example, one report that contains coverage information for moduleA and moduleB, can be uploaded in a manner that is valid but technically incorrect:
bash <(curl https://codecov.io/bash) -t <token> -f coverage.xml -F moduleA -F moduleB
This will apply the entire coverage of the report to both flags, resulting in incorrect coverage. Specifically, the moduleA
and moduleB
flags will display the coverage of the entire uploaded report, not just the subset of the report that happens to cover the files under their purview.
Flag Creation
For coverage information to be properly tracked per flag, the flags must be applied at upload time.
To apply a flag simply submit coverage reports with the -F flagname
included in the upload command.
The following example illustrates how you may run specific test groups and upload coverage, flagging each coverage group. This strategy for uploading is preferred as it cleanly associates a single flag with a single report.
# example running unittests only
py.test --cov=./ -k tests/unittests/
bash <(curl -s https://codecov.io/bash) -c -F unittests
# example running integration tests only
py.test --cov=./ -k tests/integration/
bash <(curl -s https://codecov.io/bash) -c -F integration
# example running ui tests only
npm test
bash <(curl -s https://codecov.io/bash) -c -F ui
Remove old reports after uploading (optional)
Apply the
-c
argument to clear the workspace of all coverage reports before running you next set of tests.
Flags must be lowercase, alphanumeric, periods, or hyphens and not exceed 45 characters
Only lowercase, alphanumeric values are accepted.
^[a-z0-9_\.\-]{1,45}$
Advanced Flag Use Cases
Carryforward Flags
If you do not test all of your repo code on each commit, Codecov uses a feature called Carryforward Flags to help only update coverage on tests that were run. Carryforward Flags is built on top of basic Flags.
Carryforward Flag Usage
It is highly recommended to read the full documentation on carryforward flags before using them in your project.
Carryforward Flags are used by appending your YAML with carryforward: true
:
flags:
ui:
paths:
- ui_1.py
- ui_2.py
carryforward: true
unit:
paths:
- unit_1.py
- unit_2.py
carryforward: true
enterprise:
paths:
- ent_1.py
- ent_2.py
carryforward: false
# If no Carryfoward flag specified in YAML, the
# default configuration is false.
Create custom notifications
Flags can be used to create custom notifications to your repository provider. This makes it easy to see per-flag coverage information alongside pull requests.
You can specify Flags in your Codecov Yaml for statuses and all notifications. Note that a flag must be specifically stated in the status:
section of the YAML to have a custom notification associated.
coverage:
status:
project:
default: off
frontend:
flags:
- frontend
backend:
target: 50%
flags:
- backend
api:
target: 89%
flags:
- api
flags:
# filter the folder(s) you wish to measure by that flag
backend:
# only include files in the backend folder
paths:
- app/backend/
frontend:
paths:
- app/frontend/
api:
paths:
- app/api/
tests:
paths:
- tests/


Custom Statuses only report on specific Flags.
Hide Builds (e.g., nightly builds)
Codecov provides a strategy to isolate specific builds from the master report while maintaining the report's integrity. When reports are not joined into the master report, they will be ignored for comparison, although they will remain accessible for source overlay, api, badges, and graphing.
A nightly build is an example of this feature. What follows is a Yaml configuration for a nightly build.
flags:
nightly:
joined: false
Now that we have configured the flag nightly
to not join into the master report, let's upload a report flagged as nightly
.
bash <(curl -s https://codecov.io/bash) -F nightly
By adding -F nightly
we mark all the coverage report data for this build as nightly
coverage data.
Updated about a month ago