Introduce a local coverage plugin + add conditional exclusions by sirosen · Pull Request #2347 · jazzband/pip-tools

sirosen · 2026-03-05T01:44:24Z

This PR is a continuation from #2268 .
As I mentioned in #2264 , I've worked up a local plugin and tried out using PYTHONPATH to make it importable under coverage, and locally my testing of it seems to work.

The plugin is defined in plugins/coverage/ as a module, and injects coverage excludes of the form

pragma: pip{comparator}{version} no cover

I've added it on top of the commits from #2268, rebased, and updated to use the new "pip-version pragmas" where possible.

Contributor checklist

Included tests for the changes.
A change note is created in changelog.d/ (see changelog.d/README.md
for instructions) or the PR text says "no changelog needed".

Maintainer checklist

If no changelog is needed, apply the bot:chronographer:skip label.
Assign the PR to an existing or new milestone for the target version
(following Semantic Versioning).

These are either chronically uncovered up to a certain line or have flaky coverage. Either way, having them in is rather harmful for measurements.

These are version-dependent but are sometimes applied to conditionals checking the deps versions. They won't work perfectly but it's a start.

A new local plugin in `plugins/coverage/` as a standalone module, which injects coverage excludes of the form pragma: pip{comparator}{version} no cover In order for this plugin to be picked up, `.coveragerc` is updated to list the plugin by name and `PYTHONPATH` is set via `setenv` in order to make the module importable. All conditional checks on the pip version are now marked with the relevant matching pragmas.

When looking at lost coverage reporting in codecov, it appears that it is counting the new plugin code as "uncovered". Though true -- it isn't being tested rigorously -- trying to measure it seems incorrect at present.

sirosen · 2026-03-05T19:34:26Z

codecov was unhappy -- my guess about the issue, based on the output, was that it was treating the coverage plugin as "uncovered", so I've pushed a fix for that.

If the coverage still looks bad, I may have to look more deeply and potentially mark this draft until I can fix.

webknjaz

Cool that you got to looking into this! I've left a few comments inline.

webknjaz · 2026-03-05T19:44:35Z

tests/test_cli_compile.py


    # for older pip versions, recompute the output path to be relative to the input path
-    if not pip_produces_absolute_paths:
+    if not pip_produces_absolute_paths:  # FIXME: figure out how to cover piplowest


Could you elaborate?

Oh, that wasn't supposed to still be there! 😳

I think that's from before I had the plugin working.
But it's wrong. Let me replace with a pragma.

webknjaz · 2026-03-05T19:45:53Z

tests/conftest.py

 @pytest.fixture
 def runner():
-    if Version(version_of("click")) < Version("8.2"):
+    # Coverage is excluded because we only test with the latest Click


Should we fix that? Or maybe make it not branchy by updating deps in the meta?

I think we should go after a fix, but that it doesn't fit within the scope of this PR.

I've been considering the possible argument for vendoring click, on account of the fact that it would allow pip-tools to be used in environments with really old click versions.

Like pip, I think we should not have Python package dependencies, other than pip itself. (Also like pip, I think this will pose a licensing, SBOM, and transparency challenge! )

webknjaz · 2026-03-05T19:50:46Z

.coveragerc

+    piptools_coverage
 omit =
    piptools/_compat/*
+    # any local plugins we use aren't part of normal testing/coverage


Honestly, I'd argue that our own tooling is subject to being tested as well. Especially since it may grow into something that could eventually be ready to be moved out. So if possible, I'd add a test for the plugin.

Although I agree in principle, I think we'll have trouble unless we add a dedicated test job in tox and CI, with a dedicated and distinct .coveragerc. The reason is that coverage imports this code on startup, so it's imported too early to measure. I can't think of a way to measure it right now, though I was also considering reaching out to Ned (he's active in the main Python discord) to see if he could help point me in the right direction.

I'm happy to write tests for it, but I think it might belong outside of coverage reporting until I can figure that out?
I was hoping we could include it early, and I could continue to work on this.

webknjaz · 2026-03-05T19:52:46Z

plugins/coverage/piptools_coverage.py

FWIW, I'm not sure I like a top-level public name like plugins/. Wonder if it could live in tests/_plugins/ or smth like that since it's a part of our test toolchain.

I can move it around. tests/_plugins/_coverage/ works for me, if that sounds good?

webknjaz · 2026-03-05T19:53:52Z

plugins/coverage/piptools_coverage.py

+        config.set_option("report:exclude_lines", sorted(exclude))
+
+
+def coverage_init(registry: Plugins, options: dict[str, _t.Any]) -> None:


MyPy recommends object instead of _t.Any. Can we use that?

When I circle back on this to do more work, I'll give it a try. 👍

webknjaz · 2026-03-05T19:54:17Z

tests/test_cli_compile.py

    (first_posarg, *_tail_args), _kwargs = PyPIRepository.call_args

-    if _pip_api.PIP_VERSION_MAJOR_MINOR >= (25, 3):  # pragma: >=3.9 cover
+    if _pip_api.PIP_VERSION_MAJOR_MINOR >= (25, 3):  # pragma: pip<25.3 no cover


This feels a bit contradictary: the check is <= and the pragma's inversed. And then the opposite branch adds inversion that looks like what's in the if-expression down below. So I'd suggest supporting cover, not just no cover.

Althouh, can we analyse expressions with the known constants like _pip_api.PIP_VERSION_MAJOR_MINOR / _pip_api.PIP_VERSION and apply coverage inclusion/exclusion based on that automatically even w/o the comments? The else-branch could be handled the same way.

Why don't I work on cover first, and we can think about supporting comparisons with _pip_api.PIP_VERSION attributes as a future bit of work?

webknjaz · 2026-03-05T19:56:11Z

tox.ini

    pipmain: https://github.com/pypa/pip/archive/main.zip
 setenv =
+    # local plugin inclusion
+    coverage: PYTHONPATH=./plugins/coverage/


Do we know if pytest's pythonpath setting would be enough instead of this?

I can give it a try! 👍

webknjaz · 2026-03-05T19:58:09Z

plugins/coverage/piptools_coverage.py

+CURRENT_YEAR_EPOCH = datetime.date.today().year - 2000
+_PRAGMA_SUPPORTED_PIP_VERSIONS: list[tuple[int, int]] = [
+    (major, minor)
+    for major in range(22, CURRENT_YEAR_EPOCH + 1)


Can we read the lowest supported from pyproject.toml?

Oh, that's a good idea. Yes, we can extract it from there.

sirosen · 2026-03-05T20:42:56Z

It looks like codecov is still displeased with me. 😀

I've marked as a draft while I continue to work on this. I'll mark it ready when I feel that I've gotten it to a good place again. I have lots of lovely feedback to act upon, noted above, so changes to be made.

As for codecov, I'm not totally sure. I clearly misdiagnosed the "loss of coverage" it's flagging, but the reporting it shows doesn't quite match my expectations there. I think it might be that it sees the missing else branches of various conditionals as "untested" because they aren't explicit and flagged with appropriate pragmas. That will be one of my next experiments.

In order to support these, the logic of the plugin had to become a bit more organized -- it now uses a list of supported operators to build out the full suite of pragmas. With these new pragmas available, update the various usage sites to use the affirmative 'cover' pragmas on comparisons, with the corresponding 'no cover' pragmaes on else branches. To improve branch coverage, all of the `else` branches are explicitly enumerated and captured with the appropriate pragmas, even though many are `else: pass`.

This is added to the python path via pytest options, and is excluded from pytest discovery (norecursedirs). Also, add more pip-version pragmas to test code which only runs on specific pip versions.

In order to determine the lowest supported version of `pip`, read the `project.dependencies` list from `pyproject.toml`, find the only listed `pip` dependency, and pull its specifier. If any of the data does not match expectations, a ValueError will be thrown (crashing any run of coverage/pytest-cov) with a message stating that the plugin needs to be updated. This lets us get a working solution without trying to solve a more general case than what we need.

Unit tests of the local coverage plugin exercise its various helpers, and the module-level docstring explains that we can test the parts even if we can't easily measure code coverage on the plugin itself.

These tests need `coverage` to be installed, and it is missing 1. In CI builds on pypy where`coverage` is not installed 2. When local tests run without the `coverage` factor

sirosen · 2026-03-06T04:29:37Z

I moved the plugin into tests/_plugins/_coverage/, and tweaked our pytest config to put it in pythonpath but exclude it from test discovery. That works!

If we go with this approach, I plan to write a short blog post to share the technique of a local coverage plugin. It's not something I've seen anyone write about before, and it wasn't hard but it did have some non-obvious requirements (most notably, pythonpath manipulation).

I've enhanced the plugin with pragma: pip{comparator}X.Y cover pragmas, and added a suite of unit tests to help get those bits correct. I find these easy to get confused when implementing -- there's some reasoning about double-negatives involved in checking this. I've been doing some local test runs and looking at coverage reporting, and it seems to be working as desired.

I've also explicitly expanded a lot of else branches for pip-version conditionals, almost all of which are else: pass, so that they can be marked with the appropriate pragmas. That fixed coverage branch reporting showing up as partial, and seems to have gotten codecov to pass. As part of that body of work, I aligned the pragmas with the comparisons being done. They now typically read like so:

if _pip_api.PIP_VERSION_MAJOR_MINOR >= (22.3):  # pragma: pip>=22.3 cover
    ...
else:  # pragma: pip>=22.3 no cover
    ...

That is, I've been using the matching comparison in both branches, but with cover vs no cover.

Deriving the minimum pip version from pyproject.toml involved making a few decisions:

How do we find pyproject.toml? I went with cwd() / pyproject.toml, since tox will always execute from the repo root.
What do we do if there are 0 or multiple pip versions listed? I treat this as an error.
How sophisticated do we want to be in parsing the pip version specifier? Again, I treated many cases as errors.

My reasoning around bailing out early with errors is that we definitely don't want to try to solve the general case of "what is the minimum version of package X which satisfies this dependency list?" as it's a hard problem. If we make some simplifying assumptions -- and treat anything unexpected as an error -- we get an easy problem for now and can improve it if we ever need to.

I'm watching CI to see if it comes up healthy. If everything looks okay, I feel like I can mark this ready.

Local coverage runs won't report on this file, as configured, but codecov is still flagging it as uncovered code.

webknjaz and others added 3 commits March 3, 2026 18:38

Exclude tests marked as xfail from coverage

9929d98

These are either chronically uncovered up to a certain line or have flaky coverage. Either way, having them in is rather harmful for measurements.

🧪 Add conditional coverage pragmas if possible

795c846

These are version-dependent but are sometimes applied to conditionals checking the deps versions. They won't work perfectly but it's a start.

sirosen requested a review from a team as a code owner March 5, 2026 01:44

psf-chronographer bot added the bot:chronographer:provided label Mar 5, 2026

Add plugins/* to coverage excludes

e75b3f7

When looking at lost coverage reporting in codecov, it appears that it is counting the new plugin code as "uncovered". Though true -- it isn't being tested rigorously -- trying to measure it seems incorrect at present.

webknjaz reviewed Mar 5, 2026

View reviewed changes

sirosen marked this pull request as draft March 5, 2026 20:40

sirosen added 5 commits March 5, 2026 19:38

Move local coverage plugin into 'tests/_plugins/'

32bf783

This is added to the python path via pytest options, and is excluded from pytest discovery (norecursedirs). Also, add more pip-version pragmas to test code which only runs on specific pip versions.

Add unit testing of piptools_coverage.py

1d902d1

Unit tests of the local coverage plugin exercise its various helpers, and the module-level docstring explains that we can test the parts even if we can't easily measure code coverage on the plugin itself.

Use an importorskip for local coverage plugin tests

b5743a1

These tests need `coverage` to be installed, and it is missing 1. In CI builds on pypy where`coverage` is not installed 2. When local tests run without the `coverage` factor

Ignore the coverage plugin in codecov

9d33445

Local coverage runs won't report on this file, as configured, but codecov is still flagging it as uncovered code.

sirosen marked this pull request as ready for review March 6, 2026 04:52

		config.set_option("report:exclude_lines", sorted(exclude))


		def coverage_init(registry: Plugins, options: dict[str, _t.Any]) -> None:

Uh oh!

Conversation

sirosen commented Mar 5, 2026

Contributor checklist

Maintainer checklist

Uh oh!

sirosen commented Mar 5, 2026

Uh oh!

webknjaz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sirosen commented Mar 5, 2026

Uh oh!

sirosen commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants