Tip/Python: Speeding up Dependency Resolutions

written by Joseph Nix on 2023-07-12

As of a few years ago, resolving dependencies for your Python project started getting a lot better, because various tools started becoming much smarter about how they choose dependencies, and respecting the version constraints in your project, and all of the implicit constraints brought in by your direct dependencies, and their dependencies.

Trouble is, sometimes a completely consistent set of dependencies that obeys all constraints from each package is not necessarily easy. Some packages like AWS's boto3 have tight constraints and many, frequent releases. This can, sometimes rightfully, cause your tool of choice, like pip or Poetry, to search many different versions of dependencies in order to find the best match, or a match at all. So if your resolver is taking too long, if you've got your coffee, and are nearly finishing it, and the resolver is still going, just Ctrl-C that process. You don't have to wait all eternity.

Instead, run that command again, verbosely (e.g.poetry lock -vv), and see if you can figure out what's going on. In my experience, the culprit is very often boto3 / botocore, because they have so many published versions. And the last time this happened to me, it was indeed boto3 that was hanging things up.

So, the temporary solution here is to constrain boto3 directly, even if it is an indirect dependency of your project, just so you can tell your packaging tool to not worry about the million releases available, and just look at a few. In Poetry, you could add to your pyproject.toml

boto3 = {version = ">=1.28.1"}  # constrained to speed up resolution

And then voila, your resolutions will be snappy again, or at least that limiting package will be constrained. Perhaps you have more, but often for me, it's basically always just boto3. If the constraint you choose actually doesn't produce a consistent resolution, then you'll have to experiment a little until you can find one.

Now, you might ask, "Why is this temporary?" Well, if you use a constraint without an upper bound, like version = ">=1.28.1", then you'll eventually have to again check far too many after enough real-world time passes. Perhaps years, but if boto3 keeps updating, it'll keep lengthening the resolution time. You could make a similar update in the future when that becomes a problem, or put an upper bound on the constraint. That choice is a different topic though. There are reasons you may want either.