Python dependency resolution

Today pipenv install broke on the project I was working on. It worked yesterday and the day before that, but today it failed.

At first, I thought my Python or pipenv installation was broken. The project’s integration tests ran in a Docker container, so I docker run -it <image> bash'd into a container and tried pipenv install. The result was the same—“dependencies could not be resolved”. This was a surprise…how could a Dockerized application suddenly stop working? The image worked yesterday and had not changed. I downloaded a fresh copy of the repo and tried again, but the same error occurred.

If nothing changed locally, the source of the malfunction must lie with how pipenv install operates. Looking at the Pipfile, I noticed that few dependencies were pinned to a specific version. So the version that ultimately gets installed must be decided by pipenv each time pipenv install is run. If pipenv prefers to pick newer versions, then it’s possible one of my project’s dependencies updated and somehow confused pipenv to the point of being unable to resolve dependencies it had succeeded with only 24 hours ago.

Further investigation supported this idea. The dependency conflict was over a package called PyYAML; one package required PyYAML <= 3.13 while another package required PyYAML >= 4.1. Four days ago the second package had updated it’s dependency on PyYAML from >= 3.13 to >= 4.1 (presumably) causing pipenv install to break. Pipenv was not able to figure out that the previous version of the second package only required PyYAML >= 3.13.

The resolution was to pin the package that had recently changed to its previous version. Forcing pipenv to use the old version allowed it to resolve the dependencies and generate a new lockfile. This was not a clean fix as it required adding a package that is not a direct dependency to our project’s Pipfile.

Figuring this out was not a great experience. pipenv install --verbose shows how it decides what versions to install, but reading through the rounds is tedious. pipenv graph and pipenv graph --reverse helped with identifying dependencies between packages, but this was also a tedious and manual process. In 3+ years of JavaScript, I’ve never had to deal with dependency resolution—npm “just works”. Meanwhile, literally the first project I’ve worked on that uses pipenv has resulted in hours spent trying to figure out why pipenv install stopped working.

I want to be clear that I’m not hating on pipenv. Python packaging is kind of a mess and dealing with all the legacy baggage can’t be easy. JavaScript’s packaging situation is pretty crazy as well, but much of it has been contained by npm and the existence of templates like create-react-app. Hopefully initiatives like PEP 582 – Python local packages directory will move things forward and improve the situation soon.