Managing Python Dependencies Robustly.

I wanted to write a blog post about something I learned this week. We use a standard set of open source packages in our projects. Almost all of them are pretty stable and cause no problems, but on some rare occasions, rarer than in the case of code I write, there’s a bug or two that gets pushed into PyPi. Or it may not even be a bug, like a newer version that stops supporting Python 2.

So in this particular instance, our project was using the django-scarface package, that helps us sending those SNS messages pretty easily. The mistake we did, was not pinning down the version in the requirements.txt file.

The reason why I want to talk about this is to demonstrate how such a mistake can sneak up on you in production. The way the python packaging manager PIP works is that whenever you do not specify a version along with the package name, it always installs the latest version of the package. This part is probably expected by everyone. But if the package is already installed and you don’t specify a version, it will do nothing even if there is a newer version available.

So here goes a scenario that will make this bug unnoticed until very late:

So all we need to do is specify the stable versions of all the packages we use as dependencies and this problem will never occur. And you will never have to bother about ever again. Right? Not really.

These awesome OSS packages that we use to make our lives simple, can also make the same sort of mistakes that we do. They too can overlook the need to pin down their dependencies, and in that case you will run into the same problem as above or something even worse. An example of this is when Kombu, a messaging library used by Celery, released their 4.2 version that broke Celery. Celery which was careful in not leaving out a version range for Kombu in their requirements.txt, still had to face the issue as their requirements.txt read something like kombu>=4.0.2,<5.0.

How do we detect breakages like these? A CI/CD pipeline that builds and runs tests can help us in detecting PIP installation errors in some cases. The catch here is that we CI/CD pipelines get triggered only when new code is pushed. Also in the case of Celery the installation did not raise any errors, it was only when Celery service was trying to boot up it threw an error. Having an error logging service like Sentry can help in these cases. Monitoring services that run detailed health checks like executing a critical part of the system and ensuring everything works as expected, is another way to deal with this. These practices are well recommended to handle production outages/errors in general, and they’ll also work for our case.

That’s it! That’s all I have to say. The younger more naive version of myself would have trouble paying attention to such details. But I have witnessed some really ugly stack traces in Sentry which are not only helpful in debugging the code but more importantly, they often reveal how seemingly harmless mistakes can lead to breakage in live systems.

Related posts:

Pertama kali saya membuat artikel di medium untuk akun DoniAn Tutorial saya akan bagaimana cara mendaftar atau membuat akun di medium.com Tapi karena ini adalah artikel pertama di medium dan akun…

While everyone seems to “want” SEO, no one outside the digital marketing world seems to know what it is or why it’s so crucial to having a website, running a blog, or owning an online business. So if…

One of the first projects to grasp the vision of Pocket Network, Portis.io, has now fully integrated Pocket-JS-Core software into their Web3 Provider. This means that with a simple config, DApps…