Couple of weeks ago, I decided to properly consolidate the requirements for django based apps that are already version controlled. I didn't use to care much about the dependencies since it was relatively straightforward to get up and running by merely fetching those dependencies using PIP.
The problem mainly lied in all the packages being deployed in the global site packages on my system which meant that I could not figure out the dependencies automatically but had to manually detect them. To further understand how to manage packages in python, you can refer to
easy_install and
pip.
Here's a sample of the problem at hand, let's consider we have a project called
Foo and a project called
Bar lying in our
~/projects folder.
We assume that Foo has a dependency on the package Alpha and Beta that were installed using PIP
We assume that Bar has a dependency on the package Gamma that was installed using PIP
Here's a rough log of how the installation was done: (although the example is using only 3 dependencies in total, consider the case where Foo has 10 dependencies and Bar has 20)
~/projects/Foo$ sudo pip install Alpha
Alpha installed...
~/projects/Foo$ sudo pip install Beta
Beta installed...
~/projects/Bar$ sudo pip install Gamma
Gamma installed...
Now I want to consolidate the requirements of Foo in a file and version it so that other developers checking out the code would be able easily fetch the dependencies and get up and running with Foo instantly. In order to do that with PIP, I simply freeze the requirements:
~/projects/Foo$ pip freeze
Alpha==1.0.2
Beta==1.0.1
Gamma=2.0.0
Zeta=1.0.0
Bobo=1.1.1
Wait what? Why on earth is Gamma there! and where the hell did all the other packages come from! Well, since pip and easy_install, drop the packages in the global site packages folder, they cannot distinguish the dependencies per project. This is why, it relatively tedious (albeit not impossible) to consolidate the requirements without having to drop those dependencies within the project themselves.
P.S:
pip freeze will merely output to screen, in order to save you would simply
pip freeze > requirements
Virtualenv to the rescue
Essentially
virtualenv solves more than just dependency consolidation problem, it solves the dependencies and versioning problem across projects. virtualenv is simply a "Tool to create isolated Python environments" as its written in the official documentation on
Pypi.
You can use virtualenv and easyinstall, or virtualenv and pip. I prefer using
virtualenvwrapper (self explanatory name) and pip since they're less verbose than the earlier. I'm not going to cover virtualenv but i'll jump straight to virtualenvwrapper, if you want to get more details on virtualenv alone (and i don't see why you need to), go ahead and do so.
Let's install virtualenv and virtualenvwrapper:
$ sudo pip install virtualenv
$ sudo pip install virtualenvwrapper
virtualenvwrapper dropped a script in
/usr/local/bin called
virtualenvwrapper.sh, this script is responsible for setting up all the actions that you can invoke on virtualenv. You will need to define two entries in your main shell configuration but they are relatively simple:
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
What is WORKON_HOME? its the location of your virtual environments.
Let's get dirty with some samples: (assume that the previous packages alpha, gamma and the lot were already installed, prior to virtualenv)
~/$ mkvirtualenv Foo
Foo env is created.
~/$ workon Foo
(Foo)~/$
What we just did is create a virtual environment and switched to it using the 'workon' command. You can identify what virtualenv you're on right now by merely looking at the string preceding the shell prompt. Let's continue:
(Foo)~/$ cdvirtualenv
(Foo)~/.virtualenv/Foo$
Notice how cdvirtualenv moved you directly into Foo's home which it identified using WORKON_HOME env variable.
(Foo)~/.virtualenv/Foo$ pip freeze
(Foo)~/.virtualenv/Foo$
Wait what? Where are the packages that were listed in our first example? Well, since this is a virtual environment, it gives you a vanilla setup that is irrespective of the global site packages (There's also a separate python version corresponding to the existing on at the time of creation, that way you get to stick with the exact same version you used when you created the project.
(Foo)~/.virtualenv/Foo$ pip install Alpha
Alpha installed...
(Foo)~/.virtualenv/Foo$ pip install Beta
Beta installed...
(Foo)~/.virtualenv/Foo$ pip freeze
Alpha==1.0.2
Beta==1.0.1
(Foo)~/.virtualenv/Foo$ deactivate
~/.virtualenv/Foo$
So far so good, let's create Bar:
~/.virtualenv/Foo$ cd ~/
~/$ mkvirtualenv Bar
~/$ workon Bar
(Bar)~/$ cdvirtualenv
(Bar)~/.virtualenv/Bar$ pip install Gamma
Gamma installed...
(Bar)~/.virtualenv/Bar$ pip freeze
Gamma==2.0.0
How nice! Now our requirements and dependencies are consolidated!
There's one more issue that needs to be tackled. Since pip downloads and installs the packages, how can we deal with similar dependencies across projects? Do we have to download the same package everytime? (That would be a bitch on lebanese connections!). Luckily there's a solution with pip. Add the following entry in your profile:
export PIP_DOWNLOAD_CACHE=$HOME/.pip_cache
Now everytime you install a package using pip, it will also drop it within the cache so that whenever you try to install a package, it will fetch it from the cache (unless a newer version is found).
Here's a small
recording to see it live. It's useless and pointless but i wanted to try recording :P
That's pretty much it for now, please note that this is in no way comprehensive and you will need to refer to most of the links here in order to get more details, but i do hope you get the gist of things.