Pipenv

When it comes to manage environments and dependencies in Python projects, a very popular combination includes Virtualenvwrapper to manage environments, and pip to install/remove Python packages. Of course, every time you start working on a project you need to remember to activate the virtual environment of that particular project. For example, with Virtualenvwrapper you would use workon <your-python-project>.

A few weeks ago I found out about Pipenv, an experimental project by Kenneth Reitz. While I was reading the documentation I couldn’t help but notice how much Pipenv resembles the Javascript package manager yarn. With yarn — or npm — you have a single tool that you can use to install/remove packages and create/manage environments.

In this post I will show you how to use Pipenv in two scenarios: when starting a new project, and when installing the dependencies of a pre-existing project.

If you want to follow along, start with installing Pipenv globally:

pip install pipenv

# Starting a new Python project

Create a dummy repository and cd into it:

mkdir my-dummy-repo
cd my-dummy-repo

Create a Python 2 virtual environment:

pipenv --two

You should see this output in your terminal:

Creating a Pipfile for this project...
Creating a virtualenv for this project...
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/jack/.virtualenvs/my-dummy-repo-zmrIuv74/bin/python2
Also creating executable in /home/jack/.virtualenvs/my-dummy-repo-zmrIuv74/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/jack/.virtualenvs/my-dummy-repo-zmrIuv74

The command you have just entered creates a new virtual environment and a Pipfile, a replacement of requirements.txt. In a Pipfile, all Python dependencies are declared with a TOML syntax. To be honest, I don’t particularly like the TOML syntax, and I find the package.json file (generated by npm or yarn) much more readable. Anyway, here is how the Pipfile looks at this point:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

Let’s say that we didn’t really want to create a Python 2 environment, but a Python 3.6 environment. No big deal, with Pipenv we can remove the virtual environment with:

pipenv --rm

and create a Python 3.6 environment with:

pipenv --python python3.6

If you take a look at the Pipfile, you will see that nothing has changed. This is perfectly ok. We have just removed a virtual environment and created a new one. We didn’t change any dependency.

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

Bu where is this virtual environment?

Pipenv creates all virtual environments at the location specified by the environment variable WORKON_HOME, just as virtualenvwrapper does. You can check where the virtual environment is by typing:

pipenv --venv
# or...
echo $VIRTUAL_ENV
# or...
pip -V

In my case WORKON_HOME is /home/jack/.virtualenvs/, that’s why the output of pipenv --venv is /home/jack/.virtualenvs/my-dummy-repo-zmrIuv74.

Note: you can also activate, deactivate, remove this virtual environment with the same commands you would use for Virtualenvwrapper: workon <your-virtual-environment>, deactivate, rmvirtualenv <your-virtual-environment>.

Let’s install some packages.

Let’s say that for your project you need pandas. You can install the latest version with:

pipenv install pandas

Of course you can also choose a specific version when installing a dependency:

pipenv install requests==2.13.0

Finally, let’s install a development dependency:

pipenv install ddt --dev

If you now type pip list in the terminal, you will see:

ddt (1.1.1)
numpy (1.13.1)
pandas (0.20.3)
pip (9.0.1)
python-dateutil (2.6.1)
pytz (2017.2)
requests (2.13.0)
setuptools (36.2.4)
six (1.10.0)
wheel (0.30.0a0)

Your Pipfile will look like this:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[dev-packages]
ddt = "*"

[packages]
pandas = "*"
requests = "==2.13.0"

Now your dependencies are set on your system, but you still need to “freeze” them for everyone else. With the usual pip workflow, you could use pip freeze > requirements.txt. With Pipenv you have to type a lot less:

pipenv lock

which gives you this output:

Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!

The first time I used pipenv, it felt a bit weird. I am familiar with yarn, and with yarn I don’t have to run two commands to install/lock the dependencies. I would just need to run yarn add d3, or yarn add webpack --dev. In fact, someone opened an issue on GitHub and asked Kenneth why the need of manually invoking pipenv lock after pipenv install (or pipenv uninstall). Pipenv is still in an early stage, and Kenneth commented that he decided to have two commands because pipenv lock is quite slow (on my computer, locking the requirements for this dummy project took roughly 10 seconds).

It should be noted that you can easily combine install and lock in a single command:

pipenv install pandas --lock

Now that all dependencies are locked, let’s have a look at the Pipfile.lock file (basically a json file):

{
  "_meta": {
    "hash": {
      "sha256":
        "a770116c27db7e68723c431e7a00bc50e32d6ae6c719998e01a332da646757b6"
    },
    "requires": {},
    "sources": [
      {
        "url": "https://pypi.python.org/simple",
        "verify_ssl": true
      }
    ]
  },
  "default": {
    "numpy": {
      "version": "==1.13.1"
    },
    "pandas": {
      "version": "==0.20.3"
    },
    "python-dateutil": {
      "version": "==2.6.1"
    },
    "pytz": {
      "version": "==2017.2"
    },
    "requests": {
      "version": "==2.13.0"
    },
    "six": {
      "version": "==1.10.0"
    }
  },
  "develop": {
    "ddt": {
      "version": "==1.1.1"
    }
  }
}

Another handy command is pipenv update, which updates pip to the latest version, uninstalls all packages, and finally re-installs all packages declared in the [packages] section of the Pipfile (for each package, it installs the latest compatible version).

Updating all dependencies from Pipfile...
Found 6 installed package(s), purging...
Uninstalling ddt-1.1.1:
  Successfully uninstalled ddt-1.1.1
Uninstalling numpy-1.13.1:
  Successfully uninstalled numpy-1.13.1
Uninstalling pandas-0.20.3:
  Successfully uninstalled pandas-0.20.3
Uninstalling python-dateutil-2.6.1:
  Successfully uninstalled python-dateutil-2.6.1
Uninstalling pytz-2017.2:
  Successfully uninstalled pytz-2017.2
Uninstalling requests-2.13.0:
  Successfully uninstalled requests-2.13.0

Environment now purged and fresh!
Pipfile found at /home/jack/Repos/my-dummy-repo/Pipfile. Considering this to be the project home.
Installing dependencies from Pipfile.lock...
Requirement already satisfied: six==1.10.0 in /home/jack/.virtualenvs/my-dummy-repo-zmrIuv74/lib/python3.6/site-packages
Collecting pytz==2017.2
  Using cached pytz-2017.2-py2.py3-none-any.whl
Collecting python-dateutil==2.6.1
  Using cached python_dateutil-2.6.1-py2.py3-none-any.whl
Collecting requests==2.13.0
  Using cached requests-2.13.0-py2.py3-none-any.whl
Collecting numpy==1.13.1
  Using cached numpy-1.13.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting pandas==0.20.3
  Using cached pandas-0.20.3-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: pytz, python-dateutil, requests, numpy, pandas
Successfully installed numpy-1.13.1 pandas-0.20.3 python-dateutil-2.6.1 pytz-2017.2 requests-2.13.0

⠸
All dependencies are now up-to-date!

Note: you should put Pipfile and Pipfile.lock under version control.

# Installing dependencies of a pre-existing project

Some weeks ago I discovered Plotly Dash, a cool project based on React.js and the Plotly API. Dash allows you to develop reactive web apps and interactive dashboards in Python. I felt super-exited about it, because I have been looking for the Python equivalent of the R Shiny package for a long time. I also wanted to try Pipenv in a real project, so I decided to develop a small project with Dash and to use Pipenv to manage the virtual environment and the project dependencies.

You can clone my Dash project via SSH:

git clone git@github.com:jackdbd/dash-earthquakes.git

Then go to the project directory:

cd dash-earthquakes

Now you have to create the virtual environment. I developed this project in a Python 3.6 environment, so it makes sense for you to use the same setup:

pipenv --python python3.6

Install all dependencies. You could use the --dev flag to include all development dependencies, but in this small project I don’t have them:

pipenv install

You can also update and lock the dependencies once again:

pipenv update
pipenv lock

Finally, you can have a look at the packages available in this virtual environment by typing:

pip list

# Pipenv and Travis CI

I use Pipenv for dash-earthquakes, a very simple dashboard that displays the most recent eathquakes in the world (if you are interested in how I created this application, I wrote an article about it).

I wanted to setup Travis for this project, but it took me a while to figure out how to use pipenv in this environment. A possible solution comes from the pipenv advanced documentation: use a Makefile to install pipenv and run the tests.

Here is how the Makefile looks like.

help:
	@echo '    init'
	@echo '        install pipenv and all project dependencies'
	@echo '    test'
	@echo '        run all tests'

init:
	@echo 'Install python dependencies'
	pip install pipenv
	pipenv install

test:
	@echo 'Run all tests'
	pipenv run py.test tests

And here is a basic .travis.yml file.

language: python
python:
  - 3.5
  - 3.6
install:
  - make init
script:
  - make test

# Pipen and PyUp

I usually keep track of my Python dependencies with the PyUp bot. Unfortunately, it seems that PyUp does not support pipenv at the moment.

Of course you can activate the virtual environment and create a requirements.txt with pip freeze > requirements.txt, but you would still need to check that file in, to make it available for PyUp. That is confusing, because pipenv don’t use requirement files.

• article
• Python
• virtual environment