Python Virtual Environment Tutorial
This document will guide you through setting up a Python Virtual Environment.
At this point, you should already have a folder for your project (instructions). In this tutorial, we’ll use project 1 as an example. Your folder location might be different.
$ pwd /Users/awdeorio/src/eecs485/p1-insta485-static
Restarting this tutorial
If you made a mistake with these Python instructions, here’s how to start over. First, close your shell and reopen it to ensure that environment variables are reset. Then, delete the virtual environment.
$ pwd /Users/awdeorio/src/eecs485/p1-insta485-static $ rm -rf env
Create a Python virtual environment
This section will help you install the Python tools and packages locally, which won’t affect Python tools and packages installed elsewhere on your computer.
After finishing this section, you’ll have a folder called
env/ that contains all the Python packages you need for this project.
Create a virtual environment in your project’s root directory. (More on venv and the creation of virtual environments)
$ pwd /Users/awdeorio/src/eecs485/p1-insta485-static $ python3 -m venv env
Pitfall: If you’re on OSX and have previously installed Anaconda, use the full path to the Python executable.
$ which python /Users/awdeorio/anaconda/bin/python $ rm -rf env # Remove environment from previous step and start over $ /usr/local/bin/python3 -m venv env
Pitfall: If the
PYTHONPATH environment variable is set, then this can cause problems. Check this when you first start a new shell.
$ echo $PYTHONPATH # Output isn't blank, problem! /Users/awdeorio/anaconda/lib/ $ rm -rf env # Remove environment from previous step and start over $ unset PYTHONPATH $ python3 -m venv env
Activate virtual environment. You’ll need to do this every time you start a new shell.
$ source env/bin/activate
We now have a complete local environment for Python. Everything lives in one directory. Environment variables point to this virtual environment.
$ echo $VIRTUAL_ENV /Users/awdeorio/src/eecs485/p1-insta485-static/env
We have a Python interpreter installed inside the virtual environment.
which python tells you exactly which python executable file will be used when you type
python. Because we’re in a virtual environment, there’s more than one option!
$ which python # Default python exectuable /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin/python $ which -a python # All python executables /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin/python /usr/bin/python
There’s a package manager for Python installed in the virtual environment. That will help us install Python packages later.
$ which pip /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin/pip $ pip --version pip 9.0.1 from /Users/awdeorio/src/eecs485/p1-insta485-static/env/lib/python3.6/site-packages (python 3.6) # Your version may be different
Python packages live in the virtual environment. We can see that Python’s own tools are already installed (
$ ls env/lib/python3.6/site-packages/ pip setuptools ...
Upgrade the Python tools in your virtual environment
$ pip install --upgrade pip setuptools wheel
Install HTML5 Validator. We’ll use this tool later. Your version might be different.
$ pip install html5validator $ html5validator --version html5validator 0.3.1
.gitignore and Python virtual environment
Do not commit virtual environment files (
env/) to version control. This is because
pip installs binaries inside
env/. Those binaries will be different on different operating systems and even different computers running the same OS but using different compilers.
.gitignore file you added ignores directories called
$ ls README.md env $ git status On branch master nothing to commit, working tree clean
You made a mistake in your
.gitignore if you see your Python virtual environment folder as an untracked file.
$ git status On branch master Untracked files: (use "git add <file>..." to include in what will be committed) env/ nothing added to commit but untracked files present (use "git add" to track)
Pro-tip: cleaning your repo
You can remove all untracked and git-ignored files with
$ git clean -xdi Would remove the following item: env/ *** Commands *** 1: clean 2: filter by pattern 3: select by numbers 4: ask each 5: quit 6: help What now> 1 Removing env/
Understanding Virtual Environments
This section will give more detail about virtual environments and how they work. Simply put, a virtual environment is a bunch of files (located in
env/ in this tutorial) used by Python.
An environment is a collection of environment variables that are inputs to your shell and your programs.
Print the names and values of all environment variables using the
env command. You’ll see key/value pairs used by the shell and used by programs.
$ env ... PWD=/Users/awdeorio/src/eecs485/p1-insta485-static HOME=/Users/awdeorio USER=awdeorio PATH=/usr/local/bin:/usr/bin:/bin ...
An important example of an environment variable is
PATH, which tells your shell where to look for commands like
python and so on. It’s a colon-separated list (
:). You can print the value of one variable using the dollar sign ‘
$ echo $PATH /usr/local/bin:/usr/bin:/bin $ printenv PATH # Alternative /usr/local/bin:/usr/bin:/bin $ echo $PATH | tr ':' '\n' /usr/local/bin /usr/bin /bin
Notice that each item in the list is a directory that contains executables, for example
/usr/local/bin usually contains the
python3 executable on macOS with Homebrew.
$ ls /usr/local/bin ... python3 ...
Environment variables inside a Python program
You can set any environment variable you want.
$ export MESSAGE="hello world" $ echo $MESSAGE hello world
Environment variables are accessible from programs, like this
"""test.py""" import os print(os.environ["MESSAGE"])
Set an environment variable and run the program.
$ export MESSAGE="hello world" $ python3 test.py hello world
This example shows that environment variables are simply another way to provide input to a running program.
A virtual environment is a self-contained directory that contains a Python installation and a number of additional Python packages.
As you saw earlier, the command to create a virtual environment creates a new directory,
env in this example.
$ python3 -m venv env # you ran this earlier $ ls env/ bin include lib pyvenv.cfg
The virtual environment contains a
bin/ directory with executables. It also contains a
lib/ directory where Python third party packages live. Your versions might be different.
$ ls env/bin/ ... pip python ... $ ls env/lib/python3.7/site-packages/ __pycache__ pip-19.2.3.dist-info setuptools-41.2.0.dist-info easy_install.py pkg_resources pip setuptools
pip executable installs third party packages to
lib/. Your versions of Python and jinja2 may be different.
$ ./env/bin/pip install jinja2 Successfully installed jinja2-2.11.1 $ ls env/lib/python3.7/site-packages/jinja2/ __init__.py ...
python executable in
bin/ uses the third party packages in
$ ./env/bin/python >>> import jinja2 >>> jinja2.__version__ '2.11.1'
Why virtual environments?
Virtual environments are useful when you want to install different Python programs that have different third party package dependencies. For example, you might have a virtual environment for a web course’s project, and a different one for your machine learning course’s homework assignments. The two assignments have different third party packages and different versions of those packages.
Activate a virtual environment
In the previous example, we used the virtual environment by calling its Python executable explicitly (e.g.,
./env/bin/python). As a convenience, we can temporarily make this version the default.
bin/activate script adds
env/bin to the
PATH environment variable, making it the first place to look for commands. Notice that
/Users/awdeorio/src/eecs485/p1-insta485-static/env/bin is first in the list.
$ source env/bin/activate $ echo $PATH /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin:/usr/local/bin:/usr/bin:/bin $ echo $PATH | tr ':' '\n' /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin /usr/local/bin /usr/bin /bin
Ask the shell where all the
python executables live, then which one is the default.
$ which -a python /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin/python /usr/local/bin/python /usr/bin/python $ which python /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin/python
activate script sets a
$VIRTUAL_ENV environment variable, which contains the path to the virtual environment directory.
$ echo $VIRTUAL_ENV /Users/awdeorio/src/eecs485/p1-insta485-static/env
Replicate a virtual environment
In the previous section, we created a Python virtual environment, activated it, and upgraded the Python installer tools (
wheel). We have not yet installed any new third party Python packages.
$ pwd /Users/awdeorio/src/eecs485/p1-insta485-static $ echo $VIRTUAL_ENV /Users/awdeorio/src/eecs485/p1-insta485-static/env $ pip list Package Version ---------- ------- pip 20.1.1 setuptools 47.3.1 wheel 0.34.2
requirements.txt file lists the exact third party Python packages and their versions needed to replicate another virtual environment. This is useful for ensuring that developers and production servers have identical packages with identical versions. It’s also useful for ensuring that students and the autograder have identical packages with identical versions.
See the list of package dependencies provided in a
$ cat requirements.txt astroid==2.4.2 ... zipp==3.1.0
Install the package dependencies.
$ pip install -r requirements.txt ... Successfully installed astroid-2.4.2 ... zipp-3.1.0 $ pip list Package Version ------------------ --------- astroid 2.4.2 ... zipp 3.1.0
Deactivate a virtual environment
deactivate command simply modifies two environment variables,
VIRTUAL_ENV. First, it unsets
$ deactivate $ echo $VIRTUAL_ENV # Variable not set, output is blank
PATH to it previous value, before the virtual environment was activated.
$ echo $PATH | tr ':' '\n' /usr/local/bin /usr/bin /bin
A Python virtual environment helps you manage third party packages. A pre-configured
python executable in
./env/bin/ uses the third party packages in
./env/lib/ (the name of
env/ is your choice).
Activate the virtual environment each time you start a new shell.
$ pwd /Users/awdeorio/src/eecs485/p1-insta485-static $ source env/bin/activate
activate script changes the
PATH environment variable, which temporarily changes the default
$ which python /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin/python $ which pip /Users/awdeorio/src/eecs485/p1-insta485-static/env/bin/pip