March 16, 2021

Setting Up Your Python Environment

Brian Lipp

If you run into issues with versioning and package management when working on multiple Python projects, don't worry, you're not alone. I will introduce two projects to hopefully help you tackle these issues: Pyenv and Pipenv. You might be thinking "why didn't he mention the Conda project?" Well, I chose not to include it simply because, in my experience, Conda and the pip resolving package dependency issues cause problems 90% of the time. My approach should work 100% of the time and allow you to fully control your Python environment. Let's dive in!

Pyenv

Pyenv is a project you can use to control Python’s version. More often than not, issues arise when moving to different versions of Python. For example, jumping from 3.7 to 3.9 introduces significant features that do not exist in older versions.

One criticism I have of Pyenv is that it's installed via a script, making it hard to install code in some corporate environments. I will assume your work environment is Linux or Mac. For windows please follow these directions.

Install

Installing Pyenv is typically done via running the following script:

curl https://pyenv.run | bash

Shell Setup

Pyenv requires you to run the following code in the shell prior to usage. This can be run via the .bashrc file at the initialization of the shell.

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Install Python

Next, we will run the install argument with a version, this will install on the system of Python version we requested.

pyenv install 3.9.2
pyenv versions

Activate

Once your environment is created, all that's needed is to activate the environment.

pyenv global 3.9.2

IDE integration

Your IDE of choice will be able to access this version of python using the following path:

~/.pyenv/versions/3.9.2/bin/python

Pipenv

Documentation

Pipenv is a project that extends the pip Python package manager. It's a significant improvement in many areas, but specifically in dependency management. Once you learn how to use Pipenv, it's very easy to accomplish any task.

Activate the Python Environment

Before we do anything, we will activate the Python environment created in the first section. Pipenv should be able to find the Python interpreter without any configuration.

pyenv global 3.9.2

Install

We will use the pip Python package manager to install pipenv, and then we will stop using pip.

pip install pipenv

Create a Pipenv Environment

Now that pipenv is installed we are going to create an example project folder, and then initialize pipenv in that folder.

mkdir my_project
cd my_project
pipenv --python 3.9.2 install
pipenv lock

Pipfile & lock file

Pipenv uses two main files: the Pipefile specifies packages for your environment and the Pipe.lock file, generated from the Pipefile with:

pipenv lock

The lock file lists all dependencies used for the project, including sub dependencies. You should commit both files to your git repo.

The Pipfile is used to generate the lock file and contains configurations like repository and Python version.

[[source]]
#url = "your pypi url for packages"
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pandas = "==0.24.2"
requests = "*"

[requires]
python_version = "3.9"

Syncing with the Lock File

Once you have a lock file, you can now sync your environment with what is defined in the lock file:

pipenv sync

If you want to manually install a package into your environment you can:

$ pipenv install requests~=1.2

This will update your Pipenv file.

Running Code

Now that our environments are set up, running code is trivial. All we need to do is:

pipenv run python3 my_python_app.py

Shell

If you want the terminal set to the environment, set it by using:

pipenv shell

Updating packages

If you want a list of outdated packages, run this to see all the packages needing an upgrade:

pipenv update --outdated

You can choose to blindly update all packages:

pipenv update

Or, you can update one package at a time:

pipenv update pandas

Requirements.txt

If you have a legacy requirements.txt file, Pipenv can use that specific file:

pipenv install -r requirements.txt

If you find that for some reason you need to generate a requirements.txt file, you can run:

pip freeze > requirements.txt

Now, you can manage both Python interpreter and your Python packages. It's important to control your environment for every project because it increases repeatability and reduces time wasted troubleshooting odd issues.

Let me know if you try these methods and how it affects working in your Python environment. If you want to talk specifics or learn more about setting up Python, feel free to send me a message. Our data engineers at SingleStone work with multiple distribution systems and are eager to hear about your experience.

Contributors

Brian Lipp

Alumni
Go to bio