If you run into issues with versioning and package management when working on multiple Python projects, don’t worry, you’re not alone. I will introduce two projects to hopefully help you tackle these issues: Pyenv and Pipenv. You might be thinking “why didn’t he mention the Conda project?” Well, I chose not to include it simply because, in my experience, Conda and the pip resolving package dependency issues cause problems 90% of the time. My approach should work 100% of the time and allow you to fully control your Python environment. Let’s dive in!
Pyenv is a project you can use to control Python’s version. More often than not, issues arise when moving to different versions of Python. For example, jumping from 3.7 to 3.9 introduces significant features that do not exist in older versions.
One criticism I have of Pyenv is that it’s installed via a script, making it hard to install code in some corporate environments. I will assume your work environment is Linux or Mac. For windows please follow these directions.
Installing Pyenv is typically done via running the following script:
curl https://pyenv.run | bash
Pyenv requires you to run the following code in the shell prior to usage. This can be run via the .bashrc file at the initialization of the shell.
export PATH="$HOME/.pyenv/bin:$PATH" eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)"
Next, we will run the install argument with a version, this will install on the system of Python version we requested.
pyenv install 3.9.2 pyenv versions
Once your environment is created, all that’s needed is to activate the environment.
pyenv global 3.9.2
Your IDE of choice will be able to access this version of python using the following path:
Pipenv is a project that extends the pip Python package manager. It’s a significant improvement in many areas, but specifically in dependency management. Once you learn how to use Pipenv, it’s very easy to accomplish any task.
Activate the Python Environment
Before we do anything, we will activate the Python environment created in the first section. Pipenv should be able to find the Python interpreter without any configuration.
pyenv global 3.9.2
We will use the pip Python package manager to install pipenv, and then we will stop using pip.
pip install pipenv
Create a Pipenv Environment
Now that pipenv is installed we are going to create an example project folder, and then initialize pipenv in that folder.
mkdir my_project cd my_project pipenv --python 3.9.2 install pipenv lock
Pipfile & lock file
Pipenv uses two main files: the Pipefile specifies packages for your environment and the Pipe.lock file, generated from the Pipefile with:
The lock file lists all dependencies used for the project, including sub dependencies. You should commit both files to your git repo.
The Pipfile is used to generate the lock file and contains configurations like repository and Python version.
[[source]] #url = "your pypi url for packages" url = "https://pypi.python.org/simple" verify_ssl = true name = "pypi" [packages] pandas = "==0.24.2" requests = "*" [requires] python_version = "3.9"
Syncing with the Lock File
Once you have a lock file, you can now sync your environment with what is defined in the lock file:
If you want to manually install a package into your environment you can:
$ pipenv install requests~=1.2
This will update your Pipenv file.
Now that our environments are set up, running code is trivial. All we need to do is:
pipenv run python3 my_python_app.py
If you want the terminal set to the environment, set it by using:
If you want a list of outdated packages, run this to see all the packages needing an upgrade:
pipenv update --outdated
You can choose to blindly update all packages:
Or, you can update one package at a time:
pipenv update pandas
If you have a legacy requirements.txt file, Pipenv can use that specific file:
pipenv install -r requirements.txt
If you find that for some reason you need to generate a requirements.txt file, you can run:
pip freeze > requirements.txt
Now, you can manage both Python interpreter and your Python packages. It’s important to control your environment for every project because it increases repeatability and reduces time wasted troubleshooting odd issues.
Let me know if you try these methods and how it affects working in your Python environment. If you want to talk specifics or learn more about setting up Python, feel free to send me a message. Our data engineers at SingleStone work with multiple distribution systems and are eager to hear about your experience.