Python Easy Asked at GoogleAsked at ShopifyAsked at Databricks

Why do Python projects use virtual environments, and what are the modern tools for dependency management?

For Data Scientist ML Engineer Data Engineer AI / LLM Engineer

The short answer

A virtual environment is an isolated Python installation with its own site-packages, preventing version conflicts between projects sharing the same machine. Modern tooling has moved from pip-plus-requirements.txt toward lock-file-based tools: pip-tools, Poetry, and uv, which pin exact transitive dependency versions for reproducible installs.

How to think about it

The problem virtual environments solve

Two projects on the same machine might need requests==2.28 and requests==2.31. The system Python has one site-packages directory — only one version can win. A virtual environment gives each project its own isolated site-packages, so they never conflict.

How isolation works

Without venv                   With venv
──────────────────────────     ──────────────────────────────────
System Python                  System Python
└── site-packages/             
    ├── requests 2.28          project-a/.venv/
    └── pandas  1.5            └── site-packages/
                                   ├── requests 2.28
                               project-b/.venv/
                               └── site-packages/
                                   └── requests 2.31

One version wins.              Each project has its own. No conflict.

Creating and using a venv

python -m venv .venv            # create the environment
source .venv/bin/activate       # activate (macOS/Linux)
.venv\Scripts\activate          # activate (Windows)
pip install requests==2.31      # installs only into .venv
deactivate                      # back to system Python

The environment is just a directory — delete it and start over without touching the system. Always add .venv/ to .gitignore.

Requirements files vs lock files

A requirements.txt typically lists direct dependencies with loose constraints (requests>=2.28). Running pip install on two different machines can resolve different transitive dependency trees — breaking reproducibility.

A lock file pins every package (direct and transitive) to an exact version and hash:

requests==2.31.0 \
    --hash=sha256:58cd2187...
certifi==2024.2.2 \
    --hash=sha256:abc123...

Anyone running pip install -r requirements.lock gets byte-for-byte the same environment.

Modern tools

Tool	Lock file	Speed	Notes
pip-tools	`requirements.txt` (compiled)	Moderate	Minimal, widely used
Poetry	`poetry.lock`	Moderate	Full project manager: build, publish
uv	`uv.lock`	Very fast (Rust)	Drop-in pip replacement; PEP 517/518
conda	`environment.yml`	Moderate	Handles non-Python deps; common in ML

# uv — fastest modern workflow
pip install uv
uv venv
uv pip install pandas scikit-learn
uv pip freeze > requirements.txt

PEX and containerised environments

pex bundles a virtual environment into a single executable zip — useful for shipping a CLI tool or a Spark job without managing a venv on every worker node. Docker images serve the same purpose for server workloads.

Learn it properly Environments & packaging: venv, uv, pex