Weekly Workbench Tip: Turbo-charge your Python package installation!

Sundaresh1 — Wed, 31 Jul 2024 05:33:34 GMT

Advantages offered by SAS Viya Workbench include fast availability of a compute environment, and support for multilingual analytics (Python or SAS, based on skills and preference).

When it comes to Python environments, however, users demand flexible and convenient package installation to aid rapid iteration and experimentation. The default method is to use the standard Python package installer (pip) to install packages. However, pip installs take a long time and package installation becomes complex when there are many packages and dependencies involved.

This tip suggests the use of virtual Python environments to help isolate and manage your Python package installation.

Using this opportunity, for informational purposes only, we also describe another Python installer utility called uv. The uv package is written in Rust and claims to be 10x - 100x faster than traditional pip and pip-tools, through use of a global cache and better dependency resolution. It follows a syntax very similar to pip, making it easy to adopt and use. Check out its PyPi page for more details.

IMPORTANT: Use uv based on your own discretion. The SAS Viya Workbench documentation specifies pip install as the standard approach to install packages. This tip regarding uv is only meant as a suggestion and I recommend you first try it out on a local Python environment for experience.

Virtual Environment creation

Virtual environments in Python allow you to create separate sub-environments within an existing Python environment. Any additional packages you install in the virtual environment are not accessible by the base (main) Python environment. This provides you, the Python user, benefits such as isolation, control over package dependencies, and rapid experimentation through different virtual environments. As environment definitions can be persisted, you also have an option of discarding your work in case of issues, and reverting to the base Python environment.

Creating a virtual environment in Workbench is easy and can be achieved through the following commands (for e.g., through a Terminal window on Workbench's Visual Studio Code application):

# Create a virtual env - provide your desired name in place of new_env python -m venv new_env # Activate virtual env . new_env/bin/activate

The 'uv' angle: It's also possible to create virtual environments using uv. Install uv, using pip, into your target environment just like any other Python package. Once that's done, you can create a virtual environment and activate the same.

# Install uv and upgrade pip pip install --upgrade pip uv # Create virtual env using uv uv venv new_uv_venv # Activate virtual env . new_uv_venv/bin/activate

Note how the commands for creating a virtual environment using uv are similar to traditional Python commands, i.e. python -m venv <virtual_environment_name>.

Deactivation: In order to revert to the base Python environment, use the deactivate command to leave the virtual environment. This applies for both virtual environments created through venv or uv.

# Deactivate virtual env deactivate echo "deactivated system" # OPTIONAL - Remove virtual env folder rm -rf new_env

Package installation

Once you've activated a virtual environment, install packages using the pip install command, either through a list of required packages, or a text file, conventionally named requirements.txt, containing a list of packages. Pip is a powerful utility which also provides other operations. The pip documentation is a useful reference.

# Install packages pip install --upgrade -r requirements.txt

The 'uv' angle: The uv command is as easy as the traditional pip command. Just prepend the 'uv' command to a standard pip installation command, either specifying the packages as a list, or using a requirements.txt file. As shown in the following snippet.

# Install packages from requirements.txt uv pip install --upgrade -r requirements.txt

Let's take a look at the effect. Taking an example where a minimal set of packages (for e.g., torch for machine learning, plus some additional helper packages) were specified in requirements.txt, the following animated gif shows how quickly uv downloads the same (faster than pip).

(view in My Videos)

A quick impression about uv's speed improvements

To form a quick impression, we specified an example requirements.txt file consisting of the following common machine learning and analytics packages:

torch swat tensorflow matplotlib pandas flask

Note that this is NOT meant as an exact test, rather a rough assessment which was sufficient to form an initial impression. We found that using uv took, on an average, 30% of the time compared to using traditional pip commands.

Note that the time to install can be affected by package composition and several other infrastructure-specific / local details. uv provides its own claims and test details on its Python project page. An assessment of uv is beyond the scope of this tip.

If you are interested in a detailed script containing the commands provided above, feel free to access them from this GitHub repository.

Have fun trying out package installation and virtual environment methods on SAS Viya Workbench and share your experiences.

topic Weekly Workbench Tip: Turbo-charge your Python package installation! in SAS Viya Workbench Discussion

Weekly Workbench Tip: Turbo-charge your Python package installation!

Virtual Environment creation

Package installation

A quick impression about uv's speed improvements