Difference between revisions of "Python"

From NixOS Wiki
Jump to: navigation, search
(change link to split-page nixpkgs manual)
Line 413: Line 413:
  
 
You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
 
You can also set <code>backend : GTK3Agg</code> in your <code>~/.config/matplotlib/matplotlibrc</code> file to avoid having to call <code>matplotlib.use('gtk3agg')</code>.
 +
 +
== Performance ==
 +
The derivation of cPython that is available via <code>nixpkgs</code> does not contain optimizations enabled, specifically Profile Guided Optimization (PGO) and Link Time Optimization (LTO). See [https://docs.python.org/3/using/configure.html#performance-options Configuring Python 3.1.3. Performance options]
 +
Additionally, when you compile something within <code>nix-shell</code> or a derivation; by default there are security hardening flags passed to the compiler which do have a small performance impact.
 +
 +
As of the time of this writing; these optimizations cause Python builds to be non-reproducible and increase install times for the derivation. For a more detailed overview of the trials and tabulations of discovering the performance regression; see [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717 Why is the nix-compiled Python slower?] thread on the nix forums.
 +
 +
 +
=== Regression ===
 +
With the <code>nixpkgs</code> version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:
 +
<syntaxhighlight lang=python>## Ubuntu's Python 3.8
 +
username:dir$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
 +
[7.831622750498354, 7.82998560462147, 7.830805554986, 7.823807033710182, 7.84282516874373]
 +
 +
## nix-shell's Python 3.8
 +
[nix-shell:~/src]$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
 +
[10.431915327906609, 10.435049421153963, 10.449542525224388, 10.440207410603762, 10.431304694153368]
 +
</syntaxhighlight>
 +
 +
However, synthetic benchmarks are not a reflection of a real-world use case. In most situations, the performance difference between optimized & non-optimized interpreters is minimal. For example; using <code>pylint</code> with a significant number of custom linters to go scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference, instead of 40%. Other workflows that were not performance sensitive saw no impact to their run times.
 +
 +
 +
=== Possible Optimizations ===
 +
If you run code that heavily depends on Python performance (data science, machine learning), and you want to have the most performant Python interpreter possible, here are some possible things you can do:
 +
 +
* Enable the <code>enableOptimizations</code> flag for your Python derivation. [https://discourse.nixos.org/t/why-is-the-nix-compiled-python-slower/18717/10 Example] Do note that this will cause you to compile Python the first time that you run it; which will take a few minutes.
 +
* Switch to a newer version of Python. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement; but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.
 +
* Disable hardening, although this only yields a small performance boost; and it has impacts beyond Python code. [https://nixos.org/manual/nixpkgs/stable/#sec-hardening-in-nixpkgs Hardening in Nixpkgs]
 +
 +
'''Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.'''
  
 
== See also ==
 
== See also ==

Revision as of 18:42, 29 April 2022

The Python packages available to the interpreter must be declared when installing Python.

To install, say Python 3 with pandas and requests, define a new package python-with-my-packages:

with pkgs;
let
  my-python-packages = python-packages: with python-packages; [
    pandas
    requests
    # other python packages you want
  ]; 
  python-with-my-packages = python3.withPackages my-python-packages;
in ...

You can put python-with-my-packages into your environment.systemPackages for a system-wide installation, for instance. Be mindful that pythonX.withPackages creates a pythonX-Y.Z.W-env package which is read only, so you can't use pip to install packages in, say, a virtual environment (as described below). Put python in your configuration.nix if you want to use the solution as described in the section "Python Virtual Environment".

There are several versions of Python available. Replace python3 with python2 or pypy in the above snippet according to your needs.

Explanation (optional)

We defined a function my-python-packages which takes as input a set python-packages and returns a list of attributes thereof.

Example

One way to define the python package with modules inside the default configuration file is to use the systemPackages list. In such a case, you could have the following under environment.systemPackages = with pkgs; along with the rest of your packages:

(let 
  my-python-packages = python-packages: with python-packages; [ 
    pandas
    requests
     #other python packages you want
  ];
  python-with-my-packages = python3.withPackages my-python-packages;
in
python-with-my-packages)

Development shell

withPackages env

If you need only python:

# shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
  python-with-my-packages = pkgs.python3.withPackages (p: with p; [
    pandas
    requests
    # other python packages you want
  ]);
in
python-with-my-packages.env # replacement for pkgs.mkShell

mkShell

If you need python and other dependencies:

# shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
  my-python = pkgs.python3;
  python-with-my-packages = my-python.withPackages (p: with p; [
    pandas
    requests
    # other python packages you want
  ]);
in
pkgs.mkShell {
  buildInputs = [
    python-with-my-packages
    # other dependencies
  ];
  shellHook = ''
    PYTHONPATH=${python-with-my-packages}/${python-with-my-packages.sitePackages}
    # maybe set more env-vars
  '';
}

Using alternative packages

We saw above how to install Python packages using nixpkgs. Since these are written by hand by nixpkgs maintainers, it isn't uncommon for packages you want to be missing or out of date. To create a custom Python environment with your own package(s), first create a derivation for each python package (look at examples in the python-modules subfolder in Nixpkgs). Then, use those derivations with callPackage as follows:

with pkgs;
let
  my-python-package = ps: ps.callPackage ./my-package.nix {};
  python-with-my-packages = python3.withPackages(ps: with ps; [
    (my-python-package ps)
  ]);
in ...

Package and development shell for a python project

It is possible to use buildPythonApplication to package python applications. As explained in the nixpkgs manual, it uses the widely used `setup.py` file in order to package properly the application. We now show how to package a simple python application: a basic flask web server.

First, we write the python code, say in a file web_interface.py. Here we create a basic flask web server;

#!/usr/bin/env python

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080)

Then, we create the setup.py file, which basically explains which are the executables:

#!/usr/bin/env python

from setuptools import setup, find_packages

setup(name='demo-flask-vuejs-rest',
      version='1.0',
      # Modules to import from other scripts:
      packages=find_packages(),
      # Executables
      scripts=["web_interface.py"],
     )

Finally, our nix derivation is now trivial: the file derivation.nix just needs to provide the python packages (here flask):

{ lib, python3Packages }:
with python3Packages;
buildPythonApplication {
  pname = "demo-flask-vuejs-rest";
  version = "1.0";

  propagatedBuildInputs = [ flask ];

  src = ./.;
}

and we can now load this derivation from our file default.nix:

{ pkgs ? import <nixpkgs> {} }:
pkgs.callPackage ./derivation.nix {}

We can now build with:

$ nix-build
[...]
$ ./result/bin/web_interface.py 
 * Serving Flask app ".web_interface" (lazy loading)
 [...]

or just enter a nix-shell, and directly execute your program or python if it's easier to develop:

$ nix-shell
[...]
[nix-shell]$ chmod +x web_interface.py
[nix-shell]$ ./web_interface.py 
 * Serving Flask app "web_interface" (lazy loading)
[...]

[nix-shell]$ python
Python 3.8.7 (default, Dec 21 2020, 17:18:55) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
>>>

Python virtual environment

Starting from Python 3 virtual environment is natively supported. The Python 3 venv approach has the benefit of forcing you to choose a specific version of the Python 3 interpreter that should be used to create the virtual environment. This avoids any confusion as to which Python installation the new environment is based on.

Recommended usage:

  • Python 3.3-3.4 (old): the recommended way to create a virtual environment was to use the pyvenv command-line tool that also comes included with your Python 3 installation by default.
  • Python 3.6+: python3 -m venv is the way to go.

Put your packages in a requirements.txt:

pandas
requests

Then setup the virtualenv:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Installing packages with pip that need to compile code or use C libraries will sometimes fail due to not finding dependencies in the expected places. In that case you can use buildFHSUserEnv to make yourself a sandbox that appears like a more typical Linux install. For example if you were working with machine learning code you could use:

{ pkgs ? import <nixpkgs> {} }:
(pkgs.buildFHSUserEnv {
  name = "pipzone";
  targetPkgs = pkgs: (with pkgs; [
    python39
    python39Packages.pip
    python39Packages.virtualenv
    cudaPackages.cudatoolkit_11
  ]);
  runScript = "bash";
}).env

In pip-shell.nix, and enter the environment with:

nix-shell pip-shell.nix
virtualenv venv
source venv/bin/activate


Emulating virtualenv with nix-shell

In some cases virtualenv fails to install a library because it requires patching on NixOS (example 1, example 2, general issue). In this cases it is better to replace those libraries with ones from Nix.

Let's say, that nanomsg library fails to install in virtualenv. Then write a shell.nix file:

let
  pkgs = import <nixpkgs> {};
  nanomsg-py = ...build expression for this python library...;
in pkgs.mkShell {
  buildInputs = [
    pkgs.python3
    pkgs.python3.pkgs.requests
    nanomsg-py
  ];
  shellHook = ''
    # Tells pip to put packages into $PIP_PREFIX instead of the usual locations.
    # See https://pip.pypa.io/en/stable/user_guide/#environment-variables.
    export PIP_PREFIX=$(pwd)/_build/pip_packages
    export PYTHONPATH="$PIP_PREFIX/${pkgs.python3.sitePackages}:$PYTHONPATH"
    export PATH="$PIP_PREFIX/bin:$PATH"
    unset SOURCE_DATE_EPOCH
  '';
}

After entering the environment with `nix-shell`, you can install new python libraries with dump `pip install`, but nanomsg will be detected as installed.

Discussion and consequences of this approach are in PR https://github.com/NixOS/nixpkgs/pull/55265.

mach-nix (nixify requirements.txt)

Mach-nix is a tool that allows managing python environments with nix based on a `requirements.txt` file. There are two different ways of how mach-nix can be used, either by installing the mach-nix cmdline tool or by writing a nix expression. The latter is recommended for maximum reproducibility.

An example for a nix expression using mach-nix to build a python environment defined by a list of requirements.

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    # place version number with the latest one from the github releases page
    ref = "refs/tags/3.4.0";
  }) {};
in
mach-nix.mkPython {
  # contents of a requirements.txt (use builtins.readFile ./requirements.txt alternatively)
  requirements = ''
    pillow
    numpy
    requests
  '';
}

Alternatively install the mach-nix cmdline tool nix-env -if https://github.com/DavHau/mach-nix/tarball/3.4.0 -A mach-nix and run

mach-nix env ./env -r requirements.txt
# This will generate the python environment into ./env. To activate it, execute:
nix-shell ./env

Or use a single command based on nix' flakes feature.

# nix.extraOptions = ''  experimental-features = nix-command flakes    '';
nix-shell -p nixFlakes --run "nix run github:davhau/mach-nix#with.ipython.geopandas --show-trace "

Please have a look at more examples.

micromamba

Install the micromamba package (currently in nixpkgs-21.11 and later). You can create environments and install packages as documented by micromamba e.g.

micromamba create -n my-environment python=3.7 numpy=1.21.0 -c conda-forge

To activate an environment you will need a FHS environment e.g.:

$ nix-shell -E 'with import <nixpkgs> {}; (pkgs.buildFHSUserEnv { name = "fhs"; }).env'
$ eval "$(micromamba shell hook -s bash)"
$ micromamba activate my-environment
$ python
>>> import numpy as np

Eventually you'll probably want to put this in a shell.nix so you won't have to type all that stuff every time e.g.:

let
  pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/79b7bd36a358f90ffd287b061132641175b8d31e.tar.gz") {};

  fhs = pkgs.buildFHSUserEnv {
    name = "my-fhs-environment";

    targetPkgs = _: [
      pkgs.micromamba
    ];

    profile = ''
      set -e
      eval "$(micromamba shell hook -s bash)"
      export MAMBA_ROOT_PREFIX=${builtins.getEnv "PWD"}/.mamba
      micromamba create -q -n my-mamba-environment
      micromamba activate my-mamba-environment
      micromamba install --yes -f conda-requirements.txt -c conda-forge
      set +e
    '';
  };
in fhs.env


conda

Install the package conda and run

conda-shell
conda-install
conda env update --file environment.yml

Imperative use

It is also possible to use conda-install directly. On first use, run

conda-shell
conda-install

to set up conda in ~/.conda

pip2nix

pip2nix generate nix expressions for Python packages.

Also see the pypi2nix-project (abandoned in 2019).

Contribution guidelines

Libraries

According to the official guidelines for python new package expressions for libraries should be placed in pkgs/development/python-modules/<name>/default.nix. Those expressions are then referenced from pkgs/top-level/python-packages.nix like in this example:

{
  aenum = callPackage ../development/python-modules/aenum { };
}

The reasoning behind this is the large size of pkgs/top-level/python-packages.nix.

Applications

Python applications instead should be referenced directly from pkgs/top-level/all-packages.nix.

The expression should take pythonPackages as one of the arguments, which guarantees that packages belong to the same set. For example:

{ lib
, pythonPackages
}:

with pythonPackages;

buildPythonApplication rec {
# ...

Special Modules

GNOME

gobject-introspection based python modules need some environment variables to work correctly. For standalone applications, wrapGAppsHook (see the relevant documentation) wraps the executable with the necessary variables. But this is not fit for development. In this case use a nix-shell with gobject-introspection and all the libraries you are using (gtk and so on) as buildInputs. For example:

$ nix-shell -p gobjectIntrospection gtk3 'python2.withPackages (ps: with ps; [ pygobject3 ])' --run "python -c \"import pygtkcompat; pygtkcompat.enable_gtk(version='3.0')\""

Or, if you want to use matplotlib interactively:

$ nix-shell -p gobjectIntrospection gtk3 'python36.withPackages(ps : with ps; [ matplotlib pygobject3 ipython ])'
$ ipython
In [1]: import matplotlib
In [2]: matplotlib.use('gtk3agg')
In [3]: import matplotlib.pyplot as plt
In [4]: plt.ion()
In [5]: plt.plot([1,3,2,4])

You can also set backend : GTK3Agg in your ~/.config/matplotlib/matplotlibrc file to avoid having to call matplotlib.use('gtk3agg').

Performance

The derivation of cPython that is available via nixpkgs does not contain optimizations enabled, specifically Profile Guided Optimization (PGO) and Link Time Optimization (LTO). See Configuring Python 3.1.3. Performance options Additionally, when you compile something within nix-shell or a derivation; by default there are security hardening flags passed to the compiler which do have a small performance impact.

As of the time of this writing; these optimizations cause Python builds to be non-reproducible and increase install times for the derivation. For a more detailed overview of the trials and tabulations of discovering the performance regression; see Why is the nix-compiled Python slower? thread on the nix forums.


Regression

With the nixpkgs version of Python you can expect anywhere from a 30-40% regression on synthetic benchmarks. For example:

## Ubuntu's Python 3.8
username:dir$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
[7.831622750498354, 7.82998560462147, 7.830805554986, 7.823807033710182, 7.84282516874373]

## nix-shell's Python 3.8
[nix-shell:~/src]$ python3.8 -c "import timeit; print(timeit.Timer('for i in range(100): oct(i)', 'gc.enable()').repeat(5))"
[10.431915327906609, 10.435049421153963, 10.449542525224388, 10.440207410603762, 10.431304694153368]

However, synthetic benchmarks are not a reflection of a real-world use case. In most situations, the performance difference between optimized & non-optimized interpreters is minimal. For example; using pylint with a significant number of custom linters to go scan a very large Python codebase (>6000 files) resulted in only a 5.5% difference, instead of 40%. Other workflows that were not performance sensitive saw no impact to their run times.


Possible Optimizations

If you run code that heavily depends on Python performance (data science, machine learning), and you want to have the most performant Python interpreter possible, here are some possible things you can do:

  • Enable the enableOptimizations flag for your Python derivation. Example Do note that this will cause you to compile Python the first time that you run it; which will take a few minutes.
  • Switch to a newer version of Python. In the example above, going from 3.8 to 3.10 yielded an average 7.5% performance improvement; but this is only a single benchmark. Switching versions most likely won't make all your code 7.5% faster.
  • Disable hardening, although this only yields a small performance boost; and it has impacts beyond Python code. Hardening in Nixpkgs

Ultimately, it is up to your use case to determine if you need an optimized version of the Python interpreter. We encourage you to benchmark and test your code to determine if this is something that would benefit you.

See also