Conda

A cross-platform package manager

Created by Mateusz Paprocki

Before we begin — installing conda

• Miniconda (quick installation instructions)
  • only Python, conda and a few dependencies
  • small size (about 25 MB)
• Anaconda (full installation instructions)
  • Python, conda and lots of common packages (>=150)
  • big download (about 300 MB)

Miniconda installers

Install on Linux and Mac OS X:

$ bash Miniconda-latest-Linux-x86_64.sh # or MacOSX instead of Linux

Install on Windows:

Miniconda-latest-Windows-x86_64.exe

Presentation plan

• about package management in general
• package management in Python
  • setup.py, pip, wheels, ...
• conda
  • introduction
  • commands
  • packages
  • channels
  • environments
  • building packages
  • disadvantages
• final words

Why we care about package management?

• user's perspective:
  • seamlessly install software
  • possibly not an expert
• developer's perspective:
  • allow users to use his programs
  • developer is a user as well

We all simply don't want to waste time.

Package manager or package management system

• is collection of software tools
• automates the process of software:
  • installation
  • configuration
  • upgrading
  • removal
  • ...
• allows to resolve dependencies
• allows for reliability and reproducibility

Package managers

• chocolatey
• npackd

• macports
• homebrew

• apt-get (dpkg)
• yum (rpm)

Package managers

Package management in Python

The history

Installing

• setup.py install
• easy_install
• pip
• apt-get
• rpm
• emerge
• homebrew
• ...

Why not use a system package manager?

• missing or (very) outdated packages
• hard to get packages into certain repositories
• requires duplication of packages for different distributions or operating systems
• usually requires administrator privileges

setup.py install

• fine if it's pure Python, not so much if it isn't
• you have to have compilers installed

  distutils.errors.DistutilsError: Setup script exited with
  error: command 'gcc' failed with exit status 1

setup.py install — installing NumPy

$ git clone -b v1.10.1 git@github.com:numpy/numpy.git
$ cd numpy
$ python setup.py install
Running from numpy source directory.
Cythonizing sources
Processing numpy/random/mtrand/mtrand.pyx
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named Cython.Compiler.Main
Traceback (most recent call last):
  File "/home/mateusz/repos/numpy/tools/cythonize.py", line 199, in <module>
    main()
  File "/home/mateusz/repos/numpy/tools/cythonize.py", line 195, in main
    find_process_files(root_dir)
  File "/home/mateusz/repos/numpy/tools/cythonize.py", line 187, in find_process_files
    process(cur_dir, fromfile, tofile, function, hash_db)
  File "/home/mateusz/repos/numpy/tools/cythonize.py", line 161, in process
    processor_function(fromfile, tofile)
  File "/home/mateusz/repos/numpy/tools/cythonize.py", line 81, in process_pyx
    raise Exception('Cython failed')
Exception: Cython failed
Traceback (most recent call last):
  File "setup.py", line 264, in <module>
    setup_package()
  File "setup.py", line 252, in setup_package
    generate_cython()
  File "setup.py", line 199, in generate_cython
    raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!

setup.py install

pip

• only works with Python
• not so great for packages with native dependencies
  • e.g. scientific packages, but not limited to
  • try installing h5py if you don't have HDF5
• doesn't have a true dependency solver

pip — installing h5py

$ pip install h5py
Collecting h5py
  Downloading h5py-2.5.0.tar.gz (684kB)
    100% |████████████████████████████████| 688kB 167kB/s
Collecting numpy>=1.6.1 (from h5py)
  Downloading numpy-1.10.1.tar.gz (4.0MB)
    100% |████████████████████████████████| 4.1MB 78kB/s
Collecting Cython>=0.17 (from h5py)
  Downloading Cython-0.23.4.tar.gz (1.6MB)
    100% |████████████████████████████████| 1.6MB 139kB/s
Collecting six (from h5py)
  Downloading six-1.10.0-py2.py3-none-any.whl
Building wheels for collected packages: h5py, numpy, Cython

(...)

In file included from /tmp/pip-build-rDj6mj/h5py/h5py/defs.c:279:0:
/tmp/pip-build-rDj6mj/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
#include "hdf5.h"
                 ^
compilation terminated.
error: command 'gcc' failed with exit status 1

pip

What about wheels?

• Python package specific
  • can't build wheels for native libraries
  • can't make a wheel for Python itself
• still doesn't address the problem that some metadata is only in the package itself
• you are still a "self integrator"

wheel — installing h5py

Packaging problem?

Conda

• system level package manager (Python agnostic)
• Python, HDF5 and h5py are all conda packages
• cross-platform (works on Windows, OSX and Linux)
• doesn't require administrator privileges (sudo)
• installs binaries (no more compiler woes)
• metadata stored separately in the repository index
• uses a SAT solver to resolve dependencies
• allows to seamlessly manage virtual environments

Basic conda usage

conda install sympy

conda list

conda search numpy

conda create -n py3 python=3

conda remove sympy

conda install --help

Advanced conda usage

conda install -n py35 sympy

conda update --all

conda list --export pkgs.txt

conda install --file pkgs.txt

conda list --revisions

conda install --revision 23

conda clean -pt

conda create -n my-an anaconda

What is a conda package?

A tarball (.tar.bz2) with:

Python agnostic

A conda package can be anything:

• pure Python packages
• Python package with C extensions
• Python itself
• C/C++/Fortran etc. libraries (GDAL, netCDF4, dynd, ...)
• R
• node.js
• Perl
• ...

Installation

$ conda install sympy

• the tarball is unarchived in the pkgs directory
• files are hard-linked to the install path
• shebang lines and other instances of a place-holder prefix are replaced with the install prefix
• the metadata is update, so that conda knows that a package is installed
• post-link script is run (these are rare)

Installation — conda install sympy

$ conda install sympy
Fetching package metadata: ....
Solving package specifications: ...................
Package plan for installation in environment /home/mateusz/miniconda:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    fastcache-1.0.2            |           py27_0          40 KB
    sympy-0.7.6.1              |           py27_0         6.2 MB
    ------------------------------------------------------------
                                           Total:         6.2 MB

The following NEW packages will be INSTALLED:

    fastcache: 1.0.2-py27_0
    sympy:     0.7.6.1-py27_0

Proceed ([y]/n)? y

Fetching packages ...
fastcache-1.0. 100% |######################| Time: 0:00:00 164.82 kB/s
sympy-0.7.6.1- 100% |######################| Time: 0:00:20 314.02 kB/s
Extracting packages ...
[      COMPLETE      ]|#####################| 100%
Linking packages ...
[      COMPLETE      ]|#####################| 100%

Environments

$ conda create -n py35 python=3.5

• environments are simple: just link package(s) to a different directory
• hard-links are very cheap and very fast
• conda environments are completely independent installations of everything
  • no fiddling with PYTHONPATH
  • no symlinking to site-packages
• "activating" an environment just means changing your PATH, so that environment's bin/ or Scripts/ comes first

Activating an environment

$ source activate py35

$ activate py35

Use cases for environments

• trying new versions of Python
• exploring new packages from PyPI
• testing (Python 2.6, 2.7, 3.3, 3.4, 3.5, ...)
• development
• reproducible science

Creating a new environment

$ conda create -n py35 python=3.5
Fetching package metadata: ....
Solving package specifications: .
Package plan for installation in environment /home/mateusz/miniconda/envs/py35:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    xz-5.0.5                   |                0         504 KB
    python-3.5.0               |                0        16.6 MB
    setuptools-18.4            |           py35_0         345 KB
    wheel-0.26.0               |           py35_1          77 KB
    pip-7.1.2                  |           py35_0         1.4 MB
    ------------------------------------------------------------
                                           Total:        19.0 MB

The following NEW packages will be INSTALLED:

    openssl:    1.0.1k-1
    pip:        7.1.2-py35_0
    python:     3.5.0-0
    readline:   6.2-2
    setuptools: 18.4-py35_0
    sqlite:     3.8.4.1-1
    tk:         8.5.18-0
    wheel:      0.26.0-py35_1
    xz:         5.0.5-0
    zlib:       1.2.8-0

Proceed ([y]/n)? y

Creating a new environment

Fetching packages ...
xz-5.0.5-0.tar 100% |#######################| Time: 0:00:01 488.65 kB/s
python-3.5.0-0 100% |#######################| Time: 0:00:19 889.30 kB/s
setuptools-18. 100% |#######################| Time: 0:00:00 546.22 kB/s
wheel-0.26.0-p 100% |#######################| Time: 0:00:00 200.20 kB/s
pip-7.1.2-py35 100% |#######################| Time: 0:00:11 127.70 kB/s
Extracting packages ...
[      COMPLETE      ]|######################| 100%
Linking packages ...
[      COMPLETE      ]|######################| 100%
#
# To activate this environment, use:
# $ source activate py35
#
# To deactivate this environment, use:
# $ source deactivate
#

$ source activate py35
discarding /home/mateusz/miniconda/bin from PATH
prepending /home/mateusz/miniconda/envs/py35/bin to PATH

$ python --version
Python 3.5.0 :: Continuum Analytics, Inc.

$ source deactivate
discarding /home/mateusz/miniconda/envs/py35/bin from PATH

$ python --version
Python 2.7.10 :: Continuum Analytics, Inc.

Channels

$ conda install -c https://conda.anaconda.org/travis simpy

• simply URLs to locations with conda packages
• allow to explore anaconda.org

  $ conda install -c travis simpy

• configure channels in .condarc

  $ conda config --add channels travis

• use conda info inspect the configuration

  $ conda info

Building

Conda Recipes

$ conda build path/to/recipe

• meta.yaml contains metadata
• build.sh is the build script for Unix and bld.bat is the build script for Windows

Example meta.yaml

Lots more

• command line entry points
• fine-grained control over conda's relocation logic
• inequalities for version of dependencies (like >=1.2, <2.0)
• "Preprocessing selectors" allow using the same meta.yaml for many platforms
• see full documentation
• conda build is only a convenient wrapper
• you can also build packages manually just by following the package specification

Sharing

• Anaconda Cloud (anaconda.org)
  • something like GitHub, but for package maintainers
  • public and private packages
  • build automation
• self-hosting with conda index
  • anaconda.org is a convenient wrapper

Downsides

• conda uses its own build of Python (!= python.org)

  $ python
  Python 2.7.10 |Continuum Analytics, Inc.| (default, Sep 15 2015, 14:50:01)
  [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  Anaconda is brought to you by Continuum Analytics.
  Please check out: http://continuum.io/thanks and https://anaconda.org
  >>>

• developed mainly by one company
• conda packages can't be used by pip
• many programs and libraries don't have a conda package
• environments may need manual activation outside bash/zsh (see e.g. fish shell issue #540 and a solution)

Unfortunate (?) features

• conda always attempts to update installed software
  • unless it was installed with explicit version number, e.g.

    conda install numpy=1.8.1

  • because people do not update their software
  • and then they report issues about arbitrary versions of obsolete software (maintenance nightmare)
• conda doesn't distinguish between manually and automatically (dependencies) installed packages
  • this is an issue when you remove a package and its dependencies are not removed as well
  • this is a bug in conda (see issue #232)

Unfortunate (?) features - example 1

$ conda install ipython
Fetching package metadata: ....
Solving package specifications: ..................
Package plan for installation in environment /home/mateusz/miniconda:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    decorator-4.0.4            |           py27_0          11 KB
    ipython_genutils-0.1.0     |           py27_0          32 KB
    path.py-8.1.2              |           py27_0          45 KB
    pexpect-3.3                |           py27_0          60 KB
    simplegeneric-0.8.1        |           py27_0           6 KB
    conda-3.18.3               |           py27_0         175 KB
    pickleshare-0.5            |           py27_0           8 KB
    traitlets-4.0.0            |           py27_0          88 KB
    ipython-4.0.0              |           py27_0         915 KB
    ------------------------------------------------------------
                                           Total:         1.3 MB

Unfortunate (?) features - example 1

The following NEW packages will be INSTALLED:

    decorator:        4.0.4-py27_0
    ipython:          4.0.0-py27_0
    ipython_genutils: 0.1.0-py27_0
    path.py:          8.1.2-py27_0
    pexpect:          3.3-py27_0
    pickleshare:      0.5-py27_0
    simplegeneric:    0.8.1-py27_0
    traitlets:        4.0.0-py27_0

The following packages will be UPDATED:

    conda:            3.18.2-py27_0 --> 3.18.3-py27_0

Proceed ([y]/n)?

Unfortunate (?) features - example 2

$ conda create -n ipy python
(...)
$ conda list -n ipy | grep -v '^#' | wc -l
9
$ conda install -n ipy ipython
(...)
$ conda list -n ipy | grep -v '^#' | wc -l
18
$ conda remove -n ipy ipython
Fetching package metadata: ....

Package plan for package removal in environment /home/mateusz/miniconda/envs/ipy:

The following packages will be REMOVED:

    ipython: 4.0.0-py27_0

Proceed ([y]/n)? y

Unlinking packages ...
[      COMPLETE      ]|####################################| 100%
$ conda list -n ipy | grep -v '^#' | wc -l
17

Final words

• conda is completely open source (BSD)
• http://conda.pydata.org/docs/get-started.html
• https://github.com/conda/conda
• mailing list: conda@continuum.io