Creating you own conda environment
This guide will go through creating a conda environment on NERSC
installing the IPS Framework using Option 2: Module + source activate
First, you need to load the python module, then create and activate a
new conda environment. This will create the conda environment in your
home directory
module load python
conda create --name my_ips_env python=3.8 # or any version of python >=3.6
source activate my_ips_env
Next, get download the IPS Framework and install it into the conda
environment
git clone https://github.com/HPC-SimTools/IPS-framework.git
cd IPS-framework
python -m pip install ipsframework
To leave your environment
The example below show how to select the newly create conda
environment to run use, see Running Python in a batch job
#!/bin/bash
#SBATCH --constraint=haswell
#SBATCH --nodes=1
#SBATCH --time=5
module load python
source activate my_ips_env
ips.py --config=simulation.config --platform=platform.conf
Creating a shareable environment on /global/common/software
Creating an conda environment on /global/common/software is the
recommend way to have one environment shared between many uses, this
is covered by Option 4a: Install your own Python without containers.
There may also be performance benefits to running from this location
instead of your home directory.
Following the instruction we do
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p /global/common/software/myproject/env
source /global/common/software/myproject/env/bin/activate
Then install IPS into the environment, from within the IPS-framework
source directory:
python -m pip install ipsframework
The example below show how to select the newly create conda
environment to run use, see Running Python in a batch job
#!/bin/bash
#SBATCH --constraint=haswell
#SBATCH --nodes=1
#SBATCH --time=5
source /global/common/software/myproject/env/bin/activate
ips.py --config=simulation.config --platform=platform.conf
Installing dependencies
To see which packages are currently install in your environment run:
You can install just the dependencies you need by
conda install matplotlib netcdf4 ...
If you would like the same versions and dependencies in your conda
environment as found in the python modules on Cori, you can export
that environment and set your environment to be the same.
Export python/3.7-anaconda-2019.10
to yml file.
module load python/3.7-anaconda-2019.10
conda env export --name base > environment.yml
# remove mpi4py and buildtest from environment.yml as these should be installed manually
sed -i '/mpi4py/d' environment.yml
sed -i '/buildtest/d' environment.yml
Activate your conda environment and force it to match the
environment.yml
file. mpi4py
should be install separately
according to NERSC.
source activate my_ips_env
conda env update -n my_ips_env --file environment.yml
# or
source /global/common/software/myproject/env/bin/activate # your environment
# setup base environment
conda env update -n base --file environment.yml
Install mpi4py
if needed
MPICC="$(which cc) --shared" python -m pip install --no-binary mpi4py mpi4py
Alternatively you can start with the same Anaconda environment which
will have almost everything you need already and then just install the
few missing dependencies. As an example, this use the same Anaconda
distribution as in the python/3.7-anaconda-2019.10
module.
wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh
bash Anaconda3-2019.10-Linux-x86_64.sh -b -p /global/common/software/myproject/env
source /global/common/software/myproject/env
conda install netcdf4
MPICC="$(which cc) --shared" python -m pip install --no-binary mpi4py mpi4py
python -m pip install dask-mpi
User development
Miniconda or Anaconda can can be installed to your home directory for easy development of component wrappers.
wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh
bash Anaconda3-2019.10-Linux-x86_64.sh -b -p $HOME/anaconda3
source $HOME/anaconda3/bin/activate
conda install netcdf4
MPICC="$(which cc) --shared" python -m pip install --no-binary mpi4py mpi4py
python -m pip install dask-mpi
python -m pip install ipsframework
After which you can switch between youy development and the production
environment on atom project by
# switch to production environment
source /global/common/software/atom/cori/ips-framework-new/bin/activate
# switch bask to user development environment
source $HOME/miniconda3/bin/activate
To see which environment you are currently in you can run