Developing Drivers and Components for IPS Simulations

This section is for those who wish to modify and write drivers and components to construct a new simulation scenario. It is expected that readers are familiar with IPS terminology, the directory structure and have looked at some existing drivers and components before attempting to modify or create new ones. This guide will describe the elements of a simulation, how they work together, the structure of drivers and components, IPS services API, and a discussion of control flow, data flow and fault management.

Development environment

It is suggested that for developing drivers and component that you use a separate conda environment to your production environment using the latest stable release of IPS. See Building and Setting up Your Environment.

It is also recommended to write components as there own packages, see Create a component package.

Elements of a Simulation

When constructing a new simulation scenario, writing a new component or even making small modifications to existing components and drivers, it is important to consider and understand how the pieces of an IPS simulation work together. An IPS simulation scenario is specified in the configuration file. This file tells the framework how to set up the output tree for the data files, which components are needed and where the implementation is located, time loop and checkpoint parameters, and input and output files for each component and the simulation as a whole are specified. The framework uses this information to find the pieces of code and data that come together to form the simulation, as well as provide this information to the components and driver to manage the simulation and execution of tasks 1.

The framework provides services that are used by components to perform data, task, resource and configuration management, and provides an event service for exchanging messages with internal and external entities. While these services are provided as a single API to component writers, the documentation (and underlying implementation) divides them into groups of methods to perform related actions. Data management services include staging input, output and plasma state files, changing directories, and saving task restart files, among others. The framework will perform these actions for the calling component based on the files specified in the configuration file and within the method call maintaining coherent directory spaces for each component’s work, previous steps, checkpoints and globally accessible data to insure that name collisions do not corrupt data and that global files are accessed in a well-defined manner 2. Services for task management include methods for component method invocations, or calls, and executable launch on compute nodes, or task launches. The task management portion of the framework works in conjunction with the IPS resource manager to execute multiple parallel executables within a single batch allocation, allowing IPS simulations to efficiently utilize compute resources, as data dependencies allow. The IPS task manager provides blocking and non-blocking versions of call and launch_task, including a notion of task pools and the ability to wait for the completion of any or all calls or tasks in a group. These different invocation and launch methods allow a component writer to manage the control flow and implement data dependencies between components and tasks. This task management interface hides the resource management, platform specific, task scheduling, and process interactions that are performed by the framework, allowing component writers to express their simulations and component coupling more simply. The configuration manager primarily reads the configuration file and instantiates the components for the simulation so that they can interact over the course of the simulation. It also provides an interface for accessing key data elements from the configuration file, such as the time loop, handles to components and any component specific items listed in the configuration file.

Components

There are three classes of components: framework, driver, and general purpose (physics components fall into this category). In the IPS, each component executes in a separate process (a child of the framework) and implements the following methods:

init(self, timeStamp=0)

This function performs pre-simulation setup activities such as reading in global configuration parameters, checking configuration parameters, updating input files and internal state. (Component configuration parameters are populated before init is ever called.)

step(self, timeStamp=0)

This function is the main part of the component. It is responsible for launching any tasks, and managing the input, output and plasma state during the course of the step.

finalize(self, timeStamp=0)

This function is called after the simulation has completed and performs any clean up that is required by the component. Typically there is nothing to do.

checkpoint(self, timeStamp=0)

This function performs a checkpoint for the component. All of the files marked as restart files in the configuration file are automatically staged to the checkpoint area. If the component has any internal knowledge or logic, or if there are any additional files that are needed to restart, this should be done explicitly here.

restart(self, timeStamp=0)

This function replaces init when restarting a simulation from a previous simulation step. It should read in data from the appropriate files and set up the component so that it is ready to compute the next step.

The component writer will use the services API to help perform data, task, configuration and event management activities to implement these methods.

This document focuses on helping (physics) component and driver writers successfully write new components. It will take the writer step-by-step through the process of writing basic components.

1

Tasks are the binaries that are launched by components on compute nodes, where as components are Python scripts that manage the data movements and execution of the tasks (with the help of IPS services). In general, the component is aware of the driver and its existence within a coupled simulation, and the task does not.

2

The IPS uses an agreed upon file format and associated library to manage global (shared) data for the simulation, called the Plasma State. It is made up of a set of netCDF files with a defined layout so that codes can access and share the data. At the beginning of each step the component will get a local copy of the current plasma state, execute based on these values, and then update the plasma state values that it changed to the global copy. There are data management services to perform these actions, see Data Management API.

Writing Components

In this section, we take you through the steps of adding a new component to the IPS landscape. It will cover where to put source code, scripts, binaries and inputs, how to construct the component, how to add the component to the IPS build system, and some tips to make this process smoother.

Adding a New Binary

The location of the binary does not technically matter to the framework, as long as the path can be constructed by the component and the permissions are set properly to launch it when the time comes. There are two recommended ways to express the location of the binary to the component:

  1. For stable and shared binaries, the convention is to put them in the platform’s PHYS_BIN. This way, the PHYS_BIN is specified in the platform configuration file and the component can access the location of the binary relative to that location on each machine. See Platforms and Platform Configuration.

  2. The location of the binary is specified in the component’s section of the simulation configuration file. This way, the binary can be specified just before runtime and the component can access it through the framework services. This convention is typically used during testing, experimentation with new features in the code, or other circumstances where the binary may not be stable, fully compatible with other components, or ready to be shared widely.

Data Coupling Preparation

Once you have your binary built properly and available, it is time to work on the data coupling to the other components in a simulation. This is a component specific task, but it often takes conversation with the other physicists in the group as to what values need to be communicated and to develop an understanding of how they are used.

When the physics of interest is identified, adapters need to be written to translate IPS-style inputs (from the Plasma State) to the inputs the binary is expecting, and a similar adapter for the output files.

Create a Component

Now it is time to start writing the component. At this point you should have an idea of how the component will fit into a coupled simulation and the types of activities that will need to happen during the init, step, and finalize phases of execution.

  1. Create a directory for your component (if you haven’t already). The convention in the IPS repository is to put component scripts and helpers in ips/components/<port_name>/<component_name>, where port_name is the “type” of component, and the component_name is the implementation of that “type” of component. Often, component_name will contain the name of the code it executes. If there is already a component directory and existing components, then you may want to make your own directory within the existing component’s directory or just add your component in that same directory.

  2. Copy the skeleton component (ips/doc/examples/skeleton_comp.py) to the directory you choose or created. Be sure to name it such that others will easily know what the component does. For example, a component for TORIC, a code that models radio frequency heating in plasmas, is found in ips/components/rf/toric/ and called rf_ic_toric_mcmd.py.

  3. Edit skeleton. Components should be written such that the inputs, outputs, binaries and other parameters are specified in the configuration file or appear in predictable locations across platforms. The skeleton contains an outline, in comments, of the activities that a generic component does in each method invocation. You will need to fill in the outline with your own calls to the services and any additional activities in the appropriate places. Take a look at the other example components in the ips/doc/examples/ or ips/components/ for guidance. The following is an outline of the changes that need to be made:

    1. Change the name of the class and update the file to use that name every where it says # CHANGE EXAMPLE TO COMPONENT NAME.

    2. Modify init to initialize the input files that are needed for the first step. Update shared files as needed.

    3. Modify step to use the appropriate prepare_input and process_output executables. Make sure all shared files that are changed during the course of the task execution are saved to their proper locations for use by other components. Make sure that all output files that are needed for the next step are copied to archival location. If a different task launch mechanism is required, modify as needed. See Task Launch API for related services.

    4. Modify finalize to do any clean up as needed.

    5. Modify checkpoint to save all files that are needed to restart from later.

    6. Modify restart to set up the component to resume computation from a checkpointed step.

While writing your component, be sure to use try...except blocks 3 to catch problems and the services logging mechanisms to report critical errors, warnings, info and debug messages. It is strongly recommended that you use exceptions and the services logging capability for debugging and output. Not catching exceptions in the component can lead to the driver or framework catching them in a weird place and it will likely take a long time to track down where the problem occurred. The logging mechanism in the IPS provides time stamps of when the event occurred, the component that produced the message, as well as a nice way to format the message information. These messages are written to the log file (specified in the configuration file for the simulation) atomically, unlike normal print statements. Absolute ordering is not guaranteed across different components, but ordering within the same component is guaranteed. See Logging API for more information on when to use the different logging levels.

At this point, it might be a good idea to start the documentation of the component in ips/doc/component_guides/. You will find a README file in ips/doc/ that explains how to build and write IPS documentation, and another in the ips/doc/component_guides/ on what information to include in your component documentation.

3

Tutorial on exceptions

Testing and Debugging a Component

Now it is time to construct a simulation to test your new component. There are two ways to test a new component. The first is to have the IPS just run that single component without a driver, by specifying your component as the driver. The second is to plug it into an existing driver. The former will test only the task launching and data movement capabilities. The latter can also test the data coupling and call interface to the component. This section will describe how to xstest your component using an existing driver (including how to add the new component to the driver).

As you can see in the example component, almost everything is specified in the configuration file and read at run-time. This means that the configuration of components is vitally important to their success or failure. The entries in the component configuration section are made available to the component automatically, thus a component can access them by self.<entry_name>. This is useful in many cases, and you can see in the example component that self.NPROC and self.BIN_PATH are used. Global configuration parameters can also be accessed using services call get_config_param(<param_name>) (API).

Drivers access components by their port names (as specified in the configuration file). To add a new component to the driver you will either need to add a new port name or use an existing port name. ips/components/drivers/dbb/generic_driver.py is a good all-purpose driver that most components should be able to use. If you are using an existing port name, then the code should just work. It is recommended to go through the driver code to make sure the component is being used in the expected manner. To add a new port name, you will need to add code to generic_driver.step():

  • get a reference to the port (self.services.get_port(<name of port>))

  • call “init” on that component (self.services.call(comp_ref, “init”))

  • call “step” on that component (self.services.call(comp_ref, “step”))

  • call “finalize” on that component (self.services.call(comp_ref, “finalize”))

The following sections of the configuration file may need to be modified. If you are not adding the component to an existing simulation, you can copy a configuration file from the examples directory and modify it.

  1. Plasma State (Shared Files) Section

    You will need to modify this section to include any additional files needed by your component:

    # Where to put plasma state files as the simulation evolves
    STATE_WORK_DIR = ${SIM_ROOT}/work/plasma_state
    CURRENT_STATE = ${SIM_NAME}_ps.cdf
    PRIOR_STATE = ${SIM_NAME}_psp.cdf
    NEXT_STATE = ${SIM_NAME}_psn.cdf
    CURRENT_EQDSK = ${SIM_NAME}_ps.geq
    CURRENT_CQL = ${SIM_NAME}_ps_CQL.nc
    CURRENT_DQL = ${SIM_NAME}_ps_DQL.nc
    CURRENT_JSDSK = ${RUN_ID}_ps.jso
    
    # What files constitute the plasma state
    STATE_FILES1 = ${CURRENT_STATE} ${PRIOR_STATE}
                          ${NEXT_STATE}
    STATE_FILES2 = ${STATE_FILES1}  ${CURRENT_EQDSK}
                          ${CURRENT_CQL} ${CURRENT_DQL}
    STATE_FILES = ${STATE_FILES2}  ${CURRENT_JSDSK}
    
  2. Ports Section

    You will need to add the component to the ports section so that it can be properly detected by the framework and driver. An entry for DRIVER must be specified, otherwise the framework will abort. Also, a warning is produced if there is no INIT component. Note that all components added to the NAMES field must have a corresponding subsection.

    [PORTS]
        NAMES = INIT DRIVER MONITOR EPA NB
       [[DRIVER]]
            IMPLEMENTATION = EPA_IC_FP_NB_DRIVER
        [[INIT]]
            IMPLEMENTATION = minimal_state_init
        [[RF_IC]]
            IMPLEMENTATION = model_RF_IC
    
        ...
    
  3. Component Description Section

    The ports section just defines which components are going to be used in this simulation, and point to the section where they are described. The component description section is where those definitions take place:

    [TSC]
        CLASS = epa
        SUB_CLASS =
        NAME = tsc
        NPROC = 1
        BIN_PATH = /path/to/bin
        INPUT_DIR = /path/to/components/epa/tsc
        INPUT_FILES = inputa.I09001 sprsina.I09001config_nbi_ITER.dat
        OUTPUT_FILES = outputa tsc.cgm inputa log.tsc ${STATE_FILES}
        SCRIPT = ${BIN_PATH}/epa_nb_iter.py
    

    The component section starts with a label that matches what is listed as the implementation in the ports section. These must match or else the framework will not find your component and the simulation will fail before it starts (or worse, use the wrong implementation!). CLASS and SUBCLASS typically refer to the directory hierarchy and are sometimes used to identify the location of the source code and input files. Note that NAME must match the python class name that implements the component. NPROC is the number of processes that the binary needs to use when launched on compute nodes. If you have pre-built binaries that exist in another location, an additional entry in the component description section may be a convenient place to put it. INPUT_DIR, INPUT_FILES and OUTPUT_FILES specify the location and names of the input and output files, respectively. If a subset of plasma states files is all that is required by the component, they are specified here (STATE_FILES). If the entry is omitted, all of the plasma state files are used. This prevents the full set of files to be copied to and from the component’s work directory on every step, saving time and space. Lastly, SCRIPT is the Python script that contains the component code, specifically the Python class in NAME. Additionally, any component specific values maybe specified here to control logic or set data values that change often.

  4. Time Loop Section

    This may need to be modified for your component or the driver that uses your new component. During testing, a small number of steps is appropriate.

    # Time loop specification (two modes for now) EXPLICIT | REGULAR
    # For MODE = REGULAR, the framework uses the variables START, FINISH, and NSTEP
    # For MODE = EXPLICIT, the framework uses the variable VALUES (space separated
    # list of time values)
    [TIME_LOOP]
        MODE = EXPLICIT
        VALUES = 75.000 75.025 75.050 75.075 75.100 75.125
    

Tips

This section contains some useful tips on testing, debugging and documenting your new component.

  • General:

    • Naming is important. You do not want the name of your component to overlap with another, so make sure it is unique.

    • Be sure to commit all the files and directories that are needed to build and run your component. This means the executables, Makefiles, component script, helper scripts and input files.

  • Testing:

    • To test a new component, first run it as the driver component of a simulation all by itself. This will make sure that the component itself works with the framework.

    • The next step is to have a driver call just your new component to make sure it can be discovered and called by the driver properly.

    • The next step is to determine if the component can exchange global data with another component. To do this run two components in a driver and verify they are exchanging data properly.

    • When testing IPS components and simulations, it may be useful to turn on debugging information in the IPS and the underlying executables.

    • If this is a time stepping simulation, a small number of steps is useful because it will lead to shorter running times, allowing you to submit the job to a debug or other faster turnaround queue.

  • Debugging:

    • Add logging messages (services.info(), services.warning(), etc.) to make sure your component does what you think it does.

    • Remove other components from the simulation to figure out which one or which interaction is causing the problem

    • Take many checkpoints around the problem to narrow in on the problem.

    • Remove concurrency to see if one component is overwriting another’s data.

  • Documentation:

    • Document the component code such that another person can understand how it works. It helps if the structure remains the same as the example component.

    • Write a description of what the component does, the inputs it uses, outputs it produces, and what scenarios and modes it can be used in in the component documentation section.

  • Protected attributes:

    • The following Component attributes are used internally within IPS and are protected so you can not assigned to them:

      • component_id

      • services

      • config

      • start_time

      • method_name

      • args

Writing Drivers

The driver of the simulation manages the control flow and synchronization across components via time stepping or implicit means, thus orchestrating the simulation. There is only one driver per simulation and it is invoked by the framework and is responsible for invoking the components that make up the simulation scenario it implements. It is also responsible for managing data at the simulation level, including checkpoint and restart activities.

Before writing a driver, it is a good idea to have the components already written. Once the components that are to be used are chosen the data coupling and control flow must be addressed.

In order to couple components, the data that must be exchanged between them and the ordering of updates to the plasma state must be determined. Once the data dependencies are identified (which components have to run before the next, and which ones can run at the same time). You can write the body of the driver. Before going through the steps of writing a driver, review the method invocation API and plan which methods to use during the main time loop.

The framework will invoke the methods of the INIT and DRIVER components over the course of the simulation, defining the execution of the run:

  • init_comp.init() - initialization of initialization component

  • init_comp.step() - execution of initialization work

  • init_comp.finalize() - cleanup and confirmation of initialization

  • driver.init() - any initialization work (typically empty)

  • driver.step() - the bulk of the simulation

    • get references to the ports

    • call init on each port

    • get the time loop

    • implement logic of time stepping

    • during each time step:

      • perform pre-step logic that may stage data or determine which components need to run or what parameters are given to each component

      • call step on each port (as appropriate)

      • manage global plasma state at the end of each step

      • checkpoint components (frequency of checkpoints is controlled by framework)

    • call finalize on each component

  • driver.finalize() - any clean up activities (typically empty)

It is recommended that you start with the ips/components/drivers/dbb/generic_driver.py and modify it as needed. You will most likely be changing: how the components are called in the main loop (the generic driver calls each component in sequence), the pre-step logic phase, and what ports are used. The data management and checkpointing calls should remain unchanged as their behavior is controlled in the configuration file.

The process for adding a new driver to the IPS is the same as that for the component. See the appropriate sections above for adding a component.

IPS Services API

The IPS framework contains a set of managers that perform services for the components. A component uses the services API to access them, thus hiding the complexity of the framework implementation. Below are descriptions of the individual function calls grouped by type. To call any of these functions in a component replace ServicesProxy with self.services. The services object is passed to the component upon creation by the framework.

Component Invocation

Component invocation in the IPS means one component is calling another component’s function. This API provides a mechanism to invoke methods on components through the framework. There are blocking and non-blocking versions, where the non-blocking versions require a second function to check the status of the call. Note that the wait_call has an optional argument (block) that changes when and what it returns.

ServicesProxy.call(component_id, method_name, *args, **keywords)

Invoke method method_name on component component_id with optional arguments *args. Will wait until call is finished. Return result from invoking the method.

Parameters
  • component_id (ComponentID) – Component ID of requested component

  • method_name (str) – component method to call, e.g. init or step

Returns

service response message arguments

ServicesProxy.call_nonblocking(component_id, method_name, *args, **keywords)

Invoke method method_name on component component_id with optional arguments *args. Will not wait until finished.

Parameters
  • component_id (ComponentID) – Component ID of requested component

  • method_name (str) – component method to call, e.g. init or step

Returns

call_id

Return type

int

ServicesProxy.wait_call(call_id, block=True)

If block is True, return when the call has completed with the return code from the call. If block is False, raise IncompleteCallException if the call has not completed, and the return value is it has.

Parameters

call_id (int) – call ID

Returns

service response message arguments

ServicesProxy.wait_call_list(call_id_list, block=True)

Check the status of each of the call in call_id_list. If block is True, return when all calls are finished. If block is False, raise IncompleteCallException if any of the calls have not completed, otherwise return. The return value is a dictionary of call_ids and return values.

Parameters

call_id_list (list of int) – list of call ID’s

Returns

dict of call_id and return value

Return type

dict

Task Launch

The task launch interface allows components to launch and manage the execution of (parallel) executables. Similar to the component invocation interface, the behavior of launch_task() and the wait_task() variants are controlled using the block keyword argument and different interfaces to wait_task.

The task_ppn and task_cpp options all greater control over how commands are made. task_ppn will limit the number of task per node, task_ccp will limit the number of cores assigned to each process, this is only used when MPIRUN=srun, if task_cpp is not set it will be calculated automatically.

Slurm examples

The following examples show the behavior if you are running on a Cori with 32 cores per node.

Using the check-mpi.gnu.cori binary provided on Cori with nproc=8 and settings the correct OMP environment variables with omp=True without specifying other options :

self.services.launch_task(8, cwd, "check-mpi.gnu.cori", omp=True)

the srun command created will be srun -N 1 -n 8 -c 4 --threads-per-core=1 --cpu-bind=cores check-mpi.gnu.cori along with settings the environment variables for OpenMP OMP_PLACES=threads OMP_PROC_BIND=spread OMP_NUM_THREADS=4. The resulting core affinity is

Hello from rank 0, on nid00025. (core affinity = 0-3)
Hello from rank 1, on nid00025. (core affinity = 16-19)
Hello from rank 2, on nid00025. (core affinity = 4-7)
Hello from rank 3, on nid00025. (core affinity = 20-23)
Hello from rank 4, on nid00025. (core affinity = 8-11)
Hello from rank 5, on nid00025. (core affinity = 24-27)
Hello from rank 6, on nid00025. (core affinity = 12-15)
Hello from rank 7, on nid00025. (core affinity = 28-31)

If you also include the option task_ppn=4:

self.services.launch_task(8, cwd, "check-mpi.gnu.cori", task_ppn=4, omp=True)

then the command created will be srun -N 2 -n 8 -c 8 --threads-per-core=1 --cpu-bind=cores check-mpi.gnu.cori along with settings the environment variables for OpenMP OMP_PLACES=threads OMP_PROC_BIND=spread OMP_NUM_THREADS=8. The resulting core affinity is

Hello from rank 0, on nid00025. (core affinity = 0-7)
Hello from rank 1, on nid00025. (core affinity = 16-23)
Hello from rank 2, on nid00025. (core affinity = 8-15)
Hello from rank 3, on nid00025. (core affinity = 24-31)
Hello from rank 4, on nid00026. (core affinity = 0-7)
Hello from rank 5, on nid00026. (core affinity = 16-23)
Hello from rank 6, on nid00026. (core affinity = 8-15)
Hello from rank 7, on nid00026. (core affinity = 24-31)

You can limit the --cpus-per-task of the srun command by setting task_cpp, adding task_cpp=2

self.services.launch_task(8, cwd, "check-mpi.gnu.cori", task_ppn=4, task_cpp=2, omp=True)

will create the command srun -N 2 -n 8 -c 2 --threads-per-core=1 --cpu-bind=cores check-mpi.gnu.cori and set OMP_PLACES=threads OMP_PROC_BIND=spread OMP_NUM_THREADS=2. This will result in under-utilizing the nodes, which may be needed if your task is memory bound. The resulting core affinity is

Hello from rank 0, on nid00025. (core affinity = 0,1)
Hello from rank 1, on nid00025. (core affinity = 16,17)
Hello from rank 2, on nid00025. (core affinity = 2,3)
Hello from rank 3, on nid00025. (core affinity = 18,19)
Hello from rank 4, on nid00026. (core affinity = 0,1)
Hello from rank 5, on nid00026. (core affinity = 16,17)
Hello from rank 6, on nid00026. (core affinity = 2,3)
Hello from rank 7, on nid00026. (core affinity = 18,19)

Using the check-hybrid.gnu.cori binary with the same options:

self.services.launch_task(8, cwd, "check-hybrid.gnu.cori", task_ppn=4, task_cpp=2, omp=True)

the resulting core affinity of the OpenMP threads are:

Hello from rank 0, thread 0, on nid00025. (core affinity = 0)
Hello from rank 0, thread 1, on nid00025. (core affinity = 1)
Hello from rank 1, thread 0, on nid00025. (core affinity = 16)
Hello from rank 1, thread 1, on nid00025. (core affinity = 17)
Hello from rank 2, thread 0, on nid00025. (core affinity = 2)
Hello from rank 2, thread 1, on nid00025. (core affinity = 3)
Hello from rank 3, thread 0, on nid00025. (core affinity = 18)
Hello from rank 3, thread 1, on nid00025. (core affinity = 19)
Hello from rank 4, thread 0, on nid00026. (core affinity = 0)
Hello from rank 4, thread 1, on nid00026. (core affinity = 1)
Hello from rank 5, thread 0, on nid00026. (core affinity = 16)
Hello from rank 5, thread 1, on nid00026. (core affinity = 17)
Hello from rank 6, thread 0, on nid00026. (core affinity = 2)
Hello from rank 6, thread 1, on nid00026. (core affinity = 3)
Hello from rank 7, thread 0, on nid00026. (core affinity = 18)
Hello from rank 7, thread 1, on nid00026. (core affinity = 19)
Slurm with GPUs examples

Note

New in 0.8.0

The launch_task() method has an option task_gpp which allows you to set the number of GPUs per process, used as the --gpus-per-task in the srun command.

IPS will validate the number of GPUs per node requested does not exceed the number specified by the GPUS_PER_NODE parameter in the Platform Configuration File. You need to make sure that the number of GPUs per process times the number of processes per node does not exceed the GPUS_PER_NODE set.

Using the gpus_for_tasks program provided for Perlmutter (which has 4 GPUs per node) to test the behavior, you will see the following:

To launch a task with 1 process and 1 GPU per process (task_gpp) run:

self.services.launch_task(1, cwd, "gpu-per-task", task_gpp=1)

will create the command srun -N 1 -n 1 -c 64 --threads-per-core=1 --cpu-bind=cores --gpus-per-task=1 gpus_for_tasks and the output of will be:

Rank 0 out of 1 processes: I see 1 GPU(s).
0 for rank 0: 0000:03:00.0

To launch 8 processes on 2 nodes (so 4 processes per node) with 1 gpu per process run:

self.services.launch_task(8, cwd, "gpu-per-task", task_ppn=4, task_gpp=1)

will create the command srun -N 2 -n 8 -c 16 --threads-per-core=1 --cpu-bind=cores --gpus-per-task=1 gpus_for_task and the output of will be:

Rank 0 out of 8 processes: I see 1 GPU(s).
0 for rank 0: 0000:03:00.0
Rank 1 out of 8 processes: I see 1 GPU(s).
0 for rank 1: 0000:41:00.0
Rank 2 out of 8 processes: I see 1 GPU(s).
0 for rank 2: 0000:82:00.0
Rank 3 out of 8 processes: I see 1 GPU(s).
0 for rank 3: 0000:C1:00.0
Rank 4 out of 8 processes: I see 1 GPU(s).
0 for rank 4: 0000:03:00.0
Rank 5 out of 8 processes: I see 1 GPU(s).
0 for rank 5: 0000:41:00.0
Rank 6 out of 8 processes: I see 1 GPU(s).
0 for rank 6: 0000:82:00.0
Rank 7 out of 8 processes: I see 1 GPU(s).
0 for rank 7: 0000:C1:00.0

To launch 2 processes on 2 nodes (so 1 processes per node) with 4 gpu per process run:

self.services.launch_task(2, cwd, "gpu-per-task", task_ppn=1, task_gpp=4)

will create the command srun -N 2 -n 2 -c 64 --threads-per-core=1 --cpu-bind=cores --gpus-per-task=4 gpus_per_tasks and the output of will be:

Rank 0 out of 2 processes: I see 4 GPU(s).
0 for rank 0: 0000:03:00.0
1 for rank 0: 0000:41:00.0
2 for rank 0: 0000:82:00.0
3 for rank 0: 0000:C1:00.0
Rank 1 out of 2 processes: I see 4 GPU(s).
0 for rank 1: 0000:03:00.0
1 for rank 1: 0000:41:00.0
2 for rank 1: 0000:82:00.0
3 for rank 1: 0000:C1:00.0

If you try to launch a task with too many GPUs per node, e.g.:

self.services.launch_task(8, cwd, "gpu-per-task", task_gpp=1)

then it will raise an GPUResourceRequestMismatchException.

ServicesProxy.launch_task(nproc, working_dir, binary, *args, **keywords)

Launch binary in working_dir on nproc processes. *args are any arguments to be passed to the binary on the command line. **keywords are any keyword arguments used by the framework to manage how the binary is launched. Keywords may be the following:

  • task_ppn : the processes per node value for this task

  • task_cpp : the cores per process, only used when MPIRUN=srun commands

  • task_gpp : the gpus per process, only used when MPIRUN=srun commands

  • ompIf True the task will be launch with the correct OpenMP environment

    variables set, only used when MPIRUN=srun

  • block : specifies that this task will block (or raise an exception) if not enough resources are available to run immediately. If True, the task will be retried until it runs. If False, an exception is raised indicating that there are not enough resources, but it is possible to eventually run. (default = True)

  • tag : identifier for the portal. May be used to group related tasks.

  • logfile : file name for stdout (and stderr) to be redirected to for this task. By default stderr is redirected to stdout, and stdout is not redirected.

  • whole_nodes : if True, the task will be given exclusive access to any nodes it is assigned. If False, the task may be assigned nodes that other tasks are using or may use.

  • whole_sockets : if True, the task will be given exclusive access to any sockets of nodes it is assigned. If False, the task may be assigned sockets that other tasks are using or may use.

  • launch_cmd_extra_args : extra command arguments added the the MPIRUN command

Return task_id if successful. May raise exceptions related to opening the logfile, being unable to obtain enough resources to launch the task (InsufficientResourcesException), bad task launch request (ResourceRequestMismatchException, BadResourceRequestException) or problems executing the command. These exceptions may be used to retry launching the task as appropriate.

Note

This is a nonblocking function, users must use a version of ServicesProxy.wait_task() to get result.

Parameters
  • nproc (int) – number of processes

  • working_dir (str) – change to this directory before launching task

  • binary (str) – command to execute, can include arguments or can be pass in with *args

Returns

task_id (PID)

Return type

int

ServicesProxy.wait_task(task_id, timeout=- 1, delay=1)

Check the status of task task_id. Return the return value of the task when finished successfully. Raise exceptions if the task is not found, or if there are problems finalizing the task.

Parameters
  • task_id (int) – task ID (PID)

  • timeout (float) – maximum time to wait for task to finish, default -1 (no timeout)

  • delay (float) – time to wait before checking if task has timed-out

Returns

return value of task

ServicesProxy.wait_task_nonblocking(task_id)

Check the status of task task_id. If it has finished, the return value is populated with the actual value, otherwise None is returned. A KeyError exception may be raised if the task is not found.

Parameters

task_id (int) – task ID (PID)

Returns

return value of task if finished else None

ServicesProxy.wait_tasklist(task_id_list, block=True)

Check the status of a list of tasks. If block is True, return a dictionary of return values when all tasks have completed. If block is False, return a dictionary containing entries for each completed task. Note that the dictionary may be empty. Raise KeyError exception if task_id not found.

Parameters
  • task_id_list (list of int) – list of task_id’s (PID’s) to wait until completed

  • block (bool) – if to wait until all task finish

Returns

dict of task_id and return value

Return type

dict

ServicesProxy.kill_task(task_id)

Kill launched task task_id. Return if successful. Raises exceptions if the task or process cannot be found or killed successfully.

Parameters

task_id (int) – task ID

Returns

if successfully killed

Return type

bool

ServicesProxy.kill_all_tasks()

Kill all tasks associated with this component.

The task pool interface is designed for running a group of tasks that are independent of each other and can run concurrently. The services manage the execution of the tasks efficiently for the component. Users must first create an empty task pool, then add tasks to it. The tasks are submitted as a group and checked on as a group. This interface is basically a wrapper around the interface above for convenience.

ServicesProxy.create_task_pool(task_pool_name)

Create an empty pool of tasks with the name task_pool_name. Raise exception if duplicate name.

ServicesProxy.add_task(task_pool_name, task_name, nproc, working_dir, binary, *args, **keywords)

Add task task_name to task pool task_pool_name. Remaining arguments are the same as in ServicesProxy.launch_task().

ServicesProxy.submit_tasks(task_pool_name, block=True, use_dask=False, dask_nodes=1, dask_ppw=None, launch_interval=0.0, use_shifter=False, shifter_args=None, dask_worker_plugin=None, dask_worker_per_gpu=False)

Launch all unfinished tasks in task pool task_pool_name. If block is True, return when all tasks have been launched. If block is False, return when all tasks that can be launched immediately have been launched. Return number of tasks submitted.

Optionally, dask can be used to schedule and run the task pool.

ServicesProxy.get_finished_tasks(task_pool_name)

Return dictionary of finished tasks and return values in task pool task_pool_name. Raise exception if no active or finished tasks.

ServicesProxy.remove_task_pool(task_pool_name)

Kill all running tasks, clean up all finished tasks, and delete task pool.

Miscellaneous

The following services do not fit neatly into any of the other categories, but are important to the execution of the simulation.

ServicesProxy.get_working_dir()

Return the working directory of the calling component.

The structure of the working directory is defined using the configuration parameters CLASS, SUB_CLASS, and NAME of the component configuration section. The structure of the working directory is:

${SIM_ROOT}/work/$CLASS_${SUB_CLASS}_$NAME_<instance_num>
Returns

working directory

Return type

str

ServicesProxy.update_time_stamp(new_time_stamp=- 1)

Update time stamp on portal.

ServicesProxy.send_portal_event(event_type='COMPONENT_EVENT', event_comment='', event_time=None, elapsed_time=None)

Send event to web portal.

Data Management

The data management services are used by the components to manage the data needed and produced by each step, and for the driver to manage the overall simulation data. There are methods for component local, and simulation global files. Fault tolerance services are presented in another section.

Staging of local (non-shared) files:

ServicesProxy.stage_input_files(input_file_list)

Copy component input files to the component working directory (as obtained via a call to ServicesProxy.get_working_dir()). Input files are assumed to be originally located in the directory variable INPUT_DIR in the component configuration section.

File are copied using ipsframework.ipsutil.copyFiles().

Parameters

input_file_list (str or Iterable of str) – input files can space separated string or iterable of strings

ServicesProxy.stage_output_files(timeStamp, file_list, keep_old_files=True, save_plasma_state=True)

Copy associated component output files (from the working directory) to the component simulation results directory. Output files are prefixed with the configuration parameter OUTPUT_PREFIX. The simulation results directory has the format:

${SIM_ROOT}/simulation_results/<timeStamp>/components/$CLASS_${SUB_CLASS}_$NAME_${SEQ_NUM}

Additionally, plasma state files are archived for debugging purposes:

${SIM_ROOT}/history/plasma_state/<file_name>_$CLASS_${SUB_CLASS}_$NAME_<timeStamp>

Copying errors are not fatal (exception raised).

Staging of global (plasma state) files:

ServicesProxy.stage_state(state_files=None)

Copy current state to work directory.

ServicesProxy.update_state(state_files=None)

Copy local (updated) state to global state. If no state files are specified, component configuration specification is used. Raise exceptions upon copy.

ServicesProxy.merge_current_state(partial_state_file, logfile=None, merge_binary=None)

Merge partial plasma state with global state. Partial plasma state contains only the values that the component contributes to the simulation. Raise exceptions on bad merge. Optional logfile will capture stdout from merge. Optional merge_binary specifies path to executable code to do the merge (default value : “update_state”)

Configuration Parameter Access

These methods access information from the simulation configuration file.

ServicesProxy.get_port(port_name)
Parameters

port_name (str) – port name

Returns

Return a reference to the component implementing port port_name.

Return type

ipsframework.componentRegistry.ComponentID

ServicesProxy.get_config_param(param, silent=False)

Return the value of the configuration parameter param. Raise exception if not found and silent is False.

Parameters
  • param (str) – The parameter requested from simulation config

  • silent (bool) – If True and parameter isn’t found then exception is not raised, default False

Returns

dictionary of given parameter from configuration

Return type

dict

ServicesProxy.set_config_param(param, value, target_sim_name=None)

Set configuration parameter param to value. Raise exceptions if the parameter cannot be changed or if there are problems setting the value. This tell the framework to call ipsframework.configurationManager.ConfigurationManager.set_config_parameter() to change the parameter.

Parameters
  • param (str) – The parameter requested from simulation config

  • value – The value to set the parameter

Returns

return value from setting parameter

ServicesProxy.get_time_loop()

Return the list of times as specified in the configuration file.

Returns

list of times

Return type

list of float

Logging

The following logging methods can be used to write logging messages to the simulation log file. It is strongly recommended that these methods are used as opposed to print statements. The logging capability adds a timestamp and identifies the component that generated the message. The syntax for logging is a simple string or formatted string:

self.services.info('beginning step')
self.services.warning('unable to open log file %s for task %d, will use stdout instead',
                      logfile, task_id)

There is no need to include information about the component in the message as the IPS logging interface includes a time stamp and information about what component sent the message:

2011-06-13 14:17:48,118 drivers_ssfoley_branch_test_driver_1 DEBUG    __initialize__(): <branch_testing.branch_test_driver object at 0xb600d0>  branch_testing_hopper@branch_test_driver@1
2011-06-13 14:17:48,125 drivers_ssfoley_branch_test_driver_1 DEBUG    Working directory /scratch/scratchdirs/ssfoley/rm_dev/branch_testing_hopper/work/drivers_ssfoley_branch_test_driver_1 does not exist - will attempt creation
2011-06-13 14:17:48,129 drivers_ssfoley_branch_test_driver_1 DEBUG    Running - CompID =  branch_testing_hopper@branch_test_driver@1
2011-06-13 14:17:48,130 drivers_ssfoley_branch_test_driver_1 DEBUG    _init_event_service(): self.counter = 0 - <branch_testing.branch_test_driver object at 0xb600d0>
2011-06-13 14:17:51,934 drivers_ssfoley_branch_test_driver_1 INFO     ('Received Message ',)
2011-06-13 14:17:51,934 drivers_ssfoley_branch_test_driver_1 DEBUG    Calling method init args = (0,)
2011-06-13 14:17:51,938 drivers_ssfoley_branch_test_driver_1 INFO     ('Received Message ',)
2011-06-13 14:17:51,938 drivers_ssfoley_branch_test_driver_1 DEBUG    Calling method step args = (0,)
2011-06-13 14:17:51,939 drivers_ssfoley_branch_test_driver_1 DEBUG    _invoke_service(): init_task  (48, 'hw', 0, True, True, True)
2011-06-13 14:17:51,939 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|0)
2011-06-13 14:17:51,952 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|0), response = <messages.ServiceResponseMessage object at 0xb60ad0>
2011-06-13 14:17:51,954 drivers_ssfoley_branch_test_driver_1 DEBUG    Launching command : aprun -n 48 -N 24 -L 1087,1084 hw
2011-06-13 14:17:51,961 drivers_ssfoley_branch_test_driver_1 DEBUG    _invoke_service(): getTopic  ('_IPS_MONITOR',)
2011-06-13 14:17:51,962 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|1)
2011-06-13 14:17:51,972 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|1), response = <messages.ServiceResponseMessage object at 0xb60b90>
2011-06-13 14:17:51,972 drivers_ssfoley_branch_test_driver_1 DEBUG    _invoke_service(): sendEvent  ('_IPS_MONITOR', 'PORTAL_EVENT', {'sim_name': 'branch_testing_hopper', 'portal_data': {'comment': 'task_id = 1 , Tag = None , Target = aprun -n 48 -N 24 -L 1087,1084 hw ', 'code': 'drivers_ssfoley_branch_test_driver', 'ok': 'True', 'eventtype': 'IPS_LAUNCH_TASK', 'state': 'Running', 'walltime': '4.72'}})
2011-06-13 14:17:51,973 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|2)
2011-06-13 14:17:51,984 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|2), response = <messages.ServiceResponseMessage object at 0xb60d10>
2011-06-13 14:17:51,987 drivers_ssfoley_branch_test_driver_1 DEBUG    _invoke_service(): getTopic  ('_IPS_MONITOR',)
2011-06-13 14:17:51,988 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|3)
2011-06-13 14:17:52,000 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|3), response = <messages.ServiceResponseMessage object at 0xb60890>
2011-06-13 14:17:52,000 drivers_ssfoley_branch_test_driver_1 DEBUG    _invoke_service(): sendEvent  ('_IPS_MONITOR', 'PORTAL_EVENT', {'sim_name': 'branch_testing_hopper', 'portal_data': {'comment': 'task_id = 1  elapsed time = 0.00 S', 'code': 'drivers_ssfoley_branch_test_driver', 'ok': 'True', 'eventtype': 'IPS_TASK_END', 'state': 'Running', 'walltime': '4.75'}})
2011-06-13 14:17:52,000 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|4)
2011-06-13 14:17:52,012 drivers_ssfoley_branch_test_driver_1 DEBUG    _get_service_response(REQUEST|branch_testing_hopper@branch_test_driver@1|FRAMEWORK@Framework@0|4), response = <messages.ServiceResponseMessage object at 0xb60a90>
2011-06-13 14:17:52,012 drivers_ssfoley_branch_test_driver_1 DEBUG    _invoke_service(): finish_task  (1L, 1)

The table below describes the levels of logging available and when to use each one. These levels are also used to determine what messages are produced in the log file. The default level is WARNING, thus you will see WARNING, ERROR and CRITICAL messages in the log file.

Level

When it’s used

DEBUG

Detailed information, typically of interest only when diagnosing problems.

INFO

Confirmation that things are working as expected.

WARNING

An indication that something unexpected happened, or indicative of some problem in the near future (e.g. “disk space low”). The software is still working as expected.

ERROR

Due to a more serious problem, the software has not been able to perform some function.

CRITICAL

A serious error, indicating that the program itself may be unable to continue running.

For more information about the logging module and how to used it, see Logging Tutorial.

ServicesProxy.log(msg, *args)

Wrapper for ServicesProxy.info().

ServicesProxy.debug(msg, *args)

Produce debugging message in simulation log file. See logging.debug() for usage.

ServicesProxy.info(msg, *args)

Produce informational message in simulation log file. See logging.info() for usage.

ServicesProxy.warning(msg, *args)

Produce warning message in simulation log file. See logging.warning() for usage.

ServicesProxy.error(msg, *args)

Produce error message in simulation log file. See logging.error() for usage.

ServicesProxy.exception(msg, *args)

Produce exception message in simulation log file. See logging.exception() for usage.

ServicesProxy.critical(msg, *args)

Produce critical message in simulation log file. See logging.critical() for usage.

Fault Tolerance

The IPS provides services to checkpoint and restart a coupled simulation by calling the checkpoint and restart methods of each component and certain settings in the configuration file. The driver can call checkpoint_components, which will invoke the checkpoint method on each component associated with the simulation. The component’s checkpoint method uses save_restart_files to save files needed by the component to restart from the same point in the simulation. When the simulation is in restart mode, the restart method of the component is called to initialize the component, instead of the init method. The restart component method uses the get_restart_files method to stage in inputs for continuing the simulation.

ServicesProxy.save_restart_files(timeStamp, file_list)

Copy files needed for component restart to the restart directory:

${SIM_ROOT}/restart/$timestamp/components/$CLASS_${SUB_CLASS}_$NAME

Copying errors are not fatal (exception raised).

ServicesProxy.checkpoint_components(comp_id_list, time_stamp, Force=False, Protect=False)

Selectively checkpoint components in comp_id_list based on the configuration section CHECKPOINT. If Force is True, the checkpoint will be taken even if the conditions for taking the checkpoint are not met. If Protect is True, then the data from the checkpoint is protected from clean up. Force and Protect are optional and default to False.

The CHECKPOINT_MODE option controls determines if the components checkpoint methods are invoked.

Possible MODE options are:

ALL:

Checkpint every time the call is made (equivalent to always setting Force =True)

WALLTIME_REGULAR:

checkpoints are saved upon invocation of the service call checkpoint_components(), when a time interval greater than, or equal to, the value of the configuration parameter WALLTIME_INTERVAL had passed since the last checkpoint. A checkpoint is assumed to have happened (but not actually stored) when the simulation starts. Calls to checkpoint_components() before WALLTIME_INTERVAL seconds have passed since the last successful checkpoint result in a NOOP.

WALLTIME_EXPLICIT:

checkpoints are saved when the simulation wall clock time exceeds one of the (ordered) list of time values (in seconds) specified in the variable WALLTIME_VALUES. Let [t_0, t_1, …, t_n] be the list of wall clock time values specified in the configuration parameter WALLTIME_VALUES. Then checkpoint(T) = True if T >= t_j, for some j in [0,n] and there is no other time T_1, with T > T_1 >= T_j such that checkpoint(T_1) = True. If the test fails, the call results in a NOOP.

PHYSTIME_REGULAR:

checkpoints are saved at regularly spaced “physics time” intervals, specified in the configuration parameter PHYSTIME_INTERVAL. Let PHYSTIME_INTERVAL = PTI, and the physics time stamp argument in the call to checkpoint_components() be pts_i, with i = 0, 1, 2, … Then checkpoint(pts_i) = True if pts_i >= n PTI , for some n in 1, 2, 3, … and pts_i - pts_prev >= PTI, where checkpoint(pts_prev) = True and pts_prev = max (pts_0, pts_1, ..pts_i-1). If the test fails, the call results in a NOOP.

PHYSTIME_EXPLICIT:

checkpoints are saved when the physics time equals or exceeds one of the (ordered) list of physics time values (in seconds) specified in the variable PHYSTIME_VALUES. Let [pt_0, pt_1, …, pt_n] be the list of physics time values specified in the configuration parameter PHYSTIME_VALUES. Then checkpoint(pt) = True if pt >= pt_j, for some j in [0,n] and there is no other physics time pt_k, with pt > pt_k >= pt_j such that checkpoint(pt_k) = True. If the test fails, the call results in a NOOP.

The configuration parameter NUM_CHECKPOINT controls how many checkpoints to keep on disk. Checkpoints are deleted in a FIFO manner, based on their creation time. Possible values of NUM_CHECKPOINT are:

  • NUM_CHECKPOINT = n, with n > 0 –> Keep the most recent n checkpoints

  • NUM_CHECKPOINT = 0 –> No checkpoints are made/kept (except when Force = True)

  • NUM_CHECKPOINT < 0 –> Keep ALL checkpoints

Checkpoints are saved in the directory ${SIM_ROOT}/restart

ServicesProxy.get_restart_files(restart_root, timeStamp, file_list)

Copy files needed for component restart from the restart directory:

<restart_root>/restart/<timeStamp>/components/$CLASS_${SUB_CLASS}_$NAME_${SEQ_NUM}

to the component’s work directory.

Copying errors are not fatal (exception raised).

Event Service

The event service interface is used to implement the web portal connection, as well as for components to communicate asynchronously.

ServicesProxy.publish(topicName, eventName, eventBody)

Publish event consisting of eventName and eventBody to topic topicName to the IPS event service.

ServicesProxy.subscribe(topicName, callback)

Subscribe to topic topicName on the IPS event service and register callback as the method to be invoked when an event is published to that topic.

ServicesProxy.unsubscribe(topicName)

Remove subscription to topic topicName.

ServicesProxy.process_events()

Poll for events on subscribed topics.