Presentation

The Inria ParaSuite package consists in a set of benchmark applications and tools to build and to execute them. These applications are issued from research groups of the Inria. Each one uses some parallel technologies in its computation.

The package is divided as follow :

ParaSuite
├─── apps/
├─── bin/
├─── configs/
├─── datasets/
|    ├─── inputs
|    └─── references
├─── doc/
├─── logs/
├─── parasuite_env.csh
└─── parasuite_env.sh

The apps folder contains the installed benchmarks. The tools provided with the ParaSuite package and all its dependencies are stored in the bin folder. The log directory is the default storage for logfiles generated by ParaSuite. The folder named configs is used to store custom configurations (more details in the sections below). The datasets folder contains inputs data and references outputs for each applications. This folder represents the main part of the suite size. According to that, it is possible to download the suite without dataset. Dataset can be downloaded separately.

Requirement

Before using the Inria ParaSuite benchmarks, you must set up some environnement variables. This can be achieved by executing the following command from the root directory of the suite:

$ source parasuite_env.sh

The ParaSuite launcher tool requires also to have a version of python installed (get python).

To test if your environnement is correctly set, execute the following command:

$ parasuite-launcher --version

This should give you the following output:

parasuite-launcher version XXX.YYY.ZZZ

ParaSuite launcher usage

The Inria ParaSuite provides a tool to help you to build the different benchmarks, run them and check the results they produce. This tool is invoked thanks to the command below:

$ parasuite-launcher

The following sections detail how to use this tool.

Quickstart

All the options used in this quickstart are detailled in the following sections. To build a valid command you need to stack different options. Some of these options are required. You always need to specify the benchmark(s) to process and at least one step to execute.

Examples

To run the Plast benchmark:

$parasuite-launcher --bench=plast --run

To build all the MPI applications, with verbose output:

$parasuite-launcher --bench=mpi --build --verbosity=debug

To run and test the results of the bocop benchmark with the large dataset:

$parasuite-launcher --bench=bocop --run --check --size=large

Options

Benchmarks

Required

To specify the benchmark to execute, you must use the --bench option. It accepts different arguments. You can specify only one benchmark name like below. In this case only this benchmark will be proceed.

$ parasuite-launcher --bench=plast

But you can also give multiple application names at the same time, each name should be separated by a comma.

$ parasuite-launcher --bench=plast,genfield

These two ways to specify benchmarks are completed by a third one which is quite powerful. It allows you to specify the benchmarks to execute according to their categories. For instance, the benchmarks which use OpenMP or which aim computer clusters. All the benchmarks and their categories are described in the section benchmarks overview

$ parasuite-launcher --bench=openmp,cluster

Of course, you can execute all the benchmarks by specifying it like in the following command.

$ parasuite-launcher --bench=all

Note: Even if an application matches multiple specified criteria, it would be executed only once.

Steps

Required

The full execution process is divided in 5 distinct phases:

veryclean: delete all the content of the build directory.
clean: delete the files produced by the build stage.
build: compile the application and its dependencies
run: run the application (the application must be builded before)
check: check the results produced by the run stage.

For instance the command below performs the build and the execution of a benchmark.

$ parasuite-launcher --build --run

A sixth step can be invoked. This step called all invokes all the stages of the process.

$ parasuite-launcher --all

Datasets

The Inria ParaSuite package provides some datasets for each benchmark. There are different categories of datasets according to their size and their execution time. The categories are divided as follow:

Name	Execution time	Usage
tiny (default)	< 1 seconde	functional validation
small	< 1 minute	simulation
large	<10 minute	test
huge	~1 hour	real case

To specify the dataset to use, you must set the option --size.

$ parasuite-launcher --size=small

Configuration

The launcher allows you to load custom configuration files. These files can specified compilers to use or the compilation flags for instance. The usage of them is detailed in the section Configurations below.

The configuration file to load can be specified by the option --config.

$ parasuite-launcher --config=/path/to/my/config

Others options

Logging

The launcher produces log files by default. They contain the output of the different executed steps. If no directory is specified the logs are stored in parasuite_directory/logs/. For each benchmarks, two files are produced, one for standart output, one for the errors.

bench-name_timestamp.std.log
bench-name_timestamp.err.log

You can specify the directory where the log files will be stored thanks to the option --logdir.

$ parasuite-launcher --logdir=/my/dir

It is also possible to disable logging with the option --no-logging. In this case, no file are created.

Verbosity

The launcher offers different levels of verbosity. They are more or less verbose and can help you to fix a problem if something went wrong. The levels are the following (ordered from less to most verbose):

Level	Effect
None	Does not display any message, not even error.
Error	Displays only critical errors
Warning	Displays all kind of errors.
Info (default)	Displays every errors and messages to inform user about the proceed steps.
Debug	Should be use only for debug purpose.

Before each log output the level is specified. The command below set this option:

$ parasuite-launcher --verbosity=(none|error|warning|info|debug)

A shorter version of the command:

$ parasuite-launcher -v (none|error|warning|info|debug)

By default the output in the console is colorized. This simplifies the reading but can be problematical if you want to output the stream in a file. To disable the color use the option below:

$ parasuite-launcher --no-color

List

The launcher can list all the installed and ready to use applications. For each application the categories at which it belong are displayed.

$ parasuite-launcher --list

Version

You can get the current version of the launcher thanks to the following command:

$ parasuite-launcher --version

Configurations

The configuration file system allows to simplify the customizsation of the build process. You can specify plenty of options like compiler flags or compiler to use.

The system has been designed to be the most effective possible. Thanks to a system of rules you can specify options for one application or for a category of them and even for all of them. This mechanism prevents to have erroneous configuration files full of copy-paste. The configuration file is written in Json for maintainability.

The code below is an example of configuration file.

{
  "applications" :[
    { "match" : "all",
      "options": {
        "C_COMPILER": "gcc"
      }
    },
    { "match" : "plast",
      "options": {
        "C_COMPILER": "icc",
        "C_FLAGS": "-g"
      }
    }
  ]
}

This example configuration is composed of two main rules, one for every applications "match" : "*" and another only for Plast "match" : "plast". Each of these rules defines a set of options defined in the "options" field. In this field you can specify the compiler to use for instance. In this case, the compiler gcc will be used for every application but Plast which overides this definition by the compiler icc.

You can also use categories name (see section benchmarks -categories for further details) in the field "match". This can be used to specify a different compiler for all the benchmarks using MPI for instance.

{
  "applications" :[
    { "match" : "all",
      "options": {
        "C_COMPILER": "gcc"
      }
    },
    { "match" : "mpi",
      "options": {
        "C_COMPILER": "mpicc",
      }
    }
  ]
}

Hooks

The hook mechanism allows you to execute custom commands for each stage of the build process. There are two kind of hooks prehook and posthook. The prehook is executed before a step and the posthook after.

The hooks must be defined in the configuration file, see the section Configurations above. The following code snippet shows how to declare hooks.

{
  "applications" :[
    { "match" : "all",
      "steps" : [
        {
          "step" : "build",
          "prehook" : "mycommand1",
          "posthook" : "mycommand2"
        }
      ]
    },
    { "match" : "mpi",
      "steps" : [
        {
          "step" : "build",
          "prehook" : "",
          "posthook" : "mycommand2"
        },
        {
          "step" : "run",
          "prehook" : "mycommand3"
        }
      ]
    }
  ]
}

The step field specify the step for which the hooks will be executed. The steps are the one listed in section Steps above. In this example, two commands will be executed (mycommand1 executed before the step and mycommand2 after) at the build step for each application except Plast for which only mycommand2 will executed after. For this application another command mycommand3 will be invoked too after the run step.

Variables

The following sections delail for each application the variable you can specify in the configuration file.

Generic

RUNPREFIX: Specifies the prefix of the run command. For instance to time the application, you could set this variable to time .
{benchname}: This read-only variable contains the name of the benchmark currently processed.

Bocop

CXX_FLAGS: C++ compilation flags to use.
CCOMPILER: C compiler to use. (default: gcc)
NBTHREADS: Number of threads to spawn. (default: 1)

Genfield

CXX_FLAGS: C++ compilation flags to use.
C_COMPILER: C compiler to use. (default: mpicc)
CXX_COMPILER: C++ compiler to use. (default: mpicxx)
NBPROC: Number of proc to use. (default: 1)

Grph

JAVA_COMPILER: Java compiler (default: javac)
JAVA_CFLAGS: Java compilation flags to use.

Hips

NBPROC: Number of proc to use. (default: 1)

OpenNl

NVCC_FLAGS: Specify the options to send to Cuda compiler.

PaStiX

NBPROC: Number of proc to use. (default: 1)

Plast

C_FLAGS: C compilation flags to use.
CCOMPILER: C compiler to use. (default: gcc)
NBTHREADS: Number of threads to spawn. (default: 1)

Selalib

CMAKE_FORTRAN_COMPILER: Fortran compiler. (default: gfortran)
CMAKE_FFLAGS: Fortran compilation flags to use.

Examples

## Instrument benchmarks with Pin

To get info on Pin tool, check the official page.

The following example details how to use Pin to count executed instruction with hardware counter (example inscount1 provided with Pin). To see how to build this example, please refer to Pin documentation.

At this step, we assume the dynamic library inscount1.sohas been built. The nominal use of Pin is the following: /opt/pin/pin /opt/pin/inscount1.so -- application_to_run.

To add prefix to a run command, you can use the RUNPREFIX option. This option should specified in a ParaSuite configuration file. For instance the following snippet of configuration enables the instrumentation for all the benchmarks using openMP.

configs/instrumentation.cfg:

{
  "applications" :[
    { "match" : "openmp",
      "options": {
        "RUNPREFIX": "/opt/pin/pin -t /opt/pin/inscount1.so -- "
      }
    }
  ]
}

Generate output per benchmark

This example shows you how to save measured performances in a file for each benchmarks. The performances measurements are achieved thanks to perf command. In the following configuration file you can see the RUNPREFIX option contains a variable {benchname} which contains the name of the current processed benchmark.

configs/perfs.cfg

{
  "applications" :[
    { "match" : "all",
      "options": {
        "RUNPREFIX": "perf stat -o /tmp/perf_{benchname}.out -- "
      }
    }
  ]
}

This feature can be very useful if it’s coupled to the hook mechanism. The folowing example describes how to gather the cache misses generated by each benchmark and compile them in a single file.

The cache misses measurement is achieved withe the perf tool.

configs/perfs_cache_miss.cfg

{
  "applications" :[
    { "match" : "all",
      "options" : {
        "RUNPREFIX" : "perf stat -o /tmp/perf_{benchname}.out -- "
      },
      "steps" : [
      	"step" : "run",
      	"prehook" : "rm /tmp/perf_{benchname}.out",
      	"posthook" : "./scripts/extract_misses.sh {benchname} /tmp/perf_{benchname}.out >> cachemisses.csv"
      ]
    }
  ]
}

extract_misses.sh

#!/bin/sh
value=`cat $2 | grep r20d1 | awk -F',' '{print $1}'`
echo "$1,$value"

According to this configuration file, each benchmark will run with the perf tool. This tool generates an output file named /tmp/perf_{benchname}.out. After the run process the POSTHOOK command is invoked. This script extracts the data of the perf output and generates the following file in csv format.

cachemisses.csv

pastix,123456
selalib,123456
plast,123456

Content

Presentation

Requirement

ParaSuite launcher usage

Quickstart

Examples

Options

Benchmarks

Steps

Datasets

Configuration

Others options

Logging

Verbosity

List

Version

Configurations

Hooks

Variables

Generic

Bocop

Genfield

Grph

Hips

OpenNl

PaStiX

Plast

Selalib

Examples

Generate output per benchmark