MSO4SC: D2.1 End Users’ Requirements Report

image

Project Acronym MSO4SC

Project Title

Mathematical Modelling, Simulation and Optimization for Societal Challenges with Scientific Computing

Project Number

731063

Instrument

Collaborative Project

Start Date

01/10/2016

Duration

25 months (1+24)

Thematic Priority

H2020-EINFRA-2016-1

Dissemination level: Public

Work Package WP2 User Requirements and Dissemination

Due Date:

M3+1

Submission Date:

31/01/2017

Version:

06

Status

Final for submission

Author(s):

Z. Horváth (SZE), V. Mehrmann (MATHEON);

A. Steinbrecher (MATHEON), C. Fernandez (CESGA)

Reviewer(s)

  1. Brodtkorb (SINTEF)

image

The MSO4SC Project is funded by the European Commission through the H2020 Programme under Grant Agreement 731063

Version History

Version

Date

Comments, Changes, Status

Authors, contributors, reviewers

01

24/01/2017

Initial version

V. Mehrmann (MATHEON);

A. Steinbrecher (MATHEON);

02

30/01/2017

New initial version

Z. Horváth (SZE)

03

30/01/2017

Revised questionnaires report

C. Fernández (CESGA)

04

31/01/2017

Finalization

Z. Horváth (SZE)

05

31/01/2017

Format updates for submission /quality check

J. Wells (ATOS)

06

31/07/2017

Conclusions and next steps

Z. Horváth (SZE)

=

List of tables

Executive Summary

This deliverable provides the requirement analysis of MSO4SC carried out in the first 3 months of the project. By now the requirement gathering is done mainly internally based on the requirement repository of the components, which are of at least TRL 6 and internal reviewing. The requirement repository of MSO4SC will be continuously updated during the project. Beside the description of the functional and non-functional requirements of the MSO4SC e-Infrastructure, the requirement repository provides the high level definition of the pilots to be implemented in WP5 (End-user Applications Development), according to the consortium and to the end user community’s collaboration.

1. Introduction

Definition of user requirements is the starting point of each project. This is particularly important for complex e-infrastructure development projects like the present MSO4SC project.

This deliverable provides the requirement analysis of MSO4SC carried out in the first 3 months of the project. The requirement gathering is done mainly internally so far and will be continuously updated during the first 10 months of the project.

1.1. Purpose

The objective of this deliverable is to provide all partners with a common requirement gathering procedure and summarize the requirements defined so far. In addition, it describes the pilots to be implemented in WP5.

1.2. Glossary of Acronyms

This deliverable includes Acronyms of terms used within the document collected in the Glossary as follows.

Acronym Definition

D

Deliverable

HPC

High Performance Computing

MADF

Math Application Development Framework

MSO

Mathematical Modelling, Simulation and Optimization

T

Task

TRL

Technology Readiness Level

WP

Work Package

Table 1. Acronyms

2. MSO4SC Initial vision – overview and aims

The main goals of the project are the design, development, prototyping and pre-production testing of an HPC oriented cloud infrastructure for mathematical developing frameworks (MADFs) and applications based on these MADFs as services. The latter ones will serve solutions to societal challenges directly while the MADFs will ease the prototyping of other solutions. The main technology for these services is the mathematical modelling, simulation and optimization (MSO) technology.

All the services to be provided by the MSO4SC infrastructure, as standalone services, are at technology readiness level (TRL) 6 at the starting of the project, which means that all of them are already validated and demonstrated in industrially relevant environment. These components have an established user community and a well discovered stakeholder group.

The considered MADF components are Feel++ [2], FEniCS-HPC [3] and OPM [4], the end user applications are Eye2Brain [5], HifiMagnets [6], 3DAirQualityPrediction (the HPC component [7] and the complete framework [8]), OPM Flow [9] and ZIBAffinity [10]. All of them use high performance computing (HPC) resources for actual relevant applications so one of the main challenges in the development of the cloud based infrastructure is the performance of the MSO4SC services. Further key points are, based on the same reasons, the efficient data flow from the application domain to the infrastructure and the efficient post-processing including visualization of the results. An important constraint of the project imposed by the H2020 Work Programme is to bring all services to at least TRL 8 which needs the completion of a first of a kind commercial infrastructure.

Thus the most important functional properties of the infrastructure are as follows

  • high performance of the applications,

  • efficient data flow from the application domain to the e-infrastructure,

  • fast post-processing including visualization.

The main non-functional property of the MSO4SC infrastructure is the usability of services with one-click deployment from the marketplace; this is particularly important for non-professional users like authorities applying an end-user application from the MSO4SC for certain addressed societal challenges.

3. MSO4SC requirement gathering method

Requirements of MSO4SC are derived from the functional properties of the envisioned infrastructure. The most important ones are written in Section 2. It is clear that functional requirements on performance, data movement and visualization play a key role in the project.

Requirements are gathered together in various ways. The initial interviewing process was led by CESGA and ATOS regarding the infrastructure requests of the MADF and end-user application developers of MSO4SC, see Annex1 and Annex 2.

This activity has a tight relation to WP5 (end-user applications development) namely the pilots to be considered in WP5 will be defined as requirements, too.

3.1. Methodology

The requirements consist of internal and external requirements owned by consortium members and other stakeholders, respectively. The methods for collecting them will be specific to the type of ownership.

For the internal requirements definition, the consortium will re-use requirements from the requirements repository of each component [2-10] after analysing them. The definition of the most suitable use cases to assess and demonstrate the project results is done by using the template table as follows.

# Id ID Name Name for the requirement

Priority

Low/Medium/High

Req. Type

Functional/Non-functional

Description

Definition of the requirement. Describe what it is about.

Purpose

Reason to include the requirement. Justify why the requirement should be taken into account.

Actors

Actors involved in the requirement, taking into account stakeholders related to it.

Requester

List of pilots or entities which proposed or are related to the requirement and that can validate the requirement.

Validation scenario

Determine some validation criteria which would check that the requirement is fulfilled (i.e. small remark about the testing that should be done).

Related WPs

Which WPs we expect to work on this requirement

Components

Include some mapping with the parts of the high level architecture which are affected by this requirement

Relationships

List here those requirements related to this one

Table 2. Requirement table template

External requirements will be gathered in the next phase of the task T2.1. This will apply close interaction with the potential end-user communities, such as those related to EU-MATHS-IN with organizing events where the project partners invite end users to provide their desired features. In addition, key relevant system requirements are identified from the e-Infrastructure developers, who are expected to provide ideas which will improve the e-Infrastructure in any of its layers.

All these ideas together will confirm the group of requirements which, once prioritized, will be used as input for the e-Infrastructure definition.

3.2. MSO4SC stakeholders

The stakeholders group of the MSO4SC infrastructure contain the user groups of the existing MADF and end-user applications. The group of stakeholders will be expanded during the project life time continuously. In the table below those stakeholders are listed who contribute in pilots for validation.

Stakeholder BCAM-1

Tecnalia Research & Innovation: http://tecnalia.com/

BCAM-1

Nautilus Floating solutions: http://www.nautilusfs.com/en/

BCAM-1

BCAM: http://www.bcamath.org

UNISTRA-\{1,2,3}

Cemosis : www.cemosis.fr

UNISTRA-\{1,2}

LNCMI : lncmi.cnrs.fr/

UNISTRA-\{1,3}

Glick Institute : glick.medicine.iu.edu/

UNISTRA-\{1,2}

EMFL : www.emfl.eu/

ZIB-1

Harald Mückter, LMU-WSI (Ludwig-Maximilians-Universität München, Walther-Straub-Institut für Pharmakologie und Toxikologie), mueckter@lrz.uni-muenchen.de

ZIB-1

Christian Piechotta, BAM (Federal Institute for Materials Research and Testing) christian.piechotta@bam.de

ZIB-1

Karl Skriner, Charité Berlin, karl.skriner@charite.de

SZE-1

Hungarian Meteorolgy Service. www.omsz.hu

SZE-2

Table 3. Stakeholders of the requirements

3.3. Continuous updating

This document contains the MSO4SC requirements in an initial phase. By project month 10 the consortium will provide further requirements according to the results of the requirement gathering process written in Section 3.1 above. These all will be collected into a requirement repository which will be updated continuously and used by T2.2. (MSO4SC e-Infrastructure definition) and WP5 (End-user Applications Development).

4. MSO4SC requirements

In this section, we describe the requirements as the result of the gathering method shown in the previous section.

4.1. Requirements of the cloud component of the infrastructure

*# Id* Cloud-1 *Name* Application support

Priority

Req. Type

Functional

Description

The infrastructure should provide support to other MADFs and applications not described in the project

Purpose

For the sustainability of the project the infrastructure should have the availability to provide support to other user communities

Actors

(Please include other potential communities with different software requirements)

Requester

???

Validation scenario

Related WPs

WP3, WP4, WP5

Components

NA

Relationships

NA

Table 4. Requirement table Cloud-1 Application support

*# Id* Cloud-2 *Name* Infrastructures support

Priority

High

Req. Type

Functional

Description

The software developed to manage and orchestrate the infrastructure should be portable and capable to manage other infrastructures, not only the ones and the beginning of the project (CESGA & ATOS)

Purpose

Some use cases are very restricted in terms of data confidentiality, so some parts of the software have to run in local clusters of the users. This should be supported

Actors

SINTEF use case

Requester

SINTEF

Validation scenario

The pilot applications SINTEF

Related WPs

WP3, WP4, WP5

Components

NA

Relationships

NA

Table 5. Requirement table Cloud-2 Infrastructures support

*# Id* Cloud-3 *Name* Data management

Priority

High

Req. Type

Functional

Description

The software developed to manage and orchestrate the infrastructure should be able to move data among different infrastructures

Purpose

This functionality is need to provide the data for the simulations and the output results in the different places where the simulations might take place

Actors

3DAirqualityprediction use case

Requester

SZE

Validation scenario

3DAirqualityprediction use case

Related WPs

WP3, WP4, WP5

Components

NA

Relationships

NA

Table 6. Requirement table Cloud-3 Data management

*# Id* Cloud-4 *Name* Licenses and software restrictions

Priority

High

Req. Type

Functional

Description

The software developed to manage and orchestrate the infrastructure should support proprietary software and the restrictions that it might impose.

Purpose

Some use cases make use of proprietary software. This software is not installed in all the clusters because of license restrictions so the orchestration must take this in account when deploying and running the simulations

Actors

3DAirqualityprediction use case

Requester

SZE

Validation scenario

3DAirqualityprediction use case

Related WPs

WP3, WP4, WP5

Components

NA

Relationships

NA

Table 7. Requirement table Cloud-4 Licenses and software restrictions

*# Id* Cloud-5 *Name* MPI

Priority

High

Req. Type

Functional

Description

The software developed to manage and orchestrate the infrastructure should support the execution of MPI applications.

Purpose

Many use cases make use MPI to run HPC-parallel applications to reduce the execution time.

Actors

ZIBAffinity users, ZIB

Requester

ZIB

Validation scenario

The pilot applications ZIBAffinity and others

Related WPs

WP3, WP4, WP5

Components

NA

Relationships

NA

Table 8. Requirement table Cloud-5 MPI

*# Id* Cloud-6 *Name* Visualization

Priority

High

Req. Type

Functional

Description

The infrastructure should support remote visualization capabilities.

Purpose

Many use cases require visualization software in different parts of the simulation: pre-processing and post-processing, mainly. The infrastructure should support an integrated use of this software as part of the services to the users

Actors

ZIBAffinity users, ZIB

Requester

ZIB

Validation scenario

ZIBAffinity users

Related WPs

WP3, WP4, WP5

Components

NA

Relationships

NA

Table 9. Requirement table Cloud-6 Visualization

*# Id* Cloud-7 *Name* Simulation monitoring

Priority

High

Req. Type

Functional

Description

The users request a way to know of the simulation is progressing and even to know some of the intermediate results as they are being obtained during the execution and before the final results

Purpose

In some simulations it is very useful to know how the simulation is progressing to determine if there has been a mistake in the initial parameters or they should be changed, before finishing the simulations. This will optimize the use of resources because in some cases the simulation won´t have to finish

Actors

BCAM use case and other use cases

Requester

BCAM

Validation scenario

Nautilus simulation

Related WPs

WP3, WP4, WP5

Components

NA

Relationships

NA

Table 10. Requirement table Cloud-7 Simulation monitoring

4.2. MADF requirements

*# Id* UNISTRA-1 *Name* Feel++ installation

Priority

High

Req. Type

Functional

Description

Feel++ is a mathematical framework to solve PDE used for multiphysics modeling

Purpose

We develop the simulation framework required by the pilot testcases eye2brain and hifimagnet

Actors

Feel++ developers and users - UNISTRA

Requester

MADF Feel++ and pilots Eye2brain & Hifimagnet

Validation scenario

The pilot applications Eye2brain and Hifimagnet

Related WPs

WP3, WP4, WP5

Components

NA

Relationships

UNISTRA-1, UNISTRA-2

Table 11. Requirement table UNISTRA-1 Feel++ installation

# Id KTH-1 Name FEniCS-HPC installation

Priority

High

Req. Type

Functional

Description

Installation and optimization of FEniCS-HPC.

Purpose

We develop a 3D flow solver based on FEniCS-HPC for the pilot case 3DAirQualityPrediction.

Actors

FEniCS users.- BCAM & KTH

Requester

MADF FEniCS and pilot 3DAirQualityPrediction.

Validation scenario

The pilot 3DAirQualityPrediction.

Related WPs

WP4, WP5

Components

NA

Relationships

NA

Table 12. Requirement table KTH-1 FEniCS-HPC installation

# Id SINTEF-0 Name OPM installation

Priority

High

Req. Type

Functional

Description

Installation and optimization of OPM.

Purpose

The OPM software includes the components required to run the pilot case OPM Flow, which must be compiled and installed on the target architectures.

Actors

SINTEF and CESGA

Requester

MADF OPM and pilot OPM Flow.

Validation scenario

The pilot OPM Flow.

Related WPs

WP4, WP5

Components

NA

Relationships

NA

Table 13. Requirement table SINTEF-0 FEniCS-OPM installation

4.3. Requirements of the end-user applications

*# Id* UNISTRA-2 *Name* Feel++ HIFIMAGNET

Priority

High

Req. Type

Functional

Description

HIFIMAGNET pilot application for high field magnet modelling and simulation.

Purpose

We develop the simulation framework required by this pilot testcase

Actors

Feel++ developers and users – UNISTRA - LNCMI - EMFL

Requester

LNCMI - EMFL

Validation scenario

The pilot application HIFIMAGNET will be used as a benchmark for this requirement

Related WPs

WP3, WP4, WP5

Components

UNISTRA-1

Relationships

UNISTRA-1 UNISTRA-2

Table 14. Requirement table UNISTRA-2-0 Feel++ HIFIMAGNET

*# Id* UNISTRA-3 *Name* Feel++ EYE2BRAIN

Priority

High

Req. Type

Functional

Description

EYE2BRAIN pilot application for modelling and simulation of flows in the eye and the brain.

Purpose

We develop the simulation framework required by the pilot testcase

Actors

Feel++ developers and users – UNISTRA – GLICK INSTITUTE

Requester

UNISTRA – GLICK INSTITUTE

Validation scenario

The pilot application EYE2BRAIN will be used as a benchmark for this requirement

Related WPs

WP3, WP4, WP5

Components

UNISTRA-1

Relationships

UNISTRA-1, UNISTRA-2

Table 15. Requirement table UNISTRA-3-0 Feel++ EYE2BRAIN

# Id SINTEF-1 Name OPM Flow must run on CESGA

Priority

High

Req. Type

Functional

Description

OPM Flow is a reservoir simulator which is part of the OPM software suite. OPM Flow must run on the CESGA HPC infrastructure

Purpose

OPM Flow is one of the pilot scenarios described under task 3 of work package 5.

Actors

SINTEF and CESGA

Requester

The OPM Flow pilot scenario is described as part of task 3 in work package 5.

Validation scenario

The pilot of the OPM Flow application will be used as benchmark for this scenario.

Related WPs

Work package 5

Components

NA

Relationships

SINTEF-2, SINTEF-3

Table 16. Requirement table SINTEF-1 OPM Flow must run on CESGA

# Id SINTEF-2 Name Ensemble run of OPM Flow

Priority

High

Req. Type

Functional / Non-functional

Description

OPM Flow is a reservoir simulator which is part of the OPM software suite. OPM Flow must run ensembles of simulations on the CESGA HPC infrastructure.

Purpose

Ensemble runs of OPM Flow is a natural part one of the pilot scenarios described under task 3 of work package 5.

Actors

SINTEF and CESGA

Requester

The OPM Flow pilot scenario is described as part of task 3 in work package 5.

Validation scenario

The pilot of the OPM Flow application will be used as benchmark for this scenario. It will be natural to assume that the CPU time spent is close to a linear function of the number of ensemble members.

Related WPs

Work package 5

Components

NA

Relationships

SINTEF-1

Table 17. Requirement table SINTEF-2 Ensemble run of OPM Flow

# Id SINTEF-3 Name Parallel OPM Flow

Priority

High

Req. Type

Functional / Non-functional

Description

OPM Flow is a reservoir simulator which is part of the OPM software suite. OPM Flow must run parallel MPI simulations efficiently on the CESGA HPC infrastructure.

Purpose

OPM Flow is one of the pilot scenarios described under task 3 of work package 5.

Actors

SINTEF and CESGA

Requester

The OPM Flow pilot scenario is described as part of task 3 in work package 5.

Validation scenario

The pilot of the OPM Flow application will be used as benchmark for this scenario. It will be natural to assume that the OPM Flow application will scale well for a limited number of nodes (e.g, up-to 8 parallel processes).

Related WPs

Work package 5

Components

NA

Relationships

SINTEF-1

Table 18. Requirement table SINTEF-3 Parallel OPM Flow

*# Id* BCAM-1 *Name* Floating Wind Turbine pilot

Priority

High

Req. Type

Functional

Description

We are simulating the Nautilus platform behaviour offshore using FeniCS-HPC. The Nautilus platform is a floating platform supporting a wind turbine for off-shore marine energy production. The goal is to evaluate the stability of the platform.

Purpose

The meshes that we are using for the Nautilus simulations have the following properties:

  • Global number of vertices: 2297827

  • Global number of cells: 13343443

  • ca. 100000 time steps

The computational resources allocated at CESGA for FEniCS-HPC shall support those numbers.

Actors

FEniCS users.- BCAM & KTH

Requester

Nautilus platform pilot case for marine energy environment (WP5)

Validation scenario

The pilot of the Nautilus platform simulation will be used as benchmark for this requirement

Related WPs

WP5

Components

NA

Relationships

NA

Table 19. Requirement table BCAM-1 Floating Wind Turbine pilot

*# Id* BCAM-2 *Name* Postprocessing and visualization of simulation

Priority

High

Req. Type

Functional

Description

Postpossessing facilities and visualization (e.g. Paravieweb) to check the results of the simulation once it is finished.

Purpose

The analysis of the simulation progress is needed in order to check if the simulation is going in the right direction, and to give quick preliminary results. In this sense, remote checking through website will save efforts and resources.

Actors

FEniCS users.- BCAM & KTH

Requester

Nautilus platform pilot case for marine energy environment (WP5)

Validation scenario

The pilot of the Nautilus platform simulation will be used as benchmark for this requirement

Related WPs

WP5

Components

NA

Relationships

NA

Table 20. Requirement table BCAM-2 Postprocessing and visualization of simulation

*# Id* ZIB-1 *Name* ZIBAffinity development and installation

Prioity

Medium

Req. Type

Functional/Non-functional

Description

Given a (novel) chemical compound, ZIBAffinity is designed to provide life scientists from various fields (medicine, toxicology, etc.) with an estimate of its binding affinity to target molecules selected from a connected database of critical biological targets on the basis of atomistic force field simulations. The major purpose of the software is a risk assessment of anthropogenic substances and novel transformation products detected in the environment.

Purpose

Thermodynamic quantities derived from atomistic molecular dynamics simulations are computationally demanding, highly parallelizable, and demanding in terms of installation, application, and bookkeeping. Life scientists from various fields would benefit from an automated pipeline with a user-friendly interface and access to HPC.

According to the current implementation state, the calculation of each ligand-target binding affinity primarily requires the execution of 61 independent MPI processes apart from a relational database of pre-parameterized target molecules and some pre as well as post-processing with negligible computational demand.

Using 61 computing nodes consisting of 24 cores each, the molecular dynamics-based computation of the binding free energy of one protein-ligand complex takes about 0.75 to 1.5 hours depending on both the simulation box size and CPU performance. These requirements are easily met by CESGA. In combination with the planned e-infrastructure, we are convinced to find a user-friendly solution for our task.

Actors

ZIBAffinity users, ZIB

Requester

The implementation of the entire procedure will be validated using a set of ligand molecules in complex with the human estrogen receptor alpha for which experimentally determined binding affinities are known.

Validation scenario

The results are mainly validated on the basis of predicted binding modes and binding affinities that will be compared to those determined through X-ray crystallographic structures and IC50 values from binding assays, respectively.

Related WPs

WP5

Components

NA

Relationships

NA

Table 21. Requirement table ZIB-1 ZIBAffinity development and installation

# Id SZE-1 Name 3DAirQualityPrediction with statistical data

Priority

High

Req. Type

Functional/Non-functional

Description

Installation and optimization of 3DAirQualityPrediction with data resulting from statistical databases for traffic, emissions and meteorology; the dispersion is simulated via 3DAirQualityPrediction-CFD.

Purpose

We develop a 3DAirQualityPrediction scenario with simulation the dispersion only and run what-if scenarios according to environmental requirements corresponding to statistical data. The time for the analysis should meet the regulatory standards.

Actors

SZE, KTH, CESGA

Requester

3DAirQualityPrediction.

Validation scenario

The pilot-1 scenario for 3DAirQualityPrediction. The scenario consists of data from European and regional data bases corresponding to city of Gyor, see math.sze.hu/en_GB/air-project.

Related WPs

WP5

Components

NA

Relationships

KTH1

Table 22. Requirement table SZE-1 3DAirQualityPrediction with statistical data

# Id SZE-2 Name 3DAirQualityPrediction with full simulation

Priority

High

Req. Type

Functional/Non-functional

Description

Installation and optimization of 3DAirQualityPrediction with simulation in all components. The dispersion is simulated via 3DAirQualityPrediction-CFD.

Purpose

We develop a 3DAirQualityPrediction scenario with simulation the dispersion only and run what-if scenarios according to environmental requirements corresponding to statistical data.

Actors

SZE, KTH, CESGA

Requester

3DAirQualityPrediction.

Validation scenario

The pilot-2 scenario for 3DAirQualityPrediction. The validation measurements correspond to observations for the city of Gyor, see math.sze.hu/en_GB/air-project.

Related WPs

WP5

Components

NA

Relationships

KTH1

Table 23. Requirement table SZE-1 3DAirQualityPrediction with full simulation

5. Conclusions, next steps

In this report the MSO4SC internal requirement gathering process and its results are published. The gathering process is based on the existing requirement repositories of the initial services, which are least of TRL 6. In addition, using the Volere methodology and an internal interviewing protocol, a requirement table template was set up. Using this methodology we concluded the MSO4SC initial requirement repository consisting of 21 tables are composed and published.

In the next period the MSO4SC requirement repository will be continuously extended. The gathering will be focusing on external stakeholders and business properties of the infrastructure.

References

  1. MSO4SC Description of Work (DoA). Annex to the EC Contract.

  2. Feel++. www.feelpp.org

  3. FEniCS-HPC. fenicsproject.org/ , bitbucket.org/fenics-hpc/

  4. OPM. opm-project.org/

  5. Eye2Brain. www.cemosis.fr/projects/eye2brain/

  6. HifiMagnets. www.cemosis.fr/blog/category/hifimagnet/

  7. 3DAirQualityPrediction-HPC. www.adaptivesimulations.com/

  8. 3DAirQualityPrediction. math.sze.hu/en_GB/air-project

  9. OPM Flow. h0ttp://opm-project.org/?page_id=19

  10. ZIBAffinity. www.zib.de/affinity

  11. Suzanne Robertson, James Robertson; “Mastering the Requirements Process: Getting Requirements Right”; Addison-Wesley Professional, 2012.

ANNEX 1: QUESTIONNAIRE FOR THE HPC ORIENTED CLOUD INFRASTRUCTURE

Have you ever used or programmed for an HPC environment?

  • Many times

  • A few times

  • Never

In case you did, what kind of parallelism do you use?

  • Shared memory/OpenMP

  • Distributed memory/MPI

  • Hybrid (distributed + shared)

  • Don´t really know

Which were your main issues when using HPC environments?

  • Lack of knowledge, in general

  • Configuration complexity

  • Developing the right code (issues with deadlocks, etc.)

  • Optimization aspects

  • Write all the required commands for launching my application/jobs

  • No problem at all, I’ve HPC for breakfast everyday

  • I can’t get some available resources there, so please, get me some!

  • Not applicable

Imagine there is a Portal which might make your life easier when dealing with HPC/Cloud infrastructures, what would be your preferences?

  • I want to get rid of all the complicated things, just click a button and have it running

  • I enjoy understanding all the details, hacking the configuration options as much as possible

  • I want to have many services available so I can select the tools and integrate them in my workflow

  • Whatever you do will be fine for me, but give me something, please!

Have you ever tried to move (some of) your application code from HPC to Cloud?

  • No, I’m not so crazy (yet)

  • Yes, and I succeeded, getting acceptable performance

  • Yes, and I succeeded, but performance was pretty bad (or not as expected)

  • Yes, I tried and I failed, but I learnt a lot and my honor is still there

Do you need interactive communication with your simulation?

  • Yes, to change parameters while running

  • Yes, but only to know if everything is going on as expected

  • No, let’s let it alone

Do you have specific requirements for visualization?

  • I don’t do graphical visualization, just some text files are enough

  • I’m not doing complex visualization but I would like to, although I don’t know how

  • I do complex visualization and my favorite way to do it is _

Do you need support for workflows?

  • Yes. I’m not using workflows now, but I have good ideas about that

  • Yes, I have really nice workflows already implemented

  • No, I’m pretty sure that I’m fine with some calls managed by me

  • I don’t know. Can you do that?

What are your requirements in terms of computational resources?

  • Up to 24 cores per simulation

  • Up to 128 cores

  • Up to 1024 cores

  • Give me all the available cores in the world and I will use them

Have you ever used any of these infrastructures?

  • PRACE

  • EUDAT

  • EGI

  • Other distributed infrastructure:

  • I don’t know any of these names and/or what ‘distributed infrastructure’ means

Please, leave here any other comment you may have:

THANKS FOR YOUR ANSWERS!

ANNEX 2: HPC QUESTIONNAIRES REPORT

Mathematical Modelling, Simulation and Optimization for Societal Challenges with Scientific Computing

MSO4SC

Report on the

Questionnaires for HPC oriented cloud infrastructure

Project

MSO4SC

Document Responsible

Carlos Fernández (CESGA)

Version

1.0

Table of Contents

Introduction

The major objective of the MSO4SC project is to construct an e-infrastructure that provides, in a user-driven, integrative way, tailored access to the necessary services, resources and even tools for the fast prototyping, providing the service producers with the mathematical frameworks as well.

CESGA and ATOS will provide the cloud and HPC infrastructure to the users and MADFs developers. As a first interaction to know more about the cloud and HPC requirements coming from the Mathematicians industry, a questionnaire was designed to get their input and their needs.

Participants

The following is the list of participants who have answered the HPC Questionnaire:

Name Institution

Volker Mehrmann

MATHEON

Christophe Prud’homme

UNISTRA

Johan Jansson

BCAM

Atgeirr Flø Rasmussen, André R. Brodtkorb

SINTEF

Marcus Weber

ZIB

Johan Hoffman

KTH

Review of the answers

In the kickoff meeting of the project in Berlin 10th of November, there was a review of the responses sent to the HPC and Maths questionnaires, triggering some discussions around the usage of the e-Infrastructure and requirements.

The following table we show the answers of each participant, indicating in blue which was the most common answer.

image

Here we present the results of the different questions and some discussion results

  1. Have you ever used or programmed for an HPC environment?

In this first question, the answer from all the partners was “Many times”, reflecting a lot of experience when dealing with HPC infrastructures. Only SINTEF answered “a few times”.

  1. In case you did, what kind of parallelism do you use?

Here the answers present more variation, but in all of them distributed computing (MPI in this case) was present. Some users have already used shared and hybrid parallelism or even CUDA in the case of SINTEF, but during the discussion it was raised that they can live without this feature in the project. PGAS is also used but not needed now.

  1. Which where your main issues when using HPC environments?

In this question, the most common answer was “Configuration complexity and optimization aspects”. During the discussion it was pointed that this usually means that depending on the supercomputer, the software stack (scientific libraries, compiler version, etc…) might be different and in some cases very limited and without any kind of support. This means that in some situations a lot of effort is needed to get good results in terms of performance depending on the type of system and also that the users have to spend a lot of type understanding these different configurations.

  1. Imagine there is a Portal which might make your life easier when dealing with HPC/cloud infrastructures, what would be your preferences?

In this case 5 out of 6 answered that they want to have many services available. Having a single button to run the whole simulation and get the results easily is also a desired feature.

  1. Have you ever tried to move (some of) your application code from HPC to Cloud?

All except one answered that they never did, and the one who tried obtained very bad performance.

  1. Do you need interactive communication with your simulation?

Many answered that the needed or wanted to change some parameters of the simulation while running, and only two that they don´t need any kind of interaction. In any case this looks like a very desirable feature

  1. Do you have specific requirements for visualization?

There are different visualization solutions that are needed, but the most common is Paraview. Half of the answers don´t do any specific visualization but in some way they would like to explore the possibilities. Apart from Paraview, other visualization tools in use are VMD for molecular systems, Salome, ResInsight and MRST

  1. Do you need support for workflows?

The most common answer is that they are not using workflows but would be interested in using them, and two have already some workflows which would like to improve and implement in the solutions. Some solutions arrived for workflow creation like StarPU. Workflows are important for example to control the execution of different (may) taks, especially if they need to do some statistics of the results

  1. What are the requirements in terms of computational resources?

Here the answers are more different, but in general there is a request for “give me all available” so that depending on the resources the type and performance of the simulations can be adapted. There are some cases of ensemble simulations in which tens or hundreds of independent simulations are submitted and each of the independent tasks require up to 24 cores (which is a common configuration for a SMP/NUMA system nowadays), but aggregating all of them requires over hundreds or thousands of cores.

  1. Have you ever used any of these infrastructures?

PRACE was used but 3 out of the 6 participants and other local of national resources have also been used by all of the participants. Some have also their local computational resources.

Conclusions

According to the results received in the questionnaire, it is clear that the traditional HPC/supercomputer environment is not the optimal solution for this end user community. But also that the Cloud is not a viable solution as it is. A combination of the best of each world would be the ideal solution.

In the next table we propose a first list of the best solutions of each world, which combined would provide an almost ideal environment for the community.

Cloud

HPC

Pros

Flexibility

Interactive

Portability

Easy to deploy

Performance

Reproducibility of the results

Predictibility

Support

Cons

Low performance

Lack of tools

Security / Confidentiability

Lack of flexibility

Waiting time

Can´t interact with the simulation

Apart from resources from Cloud and HPC services, at least three additional services are needed to provide a full solution to the community:

  • A visualization interface not only to display results but also to interact with the simulation and complete pre-processing information (meshing). Should include among others Paraview, VMD and ResInsight.

  • Workflows to represent the different interactions between cloud and HPC services and data movement

  • An interface to run the simulations including all the processes: pre-processing, selection of the resources and post-processing, including the visualization of the results, a follow-up of the simulation to have better knowledge about hot it is running and performing, and a summary of the whole process to identify possible faults and opportunities to improve.

Glossary

Paraview: an open source multi-platform data analysis and visualization application. The data exploration can be done interactively in 3D or programmatically using ParaView`s batch processing capabilities. It was developed to analyze extremely large datasets using distributed memory computing resources

VMD: is a tool to view and analyze the results of molecular dynamics simulations

StarPU: is a task programming library for hybrid architectures. Allow programmers to exploit the computing power of the available CPUs and GPUs, while relieving them from the need to specially adapt their programs to the target machine and processing units

ResInsight: is a visualization package geared towards visualizing reservoir simulation results

MRST: is the MATLAB Reservoir Simulation Toolbox, includes visualization tools