D5.3 Pilot Implementation

image

Project Acronym MSO4SC

Project Title

Mathematical Modelling, Simulation and Optimization for Societal Challenges with Scientific Computing

Project Number

731063

Instrument

Collaborative Project

Start Date

01/10/2016

Duration

25 months (1+24)

Thematic Priority

H2020-EINFRA-2016-1

Dissemination level: Public

Work Package WP5 End-user Applications Development

Due Date:

M14 (PROJECT MONTH)

Submission Date:

22/12/2017

Version:

1.0

Status

Final

Author(s):

Johan Hoffman (KTH); Javier Nieto De Santos (ATOS), Victor Sande(CESGA), Javier Carnero (ATOS), Atgeirr Flø Rasmussen (SINTEF), Johan Jansson (BCAM/KTH), Christophe Prudhomme (UNISTRA), Christophe Trophime (UNISTRA), Vedat Durmaz (ZIB), Zoltán Horváth (SZE)

Reviewer(s)

_F. Javier Nieto (ATOS); Esther Klann (MATHEON-TUB); _

image

The MSO4SC Project is funded by the European Commission through the H2020 Programme under Grant Agreement 731063

Version History

Version

Date

Comments, Changes, Status

Authors, contributors, reviewers

0.1

6/11/2017

Initial version

Johan Hoffman (KTH)

0.2

4/12/2017

Second draft

Johan Hoffman (KTH)

0.3

6/12/2017

Third draft

Johan Hoffman (KTH)

0.4

18/12/2017

Fourth draft

Johan Hoffman (KTH), Christophe Prudhomme (UNISTRA), Christophe Trophime (UNISTRA),

0.5

19/12/2017

Fifth draft

Johan Hoffman (KTH), Christophe Prudhomme (UNISTRA), Christophe Trophime (UNISTRA),

0.6

20/12/2017

Sixth draft

Johan Hoffman (KTH), F. Javier Nieto (ATOS), Victor Sande(CESGA), Javier Carnero (ATOS), Atgeirr Flø Rasmussen (Simula), Johan Jansson (BCAM/KTH), Christophe Prudhomme (UNISTRA), Christophe Trophime (UNISTRA), Vedat Durmaz (ZIB)

0.7

21/12/2017

Seventh draft

Johan Hoffman (KTH), F. Javier Nieto (ATOS), Victor Sande(CESGA), Javier Carnero (ATOS), Atgeirr Flø Rasmussen (Simula), Johan Jansson (BCAM/KTH), Christophe Prudhomme (UNISTRA), Christophe Trophime (UNISTRA), Vedat Durmaz (ZIB), Zoltán Horváth (SZE)

1.0

22/12/2017

Modifications according to review comments

Johan Hoffman (KTH), Christophe Trophime (UNISTRA), Zoltán Horváth (SZE)

Table of Contents

List of figures

List of tables

Executive Summary

This deliverable represents the implementation of the pilot scenarios on the MSO4SC e-infrastructure. For each pilot, the status of the implementation is provided, with the implemented pilot features described connecting to D5.2 [1], and hyperlinks are provided to the software repositories. This report provides a snapshot of the pilot implementation, followed up by a second report in M24.

1. Introduction

1.1. Purpose

This deliverable provides a status report of the implementation of the pilots, including software testing and method validation, how the e-infrastructure tools are used and whether the pilot is ready for deployment or not, connecting to D5.2 [1]. A list of implemented features is provided, and a roadmap for the development is given. This report provides a snapshot of the pilot implementation, followed up by a final report in M24.

The pilots are divided into four groups; the groups of pilots associated to the three MADFs described in D4.2 [2] (FEniCS, Feel++, OPM), and the stand-alone pilots. For a general description of the pilots we refer to D5.1 [5].

In D2.1 [3] requirements gathered from within the MSO4SC consortium are described with respect to purpose, type and priority, and for each requirement a validation scenario is defined. These validation scenarios are based on the pilots developed in WP5, and this deliverable provide a snapshot of the status of the implementation of these pilots. The status of the requirements is evaluated in D5.4 [4] in terms of the validation scenarios, and in WP6 a detailed evaluation will be carried out, involving also external stakeholders, with the purpose to also validate TRL levels of the developed MADF and pilots. The MADF development is carried out in WP4, whereas in WP5 the pilots are developed.

The iterative development process follows the four steps of (i) definition of requirements (D2.1 [3]), (ii) design (D2.2 [6], D3.1 [7], D4.1 [8], D5.1 [5]), (iii) implementation (D4.2 [2] , D5.2 [1]), (iv) evaluation (D5.4 [4]), and then back to a new definition of requirements (D2.5 [9]). The pilot implementation is thus based on the requirements gathered in D2.1 [3] and the design formulated in D5.1 [5]. The evaluation in D5.4 [4] provides a snapshot of the development, and will function as input for the next iteration of the development process.

Figure 1 shows the iterative development process during the project, where the evaluation step functions as input for the next iteration of the development process, in line with the mentioned steps.

Figure : The iterative development process of the e-infrastructure.

The implementation of the pilots is in progress, as certain pilots have reached a high level of maturity whereas others are less mature at this point. For the next evaluation report scheduled for M24 a complete evaluation of requirements is planned.

1.2. Glossary of Acronyms

Acronym Definition

CAD

Computer-Aided Design

CFD

Computational Fluid Dynamics

D

Deliverable

EC

European Commission

FMI

Functional Mock-up Interface

FMU

Functional Mock-Up

HPC

High Performance Computing

IDM

Identity Manager

MADF

Mathematical Development Framework

TOSCA

Topology and Orchestration Specification for Cloud Applications

WP

Work Package

Table . Acronyms

2. MSO4SC e-infrastructure

The MSO4SC e-infrastructure will provide a complete set of services to simplify the usage of mathematical applications and to use them in an efficient way, making usage of HPC and cloud resources in a transparent way. It will provide not only access to codes, applications and hardware infrastructure, but it will make available tools to prepare the simulations, up- and download the input and output datasets and visualize the results, among other.

The complete list of services that will be provided to the user will be:

  1. MSO4SC portal: The main MSO4SC service is the user portal, a web interface publicly accessible that is the main entry point for every MSO4SC user to the platform services.

  2. Authentication & Authorization: This module has two parts, the server and client. The server will be based on the Fiware Lab IDM (Keyrock) while the client is embedded in the MSO4SC user portal.

  3. Marketplace: Entry point to manage the applications available in the platform, as well as purchases and invoices related to them.

  4. Software repository: The Software repository provides an integrated cloud service for the whole development cycle.

  5. Data repository: Manages the datasets available in the platform, being able to create new ones, revise, filter, add information, etc.

  6. Pre-processing: Deal with input data generation, manipulation and visualization. User interaction can be performed in unattended or interactive mode by means of remote visualization tools.

  7. Experiments Tool: From this module, the user is able to start a simulation, get some basic monitoring information, and pause/stop it.

  8. Post-processing: Deal with output data visualization and treatment. User interaction can be performed in unattended or interactive mode by means of remote visualization tools.

  9. Monitoring: Visualization of infrastructure and application metrics.

  10. Accounting service: will provide information about the computational resources used, mainly CPU hours and storage used.

  11. Community Tool: Reference for MSO4SC users to learn about the platform and share their knowledge.

The following table summarizes the usage and integration level of the pilots with the listed services. It is important to highlight that some of the services (such as accounting and community tool) are not fully operational and, therefore, pilots cannot really make use of them. On the other hand, the main objective during the first project iteration was to integrate pilots with the deployment mechanisms, enabling an easy usage of the computing resources.

Pilot 1 2 3 4 5 6 7 8 9 10 11

FloatingWindTurbine

3DAirQualityPred-CFD

HiFiMagnet

Eye2brain

OPMFlow

ZIBAffinity

3DAirQualityPrediction

Table : Status of pilots with respect to MSO4SC e-infrastructure services.

Green signifies that the pilot uses the service in its current state, orange refers to partial use, red no use, and white that the service is not yet in place. A more detailed status is given for each pilot below.

3. Status of pilot implementation

In this section, we describe the status of the implementation of the pilots on the MSO4SC e-infrastructure. We provide links to the software repositories, and we present the status with respect to the MSO4SC e-infrastructure services, pilot specific development and test cases, and we lay out the roadmap for each pilot. For a general description of the pilots and test cases we refer to D5.1 [5], and the associated evaluation report is available as D5.4 [4].

3.1. FEniCS pilots

3.2. FloatingWindTurbine

3.2.1. Software repository

The entire FloatingWindTurbine pilot together with the FEniCS-HPC MADF is open source under the LGPL license. The source can be found at:

Container: Singularity image available on FTII test system

Contact: Johan Jansson (jjan@kth.se)

3.2.2. Status of implementation

Services Description

MSO4SC Portal

  • TOSCA file exists

Authentication & Authorization

  • Default authentication on FTII

Software repository

  • Public repository on BitBucket

Data repository

  • Not available yet on CKAN

  • Girder repository setup in progress

Pre-Processing

  • Pre-processed mesh integrated into Singularity image

  • Automated generation of Singularity images

Experiments Tools

  • Several tests from test suite have been run

  • No monitoring filter defined

Post-Processing

  • Use of Paraview

  • ParaViewWeb not available yet

Table : Status of services integration for Floating Wind Turbine.

Status Description

CI/CO

  • Automatic generation of Singularity image

  • Continuous Integration with FEniCS-HPC "next" branch

Ports

  • Finis Terrae II (FTII) done

  • Build scripts are published on MSO4SC github

Commercial Software / License issue

  • The entire FloatingWindTurbine pilot together with the FEniCS-HPC MADF is open source under the LGPL license.

Test suite

  • Implementation of a test suite of increasing complexity:

    • one-phase submerged part of platform

    • two-phase standard MARIN benchmark

    • a prototype test of two-phase heaving platform with real platform geometry

  • Basic documentation in progress

Preparations of Data for benchmarks (see D5.1 [5])

  • Data for benchmark cases prepared

TOSCA files

  • Basic TOSCA files for main testcases have been created

  • Uploaded in MSO4SC/Resources GitHub repository

Table : Status of pilot development for Floating Wind Turbine.

3.2.3. Pilot test cases

The following features have been implemented for the pilot, listed in the table below. Compare with test cases listed in D5.1 [5].

Test case Description

Test case 1 - one-phase submerged platform

  • CAD geometry defined

  • Mesh already available

  • Adaptive mesh refinement tested

Test case 2 - two-phase MARIN

  • CAD geometry defined

  • Mesh available

  • Experiment data available

Test case 3 - prototype two-phase heaving platform

  • CAD geometry defined

  • Mesh available

  • Experiment data available

Table : Test cases status for Floating Wind Turbine.

3.2.4. Roadmap

The roadmap for the pilot development is summarized in the table below.

Feature Description Deadline

Run test cases with Orchestrator

The test cases will be run automatically by the MSO4SC Orchestrator using the developed TOSCA files.

January 12 2018

Manual

Write manual describing how to run the test cases and interpret results.

January 31 2018

Run test cases via MSO4SC portal

The test case simulations will be run via the MSO4SC portal

January 31 2018

Implement all MSO4SC services for the pilot

All MSO4SC services in Section 2 will be implemented for the pilot

February 28 2018

Table : Roadmap for the Floating Wind Turbine pilot.

3.3. 3DAirQualityPrediction-CFD

3.3.1. Software repository

The entire 3DAirQualityPrediction-CFD pilot together with the FEniCS-HPC MADF is open source under the LGPL license. The source can be found at:

Contact: Johan Jansson (jjan@kth.se)

3.3.2. Status of implementation

Services Description

MSO4SC Portal

  • TOSCA file not yet implemented, but will be very close to the FloatingWindTurbine pilot.

Authentication & Authorization

  • Default authentication on FTII

Software repository

  • Public repository on BitBucket

Data repository

  • Not available yet on CKAN

  • Girder repository setup in progress

Experiments Tools

  • Several tests from test suite have been run

  • No monitoring filter defined

Post-Processing

  • Use of Paraview

  • ParaViewWeb not available yet

Table : Status of services integration for 3DAirQualityPrediction-CFD.

Status Description

CI/CO

  • Continuous Integration with FEniCS-HPC "next" branch

Ports

  • Finis Terrae II (FTII) done

  • Build scripts are published on MSO4SC GitHub

Commercial Software / License issue

  • The entire 3DAirQualityPrediction-CDF pilot together with the FEniCS-HPC MADF is open source under the LGPL license.

Test suite

  • Basic documentation in progress

Preparations of Data for benchmarks (see D5.1 [5])

  • Data for benchmark cases prepared

TOSCA files

  • TOSCA file not yet implemented, but will be very close to the FloatingWindTurbine pilot.

Table : Status of pilot development for 3DAirQualityPrediction-CFD.

3.3.3. Pilot test cases

The following features have been implemented for the pilot, listed in the table below. Compare with test cases listed in D5.1 [5].

Test case Description

Test case 1 – urban canyon

  • Mesh from 3DAirQualityPrediction pilot uploaded – more realistic geometry than planned initially.

  • CFD simulation carried out.

Table : Test cases status for 3DAirQualityPrediction-CFD.

3.3.4. Roadmap

The roadmap for the pilot development is summarized in the table below.

Feature Description Deadline

Run test case with Orchestrator

The test case will be run automatically by the MSO4SC Orchestrator using the developed TOSCA files.

February 28 2018

Manual

Write manual describing how to run the test case and interpret results.

February 28 2018

Run test case via MSO4SC portal

The test case simulations will be run via the MSO4SC portal

March 31 2018

Implement all MSO4SC services for the pilot

All MSO4SC services in Section 2 will be implemented for the pilot

March 31 2018

Table : Roadmap for the 3DAirQualityPrediction-CFD pilot.

3.4. Feel++ pilots

3.5. HIFIMAGNET

3.5.1. Software repository

HiFiMagnet uses some commercial software (namely Nag libraries[1] and MeshGems[2] Mesher) the repository is private so far. The permissions on this repository will be managed as stated in D5.2 [1] Privacy section in the final version.

Note also that the license key needed for the commercial software need also special permission as stated in Cloud-4 specification from D2.5 [9].

Source can be found at:

Container:

Singularity images on girder

Contact: Christophe Trophime (Christophe.trophime@lncmi.cnrs.fr)

3.5.2. Status of implementation

Services Description

MSO4SC Portal

  • Deployed (see demos in EnuMath Leiden)

Authentication & Authorization

  • Deployed with elementary test case

  • Authentication on FTII

Software repository

  • Private repository on github

Data repository

  • Not available yet on CKAN

  • Girder repository setup in progress

Pre-Processing

  • Use of Salome

  • Requires license for MeshGems mesher

  • Manual generation of Docker/Singularity images

Experiments Tools

  • Several tests from testsuite have been run

  • No monitoring filter defined

Post-Processing

  • Use of Paraview

  • ParaViewWeb not available yet

Table : Status of services integration for HiFi Magnet.

Status Description

CI/CO

  • Automatic generation of a docker image. Testsuite partly integrated into the docker image creation

  • Upload to private section of Feelpp Docker cloud organization

  • Integrate github repository within Buildkite system

Ports

  • Finis Terrae II (FTII) done,

  • need to upload script on MSO4SC github

Commercial Software / License issue

  • License have been uploaded on FTII and permissions are managed by FTII sysadmins

Testsuite

  • Implementation of a testsuite based on simple academic case (eg torus of rectangular cross section)

  • Basic documentation in progress (book.hifimagnet website)

Preparations of Data for benchmarks (see D5.1 [5])

  • Data for benchmark cases prepared

  • Data will be stored either in girder / github as they are sensitive

TOSCA files

  • Basic TOSCA files for main testcases have been created

  • Upload in MSO4SC/Resources github repository

  • Tested and presented by GD in Leyden

Table : Status of development for HiFi Magnet.

3.5.3. Pilot test cases

The following features have been implemented for the pilot, listed in the table below. Compare with test cases listed in D5.1 [5].

Test case Description

Test case 1

  • CAD geometry defined

  • Mesh already available

  • Experiment data available

Test case 2

  • CAD geometry defined

  • Mesh not available

  • Experiment data to be extracted from control

Test case 3

  • Initial CAD geometry defined

  • Development of a Shape Optimization procedure based on reduced models

  • Actual machining of optimized geometry not planned

Table : Status of test cases for HiFi Magnet.

3.5.4. Roadmap

The roadmap for the pilot development is summarized in the table below.

Feature Description Deadline

Reorganize HiFiMagnet repository

  • Split into several repositories (code sources, container generation, apps packaging, Data)

  • Manage permissions on each git repository

End of March 2018

Container for Salome

  • Automate generation in progress

  • Integrate into CI/CD system

End of March 2018

Mesh generation

  • Test more complex workflow including CAD/Mesh generation in Orchestrator

End of January 2018

Documentation

  • Improve HiFiMagnet documentation

End of February 2018

Benchmark cases

  • Prepare reference results for benchmark cases

End of May 2018

Table : Roadmap for the HiFi Magnet pilot.

3.6. EYE2BRAIN

3.6.2. Status of implementation

Services Description

MSO4SC portal

  • Deployed with Level 1 and Level 2

Authentication & Authorization

  • Authentication on FTII

Software repository

  • Done in Github

Data repository

  • In progress using girder and CKAN

Pre-processing

  • Done in TOSCA file

Experiments Tool

  • Done

Post-processing

  • Done using paraview; paraviewweb interface

Table 15: Status of services integration for Eye2Brain.

Description

CI/CO

  • Automatic generation of a docker imageTestsuite partly integrated into the docker image creation

  • Upload to private section of Feelpp Docker cloud organization

  • Integrate github repository within Buildkite system

Ports

  • Finis Terrae II (FTII) done,

  • need to upload script on MSO4SC github

Commercial Software / License issue

  • License have been uploaded on FTII and permissions are managed by FTII sysadmins

Test suite

  • Implementation of a test suite based on simple academic case

  • Basic documentation in progress

Preparations of Data for benchmarks (see D5.1 [5])

  • Data for benchmark cases prepared

  • Data will be stored either in girder / github as they are sensitive

TOSCA files

  • TOSCA files for Level 1 and Level 2 have been created

  • Upload in MSO4SC/Resources github repository

Table : Status of development for Eye2Brain.

The following test cases have been implemented for the pilot, listed in the table below. Compare with test cases listed in D5.1 [5].

Test case Description

Level 1

Done.

Level 2

Done.

Level 3

Work in progress

Table : Test cases status for Eye2Brain.

3.6.3. Roadmap

The roadmap for the pilot development is summarized in the table below.

Feature Description Deadline

Test suite

Continue improving the testsuite by adding more tests, verifications and validations

End of March 2018

OpenModelica[3] integration V1

Integrate OpenModelica within Eye2brain using FMU/FMI

End of January 2018

OpenModelica integration V2

Integrate OpenModelica within Eye2brain using FMU/FMI

End of March 2018

Documentation

Improve documentation of eye2brain and its testcase; provide per test-case documentation

End of March 2018

Level 3 V1

Provide Level 3 eye2brain application

End of March 2018

Level 3 V2

Provide Level 3 eye2brain application

End of May 2018

User Interface V1

Develop a specific user interface for medical doctors

End of February 2018

User Interface V2

Develop a specific user interface for medical doctors

End of April 2018

Deployment

Work on deployment in ophthalmic clinic

End of May 2018

TOSCA files V2

Improve genericity of TOSCA files framework, provide flexible inputs

End January 2018

TOSCA files V3

Improve genericity of TOSCA files framework, provide flexible inputs

End of March 2018

Table : Roadmap for the Eye2Brain pilot.

3.7. OPM pilots

3.8. OPM Flow

3.8.1. Software repository

Source repositories:

Container:

Contact: Atgeirr Flø Rasmussen (atgeirr@sintef.no)

3.8.2. Status of implementation

Services Description

MSO4SC Portal

  • TOSCA files available

  • Test case 1 successfully run through orchestrator

Authentication & Authorization

  • Default authentication on FTII

Software repository

  • Using CESGA repository with the whole pilot

Data repository

  • Public repository for Norne

  • Private datasets for other testcases (not integrated yet)

Pre-Processing

  • Done in TOSCA file.

Experiments Tools

  • Test case 1 successfully run with the tool

Post-Processing

  • Using post-processing, but not from the portal

Table : Status of services integration for OPM Flow.

Description

CI/CO

  • Automatic generation of a docker image, testsuite automatically executed by Jenkins

  • (No automatic updating of docker image on Docker Hub yet)

Ports

  • Finis Terrae II (FTII) done.

Commercial Software / License issue

  • No commercial licenses.

Test suite

  • Implementation of a test suite based on simple case

  • Basic documentation in progress

Preparations of Data for benchmarks (see D5.1 [5])

  • Test case 1 data publicly available

  • Test case 2 data cannot be shared, not in data portal

  • Test case 3 data cannot be shared, not in data portal

  • Manual has been created

TOSCA files

  • TOSCA files suitable for test cases 1 and 2 have been created

  • Uploaded to MSO4SC/resources GitHub repository

Table : Status of development for OPM Flow.

3.8.3. Pilot test cases

The test cases have status as shown in the table below. Details are described in D5.1 [5]. Initial trials indicate acceptable low-level scaling for single cases.

Test case Description

Test case 1

Test case 2

  • Data have been obtained, but cannot be shared

  • Local tests have been done, but not through the orchestrator

Test case 3

  • Data have been obtained, but cannot be shared

  • Local tests have been done, but not through the orchestrator

Table : Test cases status for OPM Flow.

3.8.4. Roadmap

The roadmap for the pilot development is summarized in the table below.

Feature Description Deadline

TOSCA files for ensemble cases

New TOSCA files are needed to support running large ensembles.

February 2018

Documentation

New tutorials are needed for running both single and ensemble cases through the MSO4SC infrastructure

June 2018

Prepare test cases 1 and 2 for Portal

The cases must be prepared to make them possible to run from the Portal.

February 2018

Prepare test case 3 for Portal

The case must be prepared to make it possible to run from the Portal.

April 2018

Prepare simulator logging for event monitoring

Simulator log patterns must be defined to enable monitoring of simulation events.

April 2018

Prepare simulator for interactive use through Portal

The simulator system must be able to respond to pause, restart, stop and similar events initiated by the user through the Portal.

August 2018

Table : Roadmap for the OPM Flow pilot.

3.9. Other pilots

3.10. ZibAffinity

3.10.1. Software repository

The ZIBaffinity software embedded in a singularity container and all related TOSCA files as well as the sources, singularity recipe and third party software required for image building are available under the following URLs.

TOSCA blueprint files:

Singularity image recipe, ZIBaffinity sources, and third party software:

Contact: Vedat Durmaz (durmaz@zib.de), Marcus Weber (weber@zib.de)

3.10.2. Status of implementation

Services Description

MSO4SC Portal

  • Deployed

Authentication & Authorization

  • Authentication on FTII

Software repository

Data repository

  • In progress using girder and CKAN

PreProcessing

  • Done in TOSCA files

Experiments Tools

  • Run with one particular protein target are possible

  • Detailed log file is available

Post-Processing

  • Post-processing not mandatory. Imaginable using molecular visualization tool

Monitoring

  • Log file is generated and suitable for monitoring

  • Keywords for monitoring tool currently not defined

Table : Status of services integration for ZIB Affinity.

Status Description

CI/CO

  • Automatic generation of a docker imageTestsuite partly integrated into the docker image creation

  • Upload to private section of Feelpp Docker cloud organization

  • Integrate github repository within Buildkite system

Ports

  • Finis Terrae II (FTII) done,

  • need to upload script on MSO4SC github

Commercial Software / License issue

  • License have been uploaded on FTII and permissions are managed by FTII sysadmins

Testsuite

  • implementation of a testsuite based on simple academic case

  • Basic documentation in progress

Preparations of Data for benchmarks (see D5.1 [5])

  • data for benchmark cases prepared

  • data will be stored either in girder / github as they are sensitive

TOSCA files

  • basic TOSCA files for main testcases have been created

  • upload in MSO4SC/Resources github repository

Table : Status of development for ZIB Affinity.

3.10.3. Pilot test cases

The following features have been implemented for the pilot, listed in the table below. Compare with test cases listed in D5.1 [5].

Test case Description

Test case 1

Done (Absolute free energy of binding estimation for biological protein–ligand systems)

Test case 2

Done (Single unweighted contributions to the free energy of binding)

Table : Status of test cases for ZIB Affinity.

3.10.4. Roadmap

The roadmap for the pilot development is summarized in the table below.

Feature Description Deadline

Testsuite

Definition of a set of test files

End of January 2018

Software development

Adapting ZIBaffinity according to feedback from internal testers

End of March 2018

Documentation

Writing a documentation/manual for ZIBaffinity and its testcase

End of April 2018

MSO portal

Design of the ZIBaffinity part of the MSO portal

End of July 2018

TOSCA files

Adapt TOSCA definitions to MSO portal allowing flexible inputs

End of July 2018

Table : Roadmap for the ZIB Affinity pilot.

3.11. 3DAirQualityPrediction

3.11.1. Software repository

Phase 1 of the pilot is running on dedicated machines only due to license restrictions. Here follows the description of the access to all parts of it. All machines run on a HUAWEI cluster (name Huawei-HPC) running vmware ESXi enterprise class hypervisor.

Web interface to vmware (ESXi-6.5.0):

  • 10.9.0.203

The deployed machines are as follows:

  • Air Quality Prediction framework (Phase 1):

b605gw.eik.sze.hu, Ubuntu 16.04
(JAVA-based routines (e.g. automatized pre-processing tasks) also run here)

  • virtual machine for the license manager for the ANSYS Fluent:
    10.2.201.43, running CentOS 6

  • virtual machine for the microscopic traffic simulation:
    10.2.203.174, Windows 7, with VISSIM installed

  • virtual machine running Fluent solver (dispersion simulation)

10.2.201.43, (at present, it is the same machine as the license server)

  • dedicated SMP server for the dispersion simulation:

memo.sze.hu, Centos 7

To get access to the restricted domain contact Dávid Fülep at fulep@sze.hu. He provides access to the computers in the framework.

Since the elements of the framework run on different dedicated research machines, the rest of IP addresses are only accessible from the IP range of SZE. Users access virtual HPC cluster through VPN and can access deployed servers through SSH, or VNC-like remote desktop clients.

3.11.2. Status of implementation

The implementation of the 3DAirQualityPrediction pilot has of two basic phases:

  • Phase 1: implementation of the pilot as is, i.e. with proprietary software components to the MSO4SC infrastructure,

  • Phase 2: Implementation of the redesigned software architecture with open source components, in particular with 3DAirQualityPrediction-HPC.

The 3DAirQualityPrediction pilot has several proprietary software components, e.g. PTVs VISSIM for traffic simulation and ANSYS” Fluent for the dispersion simulation. These are state-of-the-art engineering solutions on their industrial domains. Although SZE purchased the licensed with unlimited time for running, licensing caused severe restrictions with the deployment into the MSO4SC infrastructure. Namely, the license managers run only under old versions of operating system and from legal point of view they may be run only on clusters belonging to the sze.hu domain. These statements became clear during the lifetime of the MSO4SC project. We have solved the problem In Phase 1 with

  • installation of a virtual machine for the license server and the traffic simulation software (this took rather a long time to figure out the proper settings and operational modes of the virtual machines);

  • installation of the software for the dispersion module at a dedicated cluster of SZE.

To cure the issues with the licensing, Phase 2 of the pilot is under construction. This will be composed of standardized input and output data formats for each module (traffic, emission, dispersion, evaluation) and, driven by a configuration file, for each module there will be options to select a solver including the Phase 1 proprietary solvers and open source solvers for each module including 3DAirQualityPrediction-HPC for the dispersion module solver.

Due to this plan, the deployment of the pilot Phase 2 is scheduled to the second part of the project (see the roadmap below).

Services Description

MSO4SC Portal

  • Ongoing work

  • Ongoing integration of SZE cluster with the Orchestrator, so remote jobs can be sent

Authentication & Authorization

  • Authentication on FTII

Software repository

  • Code and VMs are available at SZE

  • Current version contains proprietary software

Data repository

  • In progress using CKAN

Pre-Processing

  • Done, but not through TOSCA files

  • Some steps require user intervention

Experiments Tools

  • Integration of the SZE cluster with the Orchestrator is mandatory before the tool can be used

  • Ongoing preparation of the TOSCA files

Post-Processing

  • Post-processing not mandatory.

Table 27: Status of services integration for 3DAirQualityPrediction.

Status Description

CI/CO

  • Ongoing work to integrate with 3DAirQualityPrediction-CFD implemented with FEniCS

  • Code to be uploaded to CESGA GitLab repository for enabling CI.

Ports

  • Deployed at SZE cluster

Commercial Software / License issue

  • Licenses for several components made necessary to set up a VM for the licenses manager

  • Only can be run at SZE for the moment (phase 1)

Testsuite

  • Implementation of a basic testsuite

Preparations of Data for benchmarks (see D5.1 [5])

  • Data to be used is available

  • Still no data has been uploaded to the repository (CKAN)

TOSCA files

  • TOSCA files for phase 1 under definition

Table 28: Status of development for 3DAirQualityPrediction.

3.11.3. Pilot test cases

The following features have been implemented for the pilot, listed in the table below.

Test case Description

Test case 1

Done for the Phase 1 of the pilot.

Test case 2

Done for the Phase 1 of the pilot.

Test case 1 and 2 with Phase 2

To be done according to the roadmap below

Table : Test case status for 3DAirQualityPrediction.

3.11.4. Roadmap

The roadmap for the pilot development is summarized in the table below.

Feature Description Deadline

TOSCA file for Pilot Phase 1

Make the TOSCA-file of the Phase 1 and deploy it into MSO4SC e-infrastructure.

January 15, 2018

Test case 1 for Phase 1 in the infrastructure

Test requirements in the infrastructure, case 1

January 22, 2018

Test case 2 for Phase 1 in the infrastructure

Test requirements in the infrastructure, case 1

January 31, 2018

Architecture design of Phase 2

Complete professional architecture design of Phase 2

January 22, 2018

Implementation of Phase 2 of the pilot

Software development of the pilot Phase 2.

31 March, 2018

Containerization

Containerization of Phase 2 software

30 April 2018

MSO4SC Portal, monitoring

Deployment of Phase 2 to the portal and solve monitoring

30 April 2018

Software and data repositories

TOSCA file definitions and made available at gitlab

Singularity recipe, sources, third-party software in the repository

30 June 2018

PreProcessing

To be done in TOSCA files if possible (note: several preprocesing steps are user dependent and made manually)

31 July 2018

Post-Processing

Post-processing not mandatory.

31 July 2018

Table 30: Roadmap for the 3DAirQualityPrediction pilot

4. Conclusions

In this report we have described the deliverable D5.3, summarizing the implementation of pilots. It is clear that the pilots implementations have different maturity levels. While HiFi Magnet, Eye2Brain and OPM Flow have progressed more towards the integration with the MSO4SC e-Infrastructure, others are using only some basic features and are working in order to complete such integration.

As for the features themselves, all the pilots provide, at least, some basic features, in line with the definition proposed in D5.1 [5]. Such features allowed to perform a validation of requirements, but the pilots will keep progressing in order to fulfil the requirements and cover all the needed scenarios, as planned in the roadmaps.

Finally, although licenses for proprietary software are creating some issues, we have been able to deal with them in one way or another, although in one of the pilots, it became a complex point to deal with.

References

  1. MSO4SC, D5.2 Operational MSO4SC e-infrastructure, 2017.

  2. MSO4SC, D4.2 Adapted MADFs for MSO4SC e-infrastructure, 2017.

  3. MSO4SC, D2.1 End Users’ Requirements Report, 2017.

  4. MSO4SC, D5.4 Evaluation report, 2017.

  5. MSO4SC, D5.1 Case study extended design and evaluation strategy, 2017.

  6. MSO4SC, D2.2 MSO4SC e-Infrastructure Definition, 2017

  7. MSO4SC, D3.1 Detailed Specifications for the Infrastructure, Cloud Management and MSO Portal, 2017

  8. MSO4SC, D4.1 Detailed Specification for the MADFS, 2017

  9. MSO4SC, D2.5 End Users’ Requirements Report v2, 2017.


3. OpenModelica : openmodelica.org/