BeeFlow: A Workflow Management System for in Situ Processing Across HPC and Cloud Systems

Jieyang Chen; Qiang Guan; Zhao Zhang; Xin Liang; Louis Vernon; Allen McPherson; Li Ta Lo; Patricia Grubel

doi:10.1109/ICDCS.2018.00103

BeeFlow: A Workflow Management System for in Situ Processing Across HPC and Cloud Systems

Jieyang Chen
, Qiang Guan
, Zhao Zhang
, Xin Liang
, Louis Vernon
, Allen McPherson
, Li Ta Lo
, Patricia Grubel

Missouri University of Science and Technology

Research output: Contribution to journal › Article › peer-review

Abstract

In this paper, we propose BeeFlow - an in situ analysis enabled workflow management system across multiple platforms using Docker containers. BeeFlow can support both traditional workflows as well as workflows with in situ analysis. BeeFlow leverages Docker containers to provide a portable, flexible, and reproducible workflow management system across HPC and cloud platforms. We showcase how current in situ visualization workflows can apply BeeFlow with DOE production codes VPIC and Flecsale.

Original language	American English
Journal	Proceedings of the 38th International Conference on Distributed Computing Systems (2018, Vienna, Austria)
DOIs	https://doi.org/10.1109/ICDCS.2018.00103
State	Published - Jul 19 2018

Keywords

Cloud Computing
Container
High Performance Computing
Scientific Workflow

Disciplines

Computer Sciences

Access to Document

10.1109/ICDCS.2018.00103License: Unspecified

https://doi.org/10.1109/ICDCS.2018.00103

Cite this

@article{82889c3f57f14d568fc10be99dfc70d9,

title = "BeeFlow: A Workflow Management System for in Situ Processing Across HPC and Cloud Systems",

abstract = " In this paper, we propose BeeFlow - an in situ analysis enabled workflow management system across multiple platforms using Docker containers. BeeFlow can support both traditional workflows as well as workflows with in situ analysis. BeeFlow leverages Docker containers to provide a portable, flexible, and reproducible workflow management system across HPC and cloud platforms. We showcase how current in situ visualization workflows can apply BeeFlow with DOE production codes VPIC and Flecsale.",

keywords = "Cloud Computing, Container, High Performance Computing, Scientific Workflow",

author = "Jieyang Chen and Qiang Guan and Zhao Zhang and Xin Liang and Louis Vernon and Allen McPherson and Lo, \{Li Ta\} and Patricia Grubel",

year = "2018",

month = jul,

day = "19",

doi = "10.1109/ICDCS.2018.00103",

language = "American English",

journal = "Proceedings of the 38th International Conference on Distributed Computing Systems (2018, Vienna, Austria)",

}

TY - JOUR

T1 - BeeFlow: A Workflow Management System for in Situ Processing Across HPC and Cloud Systems

AU - Chen, Jieyang

AU - Guan, Qiang

AU - Zhang, Zhao

AU - Liang, Xin

AU - Vernon, Louis

AU - McPherson, Allen

AU - Lo, Li Ta

AU - Grubel, Patricia

PY - 2018/7/19

Y1 - 2018/7/19

N2 - In this paper, we propose BeeFlow - an in situ analysis enabled workflow management system across multiple platforms using Docker containers. BeeFlow can support both traditional workflows as well as workflows with in situ analysis. BeeFlow leverages Docker containers to provide a portable, flexible, and reproducible workflow management system across HPC and cloud platforms. We showcase how current in situ visualization workflows can apply BeeFlow with DOE production codes VPIC and Flecsale.

AB - In this paper, we propose BeeFlow - an in situ analysis enabled workflow management system across multiple platforms using Docker containers. BeeFlow can support both traditional workflows as well as workflows with in situ analysis. BeeFlow leverages Docker containers to provide a portable, flexible, and reproducible workflow management system across HPC and cloud platforms. We showcase how current in situ visualization workflows can apply BeeFlow with DOE production codes VPIC and Flecsale.

KW - Cloud Computing

KW - Container

KW - High Performance Computing

KW - Scientific Workflow

UR - https://scholarsmine.mst.edu/comsci_facwork/1122

UR - https://doi.org/10.1109/ICDCS.2018.00103

U2 - 10.1109/ICDCS.2018.00103

DO - 10.1109/ICDCS.2018.00103

M3 - Article

JO - Proceedings of the 38th International Conference on Distributed Computing Systems (2018, Vienna, Austria)

JF - Proceedings of the 38th International Conference on Distributed Computing Systems (2018, Vienna, Austria)

ER -

BeeFlow: A Workflow Management System for in Situ Processing Across HPC and Cloud Systems

Abstract

Keywords

Disciplines

Access to Document

Other files and links

Fingerprint

Cite this