A Workflow-Enabled Big Data Analytics Software Stack for eScience

Abstract: The availability of systems able to process and analyse big amount of data has boosted scientific advances in several fields. Workflows provide an effective tool to define and manage large sets of processing tasks. In the big data analytics area, the Ophidia project provides a cross-domain big data analytics framework for the analysis of scientific, multi-dimensional datasets. The framework exploits a server-side, declarative, parallel approach for data analysis and mining. It also features a complete workflow management system to support the execution of complex scientific data analysis, schedule tasks submission, manage operators dependencies and monitor jobs execution. The workflow management engine allows users to perform a coordinated execution of multiple data analytics operators (both single and massive - parameter sweep) in an effective manner. For the definition of the big data analytics workflow, a JSON schema has been properly designed and implemented. To aid the definition of the workflows, a visual design language consisting of several symbols, named Data Analytics Workflow Modelling Language (DAWML), has been also defined.

Event: Second International Symposium on Big Data Principles, Architectures & Applications – BDAA 2015, as part of The International Conference on High Performance Computing & Simulation (HPCS 2015), Amsterdam, The Netherlands. (BDAA - HPCS 2015).

Date: July 20 - 24, 2015

Authors: C. Palazzo (1), A. Mariello (1), S. Fiore (1), A. D’Anca (1), D. Elia (1), D. N. Williams (2), G. Aloisio (1,3) 1 - Centro Euro-Mediterraneo sui Cambiamenti Climatici, Lecce, Italy 2 - Lawrence Livermore National Laboratory, Livermore, US 3 - University of Salento, Lecce, Italy