MEWSE - Multi Engine Workflow Submission and Execution on Apache YARN
Loading...
Authors
Sundaravarathan, Kiran
Date
2015-09-15
Type
thesis
Language
eng
Keyword
Big Data , Analytic systems , Workflow Submitter , Apache YARN
Alternative Title
Abstract
In this era of BigData, designing a workflow to gain insights from the vast amount of data has become more complex. There are several different frameworks which individually process the batch and streaming data but coordinating the jobs between the engines in the workflow creates a performance penalty and other performance issues. Current workflow systems typically run only on one engine and do not offer the versatility required for today’s workflows. The process of submitting the jobs on different engines manually is not only time consuming, but also requires the expertise of working on these engines. In this thesis, we have overcome the above mentioned issues by proposing a MEWSE - Multi Engine Workflow Submission and Execution on Apache YARN. It should also have design with plug and play functionalities to allow the inclusion of new engines. MEWSE has been tested on Amazon EC2 with a sample workflow which requires the following engines, Hadoop, Mahout, java and some scripts to process the data.
Description
Thesis (Master, Computing) -- Queen's University, 2015-09-14 18:00:28.306
Citation
Publisher
License
Queen's University's Thesis/Dissertation Non-Exclusive License for Deposit to QSpace and Library and Archives Canada
ProQuest PhD and Master's Theses International Dissemination Agreement
Intellectual Property Guidelines at Queen's University
Copying and Preserving Your Thesis
This publication is made available by the authority of the copyright owner solely for the purpose of private study and research and may not be copied or reproduced except as permitted by the copyright laws without written authority from the copyright owner.
ProQuest PhD and Master's Theses International Dissemination Agreement
Intellectual Property Guidelines at Queen's University
Copying and Preserving Your Thesis
This publication is made available by the authority of the copyright owner solely for the purpose of private study and research and may not be copied or reproduced except as permitted by the copyright laws without written authority from the copyright owner.