Cuneiform. A Functional Workflow Language Implementation in Erlang. Jörgen Brandt Humboldt-Universität zu Berlin

Transcription

1 Cuneiform A Functional Workflow Language Implementation in Erlang Jörgen Brandt Humboldt-Universität zu Berlin Jörgen Brandt (HU Berlin) Cuneiform / 27

2 Cuneiform Cuneiform A Functional Language for Large Scale Scientific Data Analysis Jörgen Brandt (HU Berlin) Cuneiform / 27

3 Cuneiform Cuneiform A Functional Language for Large Scale Scientific Data Analysis Black-box operator model Pro: Operators can be any piece of software Jörgen Brandt (HU Berlin) Cuneiform / 27

4 Cuneiform Cuneiform A Functional Language for Large Scale Scientific Data Analysis Black-box operator model Pro: Operators can be any piece of software Black-box data model Pro: Input and Output data can be anything Jörgen Brandt (HU Berlin) Cuneiform / 27

5 Cuneiform Cuneiform A Functional Language for Large Scale Scientific Data Analysis Black-box operator model Pro: Operators can be any piece of software Black-box data model Pro: Input and Output data can be anything Features of advanced workflow languages Abstractions, lists, operations on lists, conditions Jörgen Brandt (HU Berlin) Cuneiform / 27

6 Cuneiform Cuneiform A Functional Language for Large Scale Scientific Data Analysis Black-box operator model Pro: Operators can be any piece of software Black-box data model Pro: Input and Output data can be anything Features of advanced workflow languages Abstractions, lists, operations on lists, conditions Light-weight Foreign Function Interface (FFI) Wrapping in R, Matlab, Octave, Python, Lisp, Perl, Bash Jörgen Brandt (HU Berlin) Cuneiform / 27

7 Cuneiform Cuneiform A Functional Language for Large Scale Scientific Data Analysis Black-box operator model Pro: Operators can be any piece of software Black-box data model Pro: Input and Output data can be anything Features of advanced workflow languages Abstractions, lists, operations on lists, conditions Light-weight Foreign Function Interface (FFI) Wrapping in R, Matlab, Octave, Python, Lisp, Perl, Bash Automatic parallelization Scalability with large data sets Jörgen Brandt (HU Berlin) Cuneiform / 27

8 Motivation New hardware is increasingly parallel, so new programming languages must support concurrency or they will die. Joe Armstrong Jörgen Brandt (HU Berlin) Cuneiform / 27

9 New Hardware Jo rgen Brandt (HU Berlin) Cuneiform / 27

10 DNA Sequencing is becoming cheap Jörgen Brandt (HU Berlin) Cuneiform / 27

11 Decentralized software development Jörgen Brandt (HU Berlin) Cuneiform / 27

12 Scientific Workflow Systems Jörgen Brandt (HU Berlin) Cuneiform / 27

13 Scientific Workflow Systems Workflows as DAGs Scientific Workflows are DAGs Jo rgen Brandt (HU Berlin) Cuneiform / 27

14 Scientific Workflow Systems Workflows as DAGs Scientific Workflows are DAGs Nodes are tasks Jo rgen Brandt (HU Berlin) Cuneiform / 27

15 Scientific Workflow Systems Workflows as DAGs Scientific Workflows are DAGs Nodes are tasks Edges are data dependencies Jo rgen Brandt (HU Berlin) Cuneiform / 27

16 Example: Galaxy Workflow System Focus on Usability Jörgen Brandt (HU Berlin) Cuneiform / 27

17 Example: Galaxy Workflow System Focus on Usability Integration of tools/libraries Jörgen Brandt (HU Berlin) Cuneiform / 27

18 Example: Galaxy Workflow System Focus on Usability Systematic documentation Integration of tools/libraries Jörgen Brandt (HU Berlin) Cuneiform / 27

19 Example: Galaxy Workflow System Focus on Usability Systematic documentation Integration of tools/libraries Reproducibility Jörgen Brandt (HU Berlin) Cuneiform / 27

20 The Next Generation Sequencing use case Jörgen Brandt (HU Berlin) Cuneiform / 27

21 The Next Generation Sequencing use case Jörgen Brandt (HU Berlin) Cuneiform / 27

22 The Next Generation Sequencing use case Jo rgen Brandt (HU Berlin) Cuneiform / 27

23 The Next Generation Sequencing use case Jo rgen Brandt (HU Berlin) Cuneiform / 27

24 The Next Generation Sequencing use case Jo rgen Brandt (HU Berlin) Cuneiform / 27

25 Desired Features in a Language Is there a language that is... Jörgen Brandt (HU Berlin) Cuneiform / 27

26 Desired Features in a Language Is there a language that is... Like a workflow language So we can integrate all the tools Jörgen Brandt (HU Berlin) Cuneiform / 27

27 Desired Features in a Language Is there a language that is... Like a workflow language So we can integrate all the tools Like MapReduce So we can derive parallelism and distribute the work Jörgen Brandt (HU Berlin) Cuneiform / 27

28 Desired Features in a Language Is there a language that is... Like a workflow language So we can integrate all the tools Like MapReduce So we can derive parallelism and distribute the work Like a functional programming language So we can write arbitrary programs using lists and operations on lists Jörgen Brandt (HU Berlin) Cuneiform / 27

29 Cuneiform example deftask gunzip( out( File ) : gz( File ) )in bash *{ gzip -c -d $gz > $out }* Jörgen Brandt (HU Berlin) Cuneiform / 27

30 Cuneiform example deftask gunzip( out( File ) : gz( File ) )in bash *{ gzip -c -d $gz > $out }* gunzip gunzip( gz: 'myarchive1.gz' ); Jörgen Brandt (HU Berlin) Cuneiform / 27

31 Cuneiform example deftask gunzip( out( File ) : gz( File ) )in bash *{ gzip -c -d $gz > $out }* gunzip gunzip( gz: 'myarchive1.gz' 'myarchive2.gz' ); Jörgen Brandt (HU Berlin) Cuneiform / 27

32 Cuneiform example deftask gunzip( out( File ) : gz( File ) )in bash *{ gzip -c -d $gz > $out }* gunzip gunzip( gz: 'myarchive1.gz' 'myarchive2.gz' 'myarchive3.gz' ); Jörgen Brandt (HU Berlin) Cuneiform / 27

33 Workflow Implementations Available Available example workflows: Variant calling joergen7/variant-call samtools-faidx tophat-single fastq-dump cuffdiff cufflinks cummerbund cuffmerge Jörgen Brandt (HU Berlin) Cuneiform / 27

34 Workflow Implementations Available Available example workflows: Variant calling joergen7/variant-call Methylation joergen7/methylation samtools-faidx fastq-dump tophat-single cuffdiff cufflinks cummerbund cuffmerge Jörgen Brandt (HU Berlin) Cuneiform / 27

35 Workflow Implementations Available Available example workflows: Variant calling joergen7/variant-call Methylation joergen7/methylation RNA-Seq com/joergen7/rna-seq fastq-dump samtools-faidx tophat-single cuffdiff cufflinks cummerbund cuffmerge Jörgen Brandt (HU Berlin) Cuneiform / 27

36 Workflow Implementations Available Available example workflows: Variant calling joergen7/variant-call Methylation joergen7/methylation RNA-Seq com/joergen7/rna-seq etc (ChIP-Seq, mirna detection, consensus prediction,... ) fastq-dump samtools-faidx tophat-single cuffdiff cufflinks cummerbund cuffmerge Jörgen Brandt (HU Berlin) Cuneiform / 27

37 Cuneiform Operational Semantics Jörgen Brandt (HU Berlin) Cuneiform / 27

38 Program Interpretation in Cuneiform Current design in Java implementation: EBNF Parser Generator (ANTLR) Scanner/ Transcrip on Abstract Program Parse tree Interpreter Result Parser Visitor Program Jörgen Brandt (HU Berlin) Cuneiform / 27

39 Program Interpretation in Cuneiform Designated design but never implemented: EBNF Opera onal Seman cs Parser Generator (ANTLR) Program Extrac on (?) Scanner/ Transcrip on Abstract Program Parse tree Interpreter Result Parser Visitor Program Jörgen Brandt (HU Berlin) Cuneiform / 27

40 Program Interpretation in Cuneiform Designated design in Erlang: EBNF Opera onal Seman cs Parser Generator (ANTLR) Program Extrac on (?) Scanner/ Transcrip on Abstract Program Parse tree Interpreter Result Parser Visitor Program Jörgen Brandt (HU Berlin) Cuneiform / 27

41 The eval Function fun eval(expr, ρ, GetFuture, Global, Fin) Result when Expr :: expr ρ :: string => expr GetFuture :: fun Global :: string => lam Fin :: id => expr Result :: expr Jörgen Brandt (HU Berlin) Cuneiform / 27

42 The eval Function fun eval(expr, ρ, GetFuture, Global, Fin) Result when Expr :: expr ρ :: string => expr GetFuture :: fun Global :: string => lam Fin :: id => expr Result :: expr Expr The expression to be evaluated Jörgen Brandt (HU Berlin) Cuneiform / 27

43 The eval Function fun eval(expr, ρ, GetFuture, Global, Fin) Result when Expr :: expr ρ :: string => expr GetFuture :: fun Global :: string => lam Fin :: id => expr Result :: expr Expr The expression to be evaluated ρ Current scope Jörgen Brandt (HU Berlin) Cuneiform / 27

44 The eval Function fun eval(expr, ρ, GetFuture, Global, Fin) Result when Expr :: expr ρ :: string => expr GetFuture :: fun Global :: string => lam Fin :: id => expr Result :: expr Expr The expression to be evaluated ρ Current scope GetFuture A function returning a future for a foreign task application Jörgen Brandt (HU Berlin) Cuneiform / 27

45 The eval Function fun eval(expr, ρ, GetFuture, Global, Fin) Result when Expr :: expr ρ :: string => expr GetFuture :: fun Global :: string => lam Fin :: id => expr Result :: expr Expr The expression to be evaluated ρ Current scope GetFuture A function returning a future for a foreign task application Global Task definitions Jörgen Brandt (HU Berlin) Cuneiform / 27

46 The eval Function fun eval(expr, ρ, GetFuture, Global, Fin) Result when Expr :: expr ρ :: string => expr GetFuture :: fun Global :: string => lam Fin :: id => expr Result :: expr Expr The expression to be evaluated ρ Current scope GetFuture A function returning a future for a foreign task application Global Task definitions Fin Results of foreign task applications Jörgen Brandt (HU Berlin) Cuneiform / 27

47 The eval Function fun eval(expr, ρ, GetFuture, Global, Fin) Result when Expr :: expr ρ :: string => expr GetFuture :: fun Global :: string => lam Fin :: id => expr Result :: expr Expr The expression to be evaluated ρ Current scope GetFuture A function returning a future for a foreign task application Global Task definitions Fin Results of foreign task applications Result The result of evaluation (may contain futures) Jörgen Brandt (HU Berlin) Cuneiform / 27

48 Computational Semantics eval(expr, ρ, GetFuture, Global, Fin) Next = step(expr, ρ, GetFuture, Global, Fin) case Next of Expr Expr eval(next, ρ, GetFuture, Global, Fin) end Jörgen Brandt (HU Berlin) Cuneiform / 27

49 Computational Semantics eval(expr, ρ, GetFuture, Global, Fin) Next = step(expr, ρ, GetFuture, Global, Fin) case Next of Expr Expr eval(next, ρ, GetFuture, Global, Fin) end single step is computed Jörgen Brandt (HU Berlin) Cuneiform / 27

50 Computational Semantics eval(expr, ρ, GetFuture, Global, Fin) Next = step(expr, ρ, GetFuture, Global, Fin) case Next of Expr Expr eval(next, ρ, GetFuture, Global, Fin) end single step is computed If the step has no effect evaluation terminates Jörgen Brandt (HU Berlin) Cuneiform / 27

51 Computational Semantics fun eval(expr, ρ, GetFuture, Global, Fin) Next = step(expr, ρ, GetFuture, Global, Fin) case Next of Expr Expr eval(next, ρ, GetFuture, Global, Fin) end single step is computed If the step has no effect evaluation terminates Otherwise eval is called recursively Jörgen Brandt (HU Berlin) Cuneiform / 27

52 Languages Suitable for Operational Semantics Common Lisp Choosing a language Functional Language Haskell Scala Erlang ML Jörgen Brandt (HU Berlin) Cuneiform / 27

53 Formalisms Suitable for Operational Semantics Common Lisp Choosing a language Functional Language With Pattern Matching Haskell Scala Erlang ML Jörgen Brandt (HU Berlin) Cuneiform / 27

54 Formalisms Suitable for Operational Semantics Common Lisp Choosing a language Functional Language With Pattern Matching Concurrency Orientation Haskell Scala Erlang ML Jörgen Brandt (HU Berlin) Cuneiform / 27

55 Program Interpretation in Cuneiform New design: EBNF Opera onal Seman cs Parser Generator (Leex/Yecc) Compiler (Erlang) Scanner/ Abstract Program Interpreter Result Parser Program Jörgen Brandt (HU Berlin) Cuneiform / 27

56 Program Interpretation in Cuneiform New design: EBNF Opera onal Seman cs Parser Generator (Leex/Yecc) Compiler (Erlang) Scanner/ Abstract Program Interpreter Result Parser Program Jörgen Brandt (HU Berlin) Cuneiform / 27

57 Fault Tolerance Jörgen Brandt (HU Berlin) Cuneiform / 27

58 Two types of complexity n Processes Distributed applications can be complex in two independent ways: n Tasks Jörgen Brandt (HU Berlin) Cuneiform / 27

59 Two types of complexity n Processes Distributed applications can be complex in two independent ways: in the number of processes involved to compute one task the more system components the more likely one component fails n Tasks Jörgen Brandt (HU Berlin) Cuneiform / 27

60 Two types of complexity n Processes n Tasks Distributed applications can be complex in two independent ways: in the number of processes involved to compute one task the more system components the more likely one component fails in the number of tasks contributing to a workflow the more tasks the more likely one task fails Jörgen Brandt (HU Berlin) Cuneiform / 27

61 Two types of complexity n Processes n Tasks Distributed applications can be complex in two independent ways: in the number of processes involved to compute one task the more system components the more likely one component fails in the number of tasks contributing to a workflow the more tasks the more likely one task fails Jörgen Brandt (HU Berlin) Cuneiform / 27

62 Two types of complexity n Processes n Tasks Distributed applications can be complex in two independent ways: in the number of processes involved to compute one task the more system components the more likely one component fails in the number of tasks contributing to a workflow the more tasks the more likely one task fails Jörgen Brandt (HU Berlin) Cuneiform / 27

63 Distributed Application: Workflow System Query Query Query Jörgen Brandt (HU Berlin) Cuneiform / 27

64 Distributed Application: Workflow System Query Query Query Cache Jörgen Brandt (HU Berlin) Cuneiform / 27

65 Distributed Application: Workflow System Query Query Query Cache Scheduler Jörgen Brandt (HU Berlin) Cuneiform / 27

66 Distributed Application: Workflow System Query Query Query Cache Scheduler FS Work FS Work FS Work Jörgen Brandt (HU Berlin) Cuneiform / 27

67 How to achieve fault tolerance when Jörgen Brandt (HU Berlin) Cuneiform / 27

68 How to achieve fault tolerance when Workflow systems are complex in both Number of processes involved in computing one task Number of tasks in one workflow so failures are likely Jörgen Brandt (HU Berlin) Cuneiform / 27

69 How to achieve fault tolerance when Workflow systems are complex in both Number of processes involved in computing one task Number of tasks in one workflow so failures are likely All components need to maintain state so plain restarting of components is not enough Jörgen Brandt (HU Berlin) Cuneiform / 27

70 How to achieve fault tolerance when Workflow systems are complex in both Number of processes involved in computing one task Number of tasks in one workflow so failures are likely All components need to maintain state so plain restarting of components is not enough Restarting of workflow helps only if workflows are small and system has few components Jörgen Brandt (HU Berlin) Cuneiform / 27

71 Generic process behaviour Golden path: P 1 sends request to P 2 P1 request P2 Jörgen Brandt (HU Berlin) Cuneiform / 27

72 Generic process behaviour Golden path: P 1 sends request to P 2 Asynchronously P 1 receives reply P1 reply request P2 Jörgen Brandt (HU Berlin) Cuneiform / 27

73 Generic process behaviour P 2 may fail: P 1 sends request to P 2 P 2 fails P1 request P2 Jörgen Brandt (HU Berlin) Cuneiform / 27

74 Generic process behaviour monitor P 2 may fail: P 1 sends request to P 2 P 1 creates monitor on P 2 P1 request P2 Jörgen Brandt (HU Berlin) Cuneiform / 27

75 Generic process behaviour monitor P 2 may fail: P 1 sends request to P 2 P 1 creates monitor on P 2 P 1 memorizes request P1 request P2 { } Jörgen Brandt (HU Berlin) Cuneiform / 27

76 Generic process behaviour monitor P 2 may fail: P 1 sends request to P 2 P 1 creates monitor on P 2 P 1 memorizes request P 2 fails P1 request P2 { } Jörgen Brandt (HU Berlin) Cuneiform / 27

77 Generic process behaviour P 2 may fail: P 1 sends request to P 2 P 1 creates monitor on P 2 P 1 memorizes request P 2 fails P1 P2 P 2 supervisor restarts P 2 { } Jörgen Brandt (HU Berlin) Cuneiform / 27

78 Generic process behaviour P 2 may fail: monitor P 1 sends request to P 2 P 1 creates monitor on P 2 P 1 memorizes request P 2 fails P 2 supervisor restarts P 2 P1 request P2 request is replayed to P 2 monitor is recreated { } Jörgen Brandt (HU Berlin) Cuneiform / 27

79 Generic process behaviour P 1 may fail: P 1 sends request to P 2 P 1 fails P1 request P2 Jörgen Brandt (HU Berlin) Cuneiform / 27

80 Generic process behaviour monitor P 1 may fail: P 1 sends request to P 2 P 2 creates monitor on P 1 P1 request P2 Jörgen Brandt (HU Berlin) Cuneiform / 27

81 Generic process behaviour monitor P 1 may fail: P 1 sends request to P 2 P 2 creates monitor on P 1 P 1 fails P1 request P2 Jörgen Brandt (HU Berlin) Cuneiform / 27

82 Generic process behaviour P 1 may fail: P 1 sends request to P 2 P 2 creates monitor on P 1 P 1 fails request is canceled supervisor restarts P 1 P1 P2 Jörgen Brandt (HU Berlin) Cuneiform / 27

83 Conclusion untar Cuneiform: gunzip bowtie2-build bowtie2-align samtools-view samtools-sort samtools-faidx samtools-mpileup varscan annovar Jörgen Brandt (HU Berlin) Cuneiform / 27

84 Conclusion untar Cuneiform: Functional gunzip bowtie2-build bowtie2-align samtools-view samtools-sort samtools-faidx samtools-mpileup varscan annovar Jörgen Brandt (HU Berlin) Cuneiform / 27

85 Conclusion untar Cuneiform: Functional Integrate anything gunzip bowtie2-build bowtie2-align samtools-view samtools-sort samtools-faidx samtools-mpileup varscan annovar Jörgen Brandt (HU Berlin) Cuneiform / 27

86 Conclusion untar Cuneiform: Functional Integrate anything Parallelism gunzip bowtie2-build bowtie2-align samtools-view samtools-sort samtools-faidx samtools-mpileup varscan annovar Jörgen Brandt (HU Berlin) Cuneiform / 27

87 Conclusion untar Cuneiform: Functional Integrate anything Parallelism gunzip bowtie2-build bowtie2-align samtools-view Runs on Hadoop samtools-sort samtools-faidx samtools-mpileup varscan annovar Jörgen Brandt (HU Berlin) Cuneiform / 27

88 Conclusion untar Cuneiform: Functional Integrate anything Parallelism Runs on Hadoop Implementation in Erlang: gunzip bowtie2-build bowtie2-align samtools-view samtools-sort samtools-faidx samtools-mpileup varscan annovar Jörgen Brandt (HU Berlin) Cuneiform / 27

89 Conclusion untar Cuneiform: Functional Integrate anything Parallelism Runs on Hadoop Implementation in Erlang: Concise stateless semantics gunzip bowtie2-build bowtie2-align samtools-view samtools-sort samtools-faidx samtools-mpileup varscan annovar Jörgen Brandt (HU Berlin) Cuneiform / 27

90 Conclusion untar Cuneiform: Functional Integrate anything Parallelism Runs on Hadoop Implementation in Erlang: Concise stateless semantics Fine-grained fault tolerance gunzip bowtie2-build bowtie2-align samtools-view samtools-sort samtools-faidx samtools-mpileup varscan annovar Jörgen Brandt (HU Berlin) Cuneiform / 27