Doman Decomposton Paallelzaton of Mesh Based pplcatons Panagots damds Thomas Bönsch Unvesty of Stuttgat Hgh-Pefomance Computng-Cente Stuttgat (HLRS) wwwhlsde Höchstlestungsechenzentum Stuttgat Outlne ntoducton Bascs Bounday Handlng Eample: Fnte Volume Flow Smulaton on Stuctued Meshes Eample: Fnte Element ppoach on an Unstuctued Mesh Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Paallelzaton - Taget Hgh pplcaton Pefomance Usng eal g MPP s no loss n effcency even when usng 00 Pocessos and moe Usng Clustes of SMP s no decease n Pefomance due to usng etenal Netwok connectons Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Polem () Flow aound a cylnde: Numecal Smulaton usng FV, FE o FD Data Stuctue: (:n,:m) Solve: (+B+C) Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Paallelzaton stateges Wok decomposton Scalng? do,00 Í,,0,,00 Flow aound a cylnde: Numecal Smulaton usng FV, FE o FD Data Stuctue: (:n,:m) Solve: (+B+C) Data decomposton Scales too much communcaton? (:0,:0) (:0,:00) (:0,0:0) (:0,:00) Doman decomposton Good Chance Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Paallelzaton Polems Decomposton (Doman, Data, Wok) Communcaton du / d ( u + u )/ d u - u + Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Concepts - Message Passng () Use defned communcaton Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat How to splt the Doman n the Dmensons () - Dmensonal (and ) - Dmensonal Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
How to splt the Doman n the Dmensons () That depends on: computatonal speed e pocesso: vectopocesso o cache communcaton speed: latency andwdth topology nume of sudomans needed load dstuton (s the effot fo evey mesh cell equal) Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Replcaton vesus Communcaton () f we need a value fom a neghou we have ascally two oppotuntes gettng the necessay value dectly fom the neghou, when needed Communcaton, ddtonal Synchonsaton calculatng the value of the neghou agan locally fom values known thee ddtonal Calculaton Selecton depends on the applcaton Doman Decomposton damds/bönsch Slde 0 Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Replcaton vesus Communcaton () Nomally eplcate the values Consde how many calculatons you can eecute whle only sendng Bt fom one pocess to anothe ( µs, 0 Gflop/s Í 000 opeatons) Sendng kbyte (00) doules (wth 00 MB/s andwdth Í µs Í 00 opeatons) vey often locks have to wat fo the neghous ut eta wok lmts paallel effcency Communcaton should only e used f one s qute sue that ths s the est soluton Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat - Dmensonal DD wth two Halo Cells Mesh Pattonng Sudoman fo each Pocess Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Eample: Paallelzaton of a D Fnte Volume Flow Solve Höchstlestungsechenzentum Stuttgat Statng Pont: Sequentell Pogam Wtten n FORTRN Usng stuctued meshes Pats of the pogam Pepocessng, eadng Data Man Loop Setup of the equaton system Pecondtonng Solvng step Postpocessng, wtng Data Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Dynamc Data Stuctues Pue FORTRN s too statc nume of pocessos can vay fom un to un sze of aays even wthn the same case can vay dynamc data stuctus use Fotan0 dynamc aays use all local memoy on a PE fo a huge FORTRN aay and setup you own memoy management second method has a polem on SMP s and cc-num s we should only use as much memoy as necessay Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Man Loop Setup of the equaton system each cell has neghous needs data fom neghou cells halo cells at the nne oundaes Pecondtonng no neghou nfomaton needed at all completely done locally Solvng step Jaco Lne elaaton wth suteatons moe complcated (net sldes) Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Heptadagonalmat D D D D D D Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Paallelzaton - Solve () Sequental: u u Paallelzaton polems: Mat s dstuted Mat nveson s a po not paallelzale Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Doman Decomposton Paallelzaton of Mesh Based pplcatons -0 Slde Doman Decomposton Höchstlestungsechenzentum Stuttgat damds/bönsch Paallelzaton - Solve () + + ) ( ) ( ) ( ) ( ) ( j j j j Mu L u Mu Lu Mu Lu M u L M L u Slde 0 Doman Decomposton Höchstlestungsechenzentum Stuttgat damds/bönsch Paallelzaton - Solve () + 0 0 0 m l u m l u m l u m m l u m l u m l u m m l u m l u m l u m l u l u M + L
Real Heptadagonalmat D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D D 0 0 0 Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Dffeence etween stong / weak couplng Solve wth weak couplng Eta computatonal effot due to addtonal solvng step (ut no factozaton) ddtonal update of ght hand sde Two tmes communcaton one afte each solvng step Solve wth stonge couplng Jaco lne elaaton method wth suteatons collapse teatons fom lne elaaton and paallelzaton no addtonal teatons much moe communcaton Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Compason of Solves - Convegence Solve Solve L - Resdual teatons Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Compason of Solves - Pefomance Tme n Seconds stong Ethenet weak Ethenet weak HPS stong HPS Nume of Pocessos Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Results - Solvng Method the pesented solve wth weak couplng woks fne fo ths CFD polems Solutons dffe n the scale of one pecent convegence ate s nealy equal to the sequental pogam Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Speedup on Cay TE Speedup Polemsze 0 Polemsze Polemsze Nume of Pocessos Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Scaleup on Cay TE Scaleup Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Nume of Pocessos Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Doman Decomposton of Unstuctued Gds Höchstlestungsechenzentum Stuttgat Unstuctued FEM Gd wth Gloal Numeng 0 Doman Decomposton damds/bönsch Slde 0 Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Shape of Coespondng System Mat 0 0 edge of FEM gd Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Nonovelappng Doman Decomposton Gd Ponts Sepaated nto nne and Bounday Ponts Renumeng of the nne and Bounday Ponts Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Nonovelappng Doman Decomposton P P 0 P Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Gd Ponts Sepaated nto nne and Bounday Ponts P P 0 P Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Doman Decomposton Paallelzaton of Mesh Based pplcatons - Slde Doman Decomposton Höchstlestungsechenzentum Stuttgat damds/bönsch Renumeng of the nne and Bounday Ponts P P P 0 nne ponts ae numeed fom up to the nume of nne ponts Bounday ponts ae numeed gloally statng fom the mamum of all nne ponts of all pattons plus Slde Doman Decomposton Höchstlestungsechenzentum Stuttgat damds/bönsch ow Shaped System Mat afte Renumeng P P P logcal ounday 0 0
Doman Decomposton Paallelzaton of Mesh Based pplcatons - Slde Doman Decomposton Höchstlestungsechenzentum Stuttgat damds/bönsch Data Dstuton P P 0 P 0 Slde Doman Decomposton Höchstlestungsechenzentum Stuttgat damds/bönsch Lnea System valale on local memoes BB B B,,n BB n BB wth BB B n B B B n n B B B n B n
Dect Sustuctung Dect factozaton ssemlng of the Schu complement system Solvng of the Schu complement system Solvng of the nteo unknowns Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Tansfomatons of the Ognal System - () B () - B n + + S S - () () - S B BB B B B ( ) B B S () () + B B n +S Doman Decomposton damds/bönsch Slde 0 Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -0
Schucomplement System SH B B SH B n SH BB Z B n SH B B z wth o n SH BB SH whee n B B Z z - ( ) - ( ) B o B - ( ) - ( ) B Z B z Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Dect factozaton P P Z z B 0 Z z B Z z B 0 P Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
ssemlng of the Schu complement system () P P BB B z B Z 0 BB B z B Z BB B BZ z 0 P Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat ssemlng of the Schu complement system () P P SH n BB Z B 0 SH B n B z 0 P Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Solvng of the Schu complement system P P SH B SH 0 0 P Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Solvng of the nteo unknowns P P B B 0 0 B B B B P Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Paallel Computatons on the Sudomans Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Ovelappng Doman Decomposton Ω 0 Ω 0 0 γ γ Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -
Lteatue Bay F Smth, Pette E Bjoestad, Wllam D Gopp: Doman Decomposton, Paallel Multlevel Methods fo Ellptc Patal Dffeental Equatons, Camdge Unvesty Pess,, SBN 0--- Davd E Keyes, Youcef Saad, Donald G Tuhla: Doman-Based Paallelsm and Polem Decomposton Methods n Computatonal Scence and Engneeng, SM,, SBN 0--- Doman Decomposton damds/bönsch Slde Höchstlestungsechenzentum Stuttgat Doman Decomposton Paallelzaton of Mesh Based pplcatons -