NAMD & HELIUM Enabling Work on the PRACE IBM Prototypes

Size: px

Start display at page:

Download "NAMD & HELIUM Enabling Work on the PRACE IBM Prototypes"

Jean Bridges
5 years ago
Views:

1 NAMD & HELIUM Enabling Work on the PRACE IBM Prototypes Xu Guo Applications Consultant EPCC, The University of Edinburgh

2 Outline PRACE project overview Application enabling work: NAMD HELIUM Summary 2

3 Outline PRACE project overview Application enabling EPCC: NAMD HELIUM Summary 3

4 PRACE Partnership for Advanced Computing in Europe Aims to provide the European researches with a persistent pan- European HPC service to enable world-class science, consisting of several tier-0 centres EU approved the PRACE Project Preparatory Phase (Grant: INFSO-RI ) Project duration: January 2008 December 2009 Project budget: 20 M, EC funding: 10 M Original 16 partners from 14 countries By the end of 2009, 20 countries involved in and more are interested The Implementation Phase will start from June,

5 PRACE Project Preparatory Phase Objective Perform all legal, administrative, and technical work to create a legal entity and start providing Tier-0 HPC services in 2010 PRACE project tasks in Preparatory Phase Define the legal & administrative framework (WP2) Dissemination, outreach & training (WP3) Cooperate with the European HPC ecosystem (WP2/3) Distributed computing (WP4) Prototype system assessment (WP5) Software enabling for prototype systems (WP6) Procurement strategy: Petaflop/s systems for 2009/2010 (WP7) Future Petaflop/s technologies, vendor cooperation (WP8) 5

6 PRACE WP6 in Preparatory Phase WP6: Software enabling for Petaflop/s systems Worked closely with other work packages Primary goal To identify and understand the software libraries, tools, benchmarks and skills required by users to ensure that their application can use a Petaflop/s system productively and efficiently. The largest technical activity in PRACE Preparatory Phase Most of the PRACE partners involved in 6

7 PABS PRACE Application Benchmark Suite A set of representative applications benchmarks To be used in the procurement process for Petaflop/s systems ALYA, AVBP, BSIT, Code_Saturne, CPMD, CP2K, ELMER, GADGET, GPAW, GROMACS, HELIUM, NAMD, NEMO, NS3D, OCTOPUS, PEPC, QCD, Quantum_Espresso, SPECFEM3D, TORB/ EUTERPE, TRIPOLI-4, and WRF Each application was ported to appropriate subset of prototypes To capture the applications requirements for petascale systems Scalability and optimisation strategies were investigated 7

8 Prototypes for Petaflop/s Systems in 2009/2010 IBM BlueGene/P (FZJ) 0.85 GHz, PowerPC 450, 4 way / IBM Power6 (SARA) 4.7GHz Pwr6, 32 way, SMT, IB Cray XT5 (CSC) 2.3 GHz Barcelona, 8 way 2.7 GHz Shanghai, 8 way IBM Cell/Power (BSC) NEC SX9, vector part (HLRS) Intel Nehalem/Xeon (CEA/FZJ) 2.93 GHz Nehalem, dual socket quad-core Nodes, IB

9 Outline PRACE project overview Application enabling work: NAMD BCO: Joachim Hein (EPCC, The University of Edinburgh, UK) Contributors: Martin Polak (ICA, Johannes Kepler University Linz, Austria), Paschalis Korosoglou (AUTH, Greece), Xu Guo (EPCC, The University of Edinburgh, UK) HELIUM Summary 9

10 NAMD Overview Molecular dynamics application from UIUC Emphasis on scalability Written in C++ using Charm++ parallel objects Domain decomposition Long range electro-static forces: Particle Mesh Ewald Load balance dynamically during first 300 steps Current version: NAMD 2.7 β2, released: November 2009 For this study: 2.7β1, released: March

11 Benchmark Test Cases Three test cases by: P. Coveney and Shunzhou Wan (UCL) Sizes: 1M, 2M and 9M atoms TCR-pMHC-CD complexes in a membrane environment Immune response research Basic set of 1M atoms contains four complexes Larger sets multiple copies of basic set in a box 2fs stepsize Trajectory of at least 10ns for scientific meaningful simulation (5,000,000 steps) 11

12 Resources for MPI Library During the load balance step, NAMD sends large number of unexpected message (no matching receive code uses MPI_Iprobe) Unexpected messages fill up the receive buffers Adding -env DCMF_RECFIFO= to the arguments of mpirun resolves the issues arising. Remarks: Over 10% of available memory as MPI buffer space! Similar issues on Cray XT 12

13 Memory Consumption Memory problems with 1GB/core, e.g. BG/P, Louhi NAMD 2.7β1 offers simulation with reduced memory footprint Specially build version for running, vanilla NAMD to compress input files Essential for 9M atom benchmark and running 1M on 4 cores/node on BG/P 2M can only run with 2 cores/node on BG/P Table for 2M atom system (Cray XT5) can not run on BG/P (4 task/node) 13

14 Version Performance & SMT: Huygens SMT gives vast performance improvement (seen on HPCx before, but Pwr6 is more dramatic) 14

15 Feasibility of A Simulation on PF system On a 20TF partition a 9M atom simulation is feasible (walltime of about a week)! 15

16 Compiler optimisations: Huygens Investigated for 2 million atom benchmark Only modest sensibility to compiler optimisations 16

17 Build/Runtime optimisations on BG/P Tested for 1 million atoms on 2048 cores (512 nodes) Change of task map from default to TXYZ Slightly detrimental: No 1 vs No2 Charm offers BG/P build with and without MPI Without MPI slightly faster: No 1 vs No 3 Disengaging DFPU for charm++ build detrimental but not much impact for NAMD build Compiler option O4 and higher break the code 17

18 Build/Runtime optimisation on BG/P: results No Task mapping MPI used Charm++ compiler options NAMD compiler options Bench-mark-time 1 default o -O3 -qstrict -qhot -O3 -Q -qhot s -qarch=450d -DCMK_OPTIMIZE=1 -qarch=450d -qtune=450 -DFFTW_ENABLE_FLOAT 2 TXYZ no -O3 -qstrict -qhot -qarch=450d -DCMK_OPTIMIZE=1 -O3 -Q -qhot -qarch=450d -qtune= s 3 default yes -O3 -qstrict -qhot -qarch=450d -DCMK_OPTIMIZE=1 4 default no -O3 -qhot -qstrict -qarch=450d -qtune=450 -DCMK_OPTIMIZE 5 default no -O3 -qhot -qstrict -qarch=450 -qtune=450 -DCMK_OPTIMIZE 6 default no -O5 -qarch=450 -qtune=450 -qipa=level=2 -DCMK_OPTIMIZE -DFFTW_ENABLE_FLOAT -O3 -Q -qhot -qarch=450d -qtune=450 -DFFTW_ENABLE_FLOAT -O3 -Q -qhot -qstrict -qarch=450 -qtune=450 - DFFTW_ENABLE_FLOAT -O3 -Q -qhot -qarch=450 -qtune=450 - DFFTW_ENABLE_FLOAT -O3 -Q -qhot -qarch=450d -qtune=450 -DFFTW_ENABLE_FLOAT s s s s 18

19 Performance Comparison M1 Benchmark 19

20 Performance Comparison M9 Benchmark Well rounded overall picture, apart from x86 Cluster (Nehalem/IB) Machine new at the time of the test (Broken? Untuned?) Memory/task is still an issue (no BGP for M9, no 40TF point for Louhi) 20

21 Outline PRACE project overview Application enabling work: NAMD HELIUM Summary BCO: Xu Guo (EPCC, The University of Edinburgh, UK) Contributor: Andrew Sunderland (STFC Daresbury Laboratory, UK) 21

22 HELIUM Overview Uses time-dependent solutions of the full-dimensional Schrödinger equation to simulate the behavior of helium atoms Developed by Queen's University Belfast Has access restrictions 22

23 HELIUM Test Cases A single Fortran 90 file with more than lines Using MPI parallelism Total work in the whole grid space is divided into Blocks Each processor works on one block Multiple decomposition approaches for a fixed total problem size by changing block size and block number Particular parameters in source code Control the simulation conditions and execution behaviour E.g. the total problem size, block size, block count, required core number, I/O frequency, etc. Will also affect the memory requirement Test cases used for PRACE Fixed problem size: 1540 grid units, 3060 grid units Total time steps: 80 Output frequency: once every 20 time steps 23

24 Porting HELIUM to IBM Power 6 Huygens Compiler IBM XL MPI Fortran Proper core numbers for execution Depended on the total problem size and block number Not all the core numbers can be selected for benchmarking Wrong core number will lead to execution failure Tasks allocation on nodes High possibility of not fully populated task allocation Important to define the total tasks number and the total nodes number clearly in the job script 24

25 Porting HELIUM to IBM Bluegene/P JUGENE Compiler IBM XL MPI Fortran Virtual node (VN) mode used More effective to utilises all four cores for computation on each node DUAL mode (2 tasks per node) and SMP mode (1 task per node) have access to greater, or even exclusive, shares of the memory hierarchy per node stack_limit = 200MB 25

26 Compiler Optimisations on IBM Power 6 Huygens Higher optimisation level without breaking the code -O3 -O4 -O5 will break the code Performance tuning -qarch=auto, qtune=auto, -qhot Link with essl library -qessl Test case size of 1540 grid units Cores -qfree=f90 -O3 -qfree=f90 -O4 -qessl -qarch=auto -qtune=auto -qhot s 714 s s 319 s 26

27 Compiler Optimisations on IBM BlueGene/P JUGENE Link with essl library Higher optimisation level without breaking the code -O3 -O4 Double hummer on produces a significant speed-up when compiled at O4 No. Flags Time 1 No flag s 2 -O s 3 -O s 4 -O s 5 -O3 qtune=450 qarch= s 6 -O3 qtune=450d qarch=450d s 7 -O s 8 -O4 qtune=450 qarch= s 9 -O4 qtune=450d qarch=450d s 27

Reducing MPI Cost Removing unnecessary MPI debugging steps at the initialisation stage The modifications has been merged into the latest version HELIUM by the developer Make the test case size of

28 Reducing MPI Cost Removing unnecessary MPI debugging steps at the initialisation stage The modifications has been merged into the latest version HELIUM by the developer Make the test case size of 3060 grid units scaled up to more than 20,000 cores on IBM BG/P JUGENE Test case size of 1540 grid units on IBM Power 6 Huygens Performance on Huygens Cores Before After s 712 s s 309 s MPI tracer profiling on Huygens (1540 cores) Point to point Collective Sync Before 10.82% 1.2% 23.56% After 7.22% 0.26% 23.83% 28

29 Manual Code Optimisations on IBM Power 6 Huygens Merge loops with the same boundaries together Expensive routines with multiple calculation loops Loop through in a proper order Introducing temporary variables Performance on Huygens Representing intermediate results Cores Before After Reduce the floating point operations s 670 s Test case size of 1540 grid units s 298 s Hardware Counter profiling on Huygens Cores L1 Cache misses L2 Cache misses Before After Before After E E E E E E E E+11 29

30 HELIUM Performance on IBM Power 6 Huygens Test case size of 1540 grid units 30

31 HELIUM Scaling Cost on IBM Power 6 Huygens Scaling Cost = Execution time * Core number A horizontal cost curve implies a linear scaling 31

32 HELIUM Strong Scaling Performance on BG/P Test cases size of 1540 grid units and 3060 grid units 32

33 HELIUM Strong Scaling Cost on BG/P Test cases size of 1540 grid units and 3060 grid units 33

34 HELIUM Weak Scaling Performance on BG/P Local block size fixed to 20 grid units 34

35 Comparison on PRACE Prototypes Test case of fixed problem size 1540 grid units 35

to PRACE prototypes Including IBM Power 6 and IBM BlueGene/P Improved the performance and scalability Both applications fit for the

36 Summary PRACE In 2008/2009, The progress of the preparatory phase project is satisfactory in all areas Project review 28/10/2009, Brussels PRACE is ready to start the Implementation Phase in 2010 Application enabling work NAMD & HELIUM Latest versions were ported to PRACE prototypes Including IBM Power 6 and IBM BlueGene/P Improved the performance and scalability Both applications fit for the future petaflop systems of the forth coming phase We are looking forward to continuing the contributions in the PRACE Implementation Phase 36

Using DDT. Debugging programs with DDT. Peter Towers. HPC Systems Section. ECMWF January 28, 2016

Using DDT. Debugging programs with DDT. Peter Towers. HPC Systems Section. ECMWF January 28, 2016 Using DDT Debugging programs with DDT Peter Towers HPC Systems Section Peter.Towers@ecmwf.int ECMWF January 28, 2016 HPCF - Debugging programs with DDT ECMWF 1 2016 Allinea DDT DDT is a very popular interactive