A history-based estimation for LHCb job requirements

Transcription

1 A history-based estimation for LHCb job requirements Nathalie Rauschmayr on behalf of LHCb Computing 13th April 2015

2 Introduction N. Rauschmayr 2 How long will a job run and how much memory might it need? Production Manager Underestimation: Job is killed. We lose the whole job! Overestimation: What to do with the remaining time

3 Introduction N. Rauschmayr 3 Well studied problem in High Performance Computing Some recent studies in HEP: CMS, WLCG multicore task force Multicore jobs: good runtime estimates allow better scheduling

4 Introduction N. Rauschmayr 3 Well studied problem in High Performance Computing Some recent studies in HEP: CMS, WLCG multicore task force Multicore jobs: good runtime estimates allow better scheduling

5 Introduction N. Rauschmayr 4 A lot of job meta data from past jobs: LHCb bookkeeping

6 Introduction N. Rauschmayr 4 A lot of job meta data from past jobs: LHCb bookkeeping Hardware specific: CPU model HS06 Cache size RAM Size

7 Introduction N. Rauschmayr 4 A lot of job meta data from past jobs: LHCb bookkeeping Hardware specific: CPU model HS06 Cache size RAM Size Input File Size Number of events Run number (LHCb RunDB) Trigger Configuration Key Avg. Multiplicity Avg. Luminosity...

8 Introduction N. Rauschmayr 4 A lot of job meta data from past jobs: LHCb bookkeeping Hardware specific: CPU model Job Start/End time HS06 Site Cache size Worker Node RAM Size Memory Footprint Input File Runtime Size Number of events Run number (LHCb RunDB) Trigger Configuration Key Avg. Multiplicity Avg. Luminosity...

9 Introduction N. Rauschmayr 4 A lot of job meta data from past jobs: LHCb bookkeeping Hardware specific: CPU model Job Start/End time HS06 Site Cache size Worker Node RAM Size Memory Footprint Input File Runtime Size Number of events Run number (LHCb RunDB) Trigger Configuration Key Avg. Multiplicity Avg. Luminosity...

10 Our Proposal N. Rauschmayr 5 Automate the prediction procedure based on prior jobs and given job meta data Supervised Learning Reduce false estimates Simplify Production Manager's life

11 Our Proposal N. Rauschmayr 6

12 Workload Analysis - Reconstruction Normalized CPU time per event: CPUTime HEPSPECValue NumberOfEvents Approximate a Gaussian distribution: Reprocessing ,000 Frequency 4,000 2, CPU-work per event in HS06.s

13 Workload Analysis - Reconstruction N. Rauschmayr 7 Normalized CPU time per event: CPUTime HEPSPECValue NumberOfEvents Approximate a Gaussian distribution: Reprocessing 2011 Reprocessing 2011 Maximum Likelihood HS06.s Frequency 6,000 4,000 2,000 Frequency 6,000 4,000 2, CPU-work per event in HS06.s 0 -sigma +sigma CPU-work per event in HS06.s

14 Workload Analysis - Reconstruction Comparison of 2011 and 2012 workloads: Reprocessing Reprocessing Maximum Likelihood HS06.s Frequency Maximum Likelihood HS06.s Frequency sigma +sigma CPU-work per event in HS06.s 0 -sigma +sigma CPU-work per event in HS06.s N. Rauschmayr 8

15 Workload Analysis - Stripping N. Rauschmayr 9 Stripping: Sort reconstructed events into different output streams Reprocessing Reprocessing 2012 Maximum Likelihood 5.87 HS06.s Frequency 1 Maximum Likelihood 5.26 HS06.s 0.5 Frequency sigma +sigma 0 +sigma +sigma CPU-work per event in HS06.s CPU-work per event in HS06.s

16 Detect Most Important Features N. Rauschmayr 10 Certain well known correlations: Beam energy versus event size

17 Detect Most Important Features N. Rauschmayr 10 Certain well known correlations: Beam energy versus event size Pileup versus complexity of reconstruction Example: Reconstruction (2011 versus 2012)

18 Detect Most Important Features N. Rauschmayr 11 Avoid overfitting...

19 Detect Most Important Features N. Rauschmayr 11 Avoid overfitting... Linear regression: runtime per event = Θ 0 + Θ 1 x 1 + Θ 2 x 2 + Θ 3 x 3 + Θ 4 x 4 + Θ 5 x 5 + Θ 6 x 6

20 Detect Most Important Features Avoid overfitting... Linear regression: runtime per event = Θ 0 + Θ 1 x 1 + Θ 2 x 2 + Θ 3 x 3 + Θ 4 x 4 + Θ 5 x 5 + Θ 6 x 6 Normalize File Size Avg. Event Size HEPSPEC Number Of Events Avg. Luminosity Avg. Multiplicity

21 Detect Most Important Features Avoid overfitting... Linear regression: runtime per event = Θ 0 + Θ 1 x 1 + Θ 2 x 2 + Θ 3 x 3 + Θ 4 x 4 + Θ 5 x 5 + Θ 6 x 6 Normalize Find best Θ File Size Avg. Event Size HEPSPEC Number Of Events Avg. Luminosity Avg. Multiplicity

22 Detect Most Important Features Avoid overfitting... Linear regression: runtime per event = Θ 0 + Θ 1 x 1 + Θ 2 x 2 + Θ 3 x 3 + Θ 4 x 4 + Θ 5 x 5 + Θ 6 x 6 Normalize Find best Θ Remove small Θ File Size Avg. Event Size HEPSPEC Number Of Events Avg. Luminosity Avg. Multiplicity x x

23 Detect Most Important Features N. Rauschmayr 11 Avoid overfitting... Linear regression: runtime per event = Θ 0 + Θ 1 x 1 + Θ 2 x 2 + Θ 3 x 3 + Θ 4 x 4 + Θ 5 x 5 + Θ 6 x 6 Normalize Find best Θ Remove small Θ Evaluate RMSE File Size Avg. Event Size HEPSPEC Number Of Events Avg. Luminosity Avg. Multiplicity x x % better than naive estimator like MLE

24 Detect Most Important Features N. Rauschmayr 12 Stripping Jobs: Normalize Find best Θ Remove small Θ Evaluate RMSE File Size Avg. Event Size HEPSPEC Number Of Events Avg. Luminosity Avg. Multiplicity x x % better than naive estimator like MLE

25 Supervised Learning N. Rauschmayr 13 Supervised learning: there exist some labelled training data What if only little amount of training data available?

26 Supervised Learning N. Rauschmayr Find similar jobs which have already run 2. Predict requirements for the next k jobs using either (MLE/LR) 3. When k jobs have finished, update prediction formula with the new results obtained 4. Repeat step 2 and 3 until all jobs have finished k Jobs Training Data k Jobs New Test Data

27 Supervised Learning N. Rauschmayr Find similar jobs which have already run 2. Predict requirements for the next k jobs using either (MLE/LR) 3. When k jobs have finished, update prediction formula with the new results obtained 4. Repeat step 2 and 3 until all jobs have finished Error Accumulated Error k Jobs + k Jobs Training Data k Jobs New Test Data

28 Supervised Learning N. Rauschmayr Find similar jobs which have already run 2. Predict requirements for the next k jobs using either (MLE/LR) 3. When k jobs have finished, update prediction formula with the new results obtained 4. Repeat step 2 and 3 until all jobs have finished Error Accumulated Error k Jobs + k Jobs + k Jobs Training Data k Jobs New Test Data

29 Supervised Learning N. Rauschmayr Find similar jobs which have already run 2. Predict requirements for the next k jobs using either (MLE/LR) 3. When k jobs have finished, update prediction formula with the new results obtained 4. Repeat step 2 and 3 until all jobs have finished Accumulated Error Error k Jobs + k Jobs + k Jobs + k Jobs Training Data k Jobs New Test Data

30 Supervised Learning Root Mean Squared Error: n i=0 (difference i) 2 n, where difference is predicted minus real values Update after 100 jobs Update after 100 jobs Accumulated RMSE MLE LR RMSE for each k Job Number 10 5 Job Number 10 5 N. Rauschmayr 15

31 Conclusion N. Rauschmayr 16 Historical data can help us to predict future jobs Certain meta data are strongly correlated with runtime Improved prediction up to 25% Both models (naive estimator and linear regression) can be easily implemented in Production and support the Production Manager in his work

32 N. Rauschmayr 17 Questions

33 Backup Slides N. Rauschmayr 18 Trigger configuration changes over different runs

34 Backup Slides N. Rauschmayr 19 Stripping Jobs: Memory and Runtime per Event Accumulated Error Update after 100 jobs MLE LR Job Number 10 5 Accumulated Error Update after 100 jobs MLE LR Job Number 10 5

35 Backup Slides Reconstruction: Memory Footprint Reprocessing 2011 Reprocessing Maximum Likelihood 1.76 GB 2 2 Frequency Maximum Likelihood 1.6 GB Frequency sigma +sigma 0 -sigma+sigma Memory in kb 10 6 Memory in kb 10 6 N. Rauschmayr 20

36 Backup Slides Stripping: Memory Footprint Change from POOL to ROOT Reprocessing Reprocessing Maximum Likelihood 3.4 GB Frequency 1.5 Maximum Likelihood 2.2 GB Frequency sigma +sigma 0 -sigma+sigma Memory in kb 10 6 Memory in kb 10 6 N. Rauschmayr 21