Protecting shared storage from rogue jobs by I/O profiling and dynamic load balancing

Size: px

Start display at page:

Download "Protecting shared storage from rogue jobs by I/O profiling and dynamic load balancing"

Austen West
5 years ago
Views:

from rogue jobs by I/O profiling and dynamic load

1 Ellexus: The I/O Profiling Company Dr Rosemary Francis, CEO and Co-founder Protecting shared storage from rogue jobs by I/O profiling and dynamic load balancing The I/O Profiling Company - Protect. Balance. Optimise.

2 Who we are Ellexus is the I/O profiling company. The company was founded to solve I/O and dependency problems for semiconductor design flows, but we now work across the HPC space in life sciences, cancer research, automotive, oil and gas etc. Our new tool Mistral has been developed with ARM to protect shared storage from bad data access patterns on large compute clusters.

3 Our products Ellexus Mistral Lightweight I/O profiling Hotspot detection Whole cluster monitoring Protects storage from rogue jobs Ellexus Breeze Profile 1 job or application Detailed I/O profiling Dependency analysis

4 Mistral: Solving the noisy neighbour problem In a compute cluster it is possible for a small number of jobs to overload the shared file system. Example of a rogue job from ARM: Lots of data is written to remote storage (/scratch). But almost nothing is written to local storage (/tmp).

5 Mistral: Protecting you against rogue jobs Hotspots in your cluster can be caused by different types of I/O: Too much data A user accidentally submitted 1,000 jobs with the debug flag left on. Millions of meta data operations One user wrote: for ( ) { open() read() close() } This crashed the meta data server. Too many small files One user accidentally created a million zerolength files.

6 Mistral Protect your storage from bad I/O patterns and rogue jobs. Load Balance your storage for good quality of service. Optimise your applications for the storage and your infrastructure for your jobs. Mistral: Presentation wide-scale Subject, I/O followed profiling by and the load Name balancing and date for shared storage

7 Mistral: How does it work? Mistral Mistral Mistral Mistral Job Job Job Job If a job carries out too much I/O Mistral will generate an alert. Job 42 exceeded 500MB/s to /share/scratch Storage

8 Mistral: Advanced systems management Use I/O data from Mistral to: Identify I/O hotspots and rogue jobs Locate bad I/O patterns Educate users I/O Jobs

9 Mistral: Advanced systems management Use I/O data from Mistral to: Identify I/O hotspots and rogue jobs Locate bad I/O patterns Educate users I/O Jobs Use Mistral I/O throttling controls to: Improve quality of service Load balance shared storage I/O Jobs

10 I/O contacts The configuration files are called I/O contracts. Mistral Job I/O contracts can be changed at run time. You can have different contracts for different jobs, users or projects Monitoring contract Throttling contract

11 Example contacts Example contract: monitoringtimeframe,60s NAME PATH/MOUNT POINT IO TYPE LIMIT SIZE M1, /work/, read, 500MB ALL M2, /work/, read, 50MB 0-2KB M3, /home/, open, 2000 Example alert: TIME RULE VIOLATION PID PROGRAM FILE JOB ID 17:30:05 17:30:05 17:30:06 M1 M2 M3 643MB/60s, 129MB/60s, 7405/60s, 2784, 2784, 2784, /tools/bin, /tools/bin, /tools/bin, /work/tools/config, /work/data/file01, /home/data/set05,

12 Use case: Protection from rogue jobs Problem Stop jobs from overloading the system, but use all the available bandwidth. Solution In the I/O contract: High monitoring limits - monitor jobs with high I/O Very high throttling limits - slow rogue jobs with very high I/O I/O Very high throttling limits High throttling limits

13 Dynamic load balancing \\ Update contracts System load IO load increases I/O load decreases When a problem arising increase monitoring to see more information or start throttling to solve the issue.

14 Dynamic load balancing Build automatic rules that react to common problems Customise the actions based on your unique I/O patterns Target specific jobs, projects or nodes. Mistral \\ Job Mistral analytics Scheduler analytics Storage analytics System knowledge Mistral Job Mistral Job

15 Use case: Healthcheck your applications Some jobs won't crash the cluster, but bad I/O patterns can harm the quality of service at scale. Solution Healthcheck new work flows and problem jobs in more detail with Mistral. Mistral Job Check for: Small reads and writes Peaks in I/O bandwidth Creation or deletion of small files Meta data operations: open(), stat() Random access patterns Make every user an I/O expert

16 Mistral Protect your storage by monitoring the whole cluster for rogue jobs. Prevent bad I/O patterns by running health checks on new work flows. Load balance your storage by installing bandwidth limits and improving quality of service. Use storage analytics to adjust the load balancing on a live cluster. Optimise applications for shared storage and optimise the compute and storage infrastructure for the jobs. Mistral: Presentation wide-scale Subject, I/O followed profiling by and the load Name balancing and date for shared storage

17 Thank you Please get in touch for more information Dr Rosemary Francis CEO and Co-founder Ellexus Ltd St John's Innovation Centre, Cowley Road, Cambridge CB4 0WS, UK Mistral: Presentation wide-scale Subject, I/O followed profiling by and the load Name balancing and date for shared storage

Delta Compressed and Deduplicated Storage Using Stream-Informed Locality

Delta Compressed and Deduplicated Storage Using Stream-Informed Locality Philip Shilane, Grant Wallace, Mark Huang, & Windsor Hsu Backup Recovery Systems Division EMC Corporation Motivation and Approach