Designing A Low-Latency Cuckoo Hash Table for Write-Intensive Workloads Using RDMA

Transcription

1 Designing A Low-Latency Cuckoo Hash for -Intensive Workloads Using RDMA Tyler Szepesi, Bernard Wong, Ben Cassell, Tim Brecht Cheriton School of Computer Science University of Waterloo April, 0

2 In-Memory Distributed Systems

3 TCP Data Transfer Application Application Kernel Kernel NIC NIC

4 RDMA Data Transfer Application Application Kernel Kernel NIC NIC

5 Current State of the Art: Pilaf/FaRM Client Server Client Application Client Shared Data

6 Current State of the Art: Pilaf/FaRM Client Server Client Application Client Shared Data

7 One-Sided RDMA for Reads and s Client Server Client Application Client Shared Data

8 Nessie Cuckoo hash table where clients manage both reads and writes Design: One-sided RDMA for reads and writes of key-value pairs Checksum to detect corrupted reads RDMA compare-and-swap to create atomic writes Lock-free to eliminate the possibility of deadlock Advantage: Full benefits from one-sided RDMA Tradeoffs compared to the state of the art: More roundtrips per operation Added complexity

9 Cuckoo Hash Key Value 0

10 Cuckoo Hash Key: Dan Value: Bacon Dan Primary Hash Secondary Hash Key Value 0 0

11 Cuckoo Hash Key Value Key: Dan Value: Bacon Dan Bacon Primary Hash Dan Secondary Hash 0

12 Cuckoo Hash Key: Ben Value: Apple Ben Primary Hash Secondary Hash 0 Key Dan Value Bacon 0

13 Cuckoo Hash Key: Ben Value: Apple Ben Primary Hash Secondary Hash 0 Key Dan Ben Value Bacon Apple 0

14 Cuckoo Hash Key: Tim Value: Eggs Tim Primary Hash Secondary Hash 0 Key Dan Ben Value Bacon Apple 0

15 Cuckoo Hash Key: Tim Value: Eggs Tim Primary Hash Secondary Hash 0 Key Dan Ben Value Bacon Apple 0

16 Cuckoo Hash Key: Tim Value: Eggs Tim Primary Hash Secondary Hash 0 Key Dan Tim Dan Ben Value Bacon Eggs Bacon Apple 0

17 Nessie

18 Nessie V# Bits

19 Nessie V# Bits

20 Nessie V# Bits C 0 N Bytes 0

21 Nessie V# Bits C 0 N Bytes Client Client Client

22 Nessie 0

23 Nessie: Key: Tim Value: Eggs 0 0

24 Nessie: Data Key: Tim Value: Eggs 0 0

25 Nessie: Read Key: Tim Value: Eggs Tim Primary Secondary 0 0

26 Nessie: Read Data at Primary Key: Tim Value: Eggs Tim Primary Secondary 0 0

27 Nessie: Read Data at Secondary Key: Tim Value: Eggs Tim Primary Secondary 0 0

28 Nessie: Update Key: Tim Value: Eggs Tim Primary Secondary 0 0

29 Nessie: Make Data Valid Key: Tim Value: Eggs Tim Primary Secondary 0 0

30 Nessie: with Conflict Client Nessie Client Key: Tim Value: Eggs 0 0 0

31 Nessie: with Conflict Client Nessie Client Data Key: Tim Value: Eggs 0 0

32 Nessie: with Conflict Client Nessie Client Data Key: Tim Value: Eggs Read Indices 0 0

33 Nessie: with Conflict Client Nessie Client Data Key: Tim Value: Eggs Read Indices Read Primary 0 0

34 Nessie: with Conflict Client Nessie Client Data Key: Tim Value: Eggs Read Indices Read Primary Read Secondary 0 0

35 Nessie: with Conflict Client Nessie Client Data Key: Tim Value: Eggs Read Indices Read Primary Read Secondary Delete Tim 0 0 0

36 Nessie: with Conflict Client Nessie Client Data Key: Tim Value: Eggs Read Indices Read Primary Read Secondary Delete Tim Ken 0 0 0

37 Nessie: with Conflict Client Nessie Client Data Key: Tim Value: Eggs Read Indices Read Primary Read Secondary Delete Tim Ken 0 0 0

38 Nessie: with Conflict Client Nessie Client Data Key: Tim Value: Eggs Read Indices Read Primary Read Secondary Make Valid Delete Tim Ken 0 0 0

39 Nessie: with Conflict Key: Problem: Tim Value: Any Eggs client can modify the index table part 0 way through a write operation. 0 Client Nessie Client Data Read Indices Read Primary Read Secondary Make Valid Delete Tim Ken 0

40 Nessie: with Conflict Key: Problem: Tim Value: Any Eggs client can modify the index table part 0 way through a write operation. Solution: ) Abort the operation 0as soon as a conflict is detected. ) Retry the operation. Client Nessie Client Data Read Indices Read Primary Read Secondary Make Valid Delete Tim Ken 0 0

41 Nessie: Handling Conflicts Client Nessie Client Data Key: Tim Value: Eggs Read Indices Read Primary Read Secondary Delete Tim Ken 0 0 0

42 Nessie: Handling Conflicts Client Nessie Client Data Key: Tim Value: Eggs Read Indices Read Primary Read Secondary CAS FAIL expected Delete Tim Ken 0 0 0

43 Nessie: Handling Conflicts Client Nessie Client Data Key: Tim Value: Eggs Read Indices Read Primary Read Secondary CAS FAIL expected Abort Delete Tim Ken 0 0 0

44 Nessie: Migration Migrate Key: Dan Destination Source 0 0

45 Nessie: Migration Client Nessie Client Migrate Key: Dan Destination Source 0 0

46 Nessie: Migration Client Nessie Client Copy Data Migrate Key: Dan Destination Source 0 0

47 Nessie: Migration Client Nessie Client Copy Data Migrate Key: Dan Destination Source Dan 0 0

48 Nessie: Migration Client Nessie Client Copy Data Migrate Key: Dan Update Dest. Destination Source Dan 0 0

49 Nessie: Migration Client Nessie Client Copy Data Migrate Key: Dan Update Dest. Destination CAS Source FAIL expected Dan 0 0

50 Nessie: Migration Client Nessie Client Copy Data Migrate Key: Dan Update Dest. Destination CAS Source FAIL expected Abort Dan 0 0 0

51 Performance Tradeoffs Expected number of roundtrips for a write operation in Nessie: Load Factor Roundtrips

52 Nessie s Design Complexity Contention over the index table creates complexity Our solution mirrors that of Hardware Transactional Memory Opimistically attempt operation Check for conflicting operations Abort and retry if a conflict is detected NIC support of HTM would greatly improve Nessie NIC handles detection and reporting of conflicts Fewer roundtrips

53 Hardware Transactional Memory XBEGIN Read to... to XEND

54 HTM + RDMA Client XBEGIN XBEGIN Read to... to XEND Read... XEND

55 Summary Low latency is critical for in-memory distributed systems Nessie: Exclusively RDMA reads, writes, and CAS operations Achieves low-latency Shared lock-free data HTM in conjunction with RDMA can greatly simplify Nessie We are excited to build Nessie and explore HTM