A 28nm SoC with a 1.2GHz 568nJ/ Prediction Sparse Deep-Neural-Network Engine with >0.1 Timing Error Rate Tolerance for IoT Applications

Transcription

1 A 28nm SoC with a 1.2GHz 568nJ/ Prediction Sparse Deep-Neural-Network Engine with >0.1 Timing Error Rate Tolerance for IoT Applications Paul Whatmough, S. K. Lee, H. Lee, S. Rama, D. Brooks, G.-Y. Wei Harvard University, Cambridge, MA 1 of 86

2 Motivation 2 of 86

3 Motivation Deep Neural Networks (DNNs) A universal classifier with many diverse applica5ons Interpret noisy real- world data from sensor- rich systems 3 of 86

4 Motivation Deep Neural Networks (DNNs) A universal classifier with many diverse applica5ons Interpret noisy real- world data from sensor- rich systems Large memory footprint and compute load Perform DNN inference on device in micro- Joules 4 of 86

5 Inputs Motivation DNN ENGINE Outputs Sensor Data + DNN Topology + Weights Input Vector Weight Neuron Hidden Layers Output Classes Argmax Classification DNN ENGINE A Shared Programmable Classifier Specialized Architecture and Efficient Digital Circuits 5 of 86

6 Inputs Motivation DNN ENGINE Outputs Sensor Data + DNN Topology + Weights Input Vector Weight Neuron Hidden Layers Output Classes Argmax Classification DNN ENGINE A Shared Programmable Classifier Specialized Architecture and Efficient Digital Circuits Parallelism: Balanced memory bandwidth and throughput 6 of 86

7 Inputs Motivation DNN ENGINE Outputs Sensor Data + DNN Topology + Weights Input Vector Weight Neuron Hidden Layers Output Classes Argmax Classification DNN ENGINE A Shared Programmable Classifier Specialized Architecture and Efficient Digital Circuits Parallelism: Balanced memory bandwidth and throughput Sparsity: HW support for sparse data and small datatypes 7 of 86

8 Inputs Motivation DNN ENGINE Outputs Sensor Data + DNN Topology + Weights Input Vector Weight Neuron Hidden Layers Output Classes Argmax Classification DNN ENGINE A Shared Programmable Classifier Specialized Architecture and Efficient Digital Circuits Parallelism: Balanced memory bandwidth and throughput Sparsity: HW support for sparse data and small datatypes Resilience: Razor adapta5on to minimize PVT margins 8 of 86

9 Outline Mo5va5on DNN Engine Parallelism Sparsity Resilience Measurement Results Summary 9 of 86

10 Fully-Connected DNN Graph Input Vector Output Classes Input Layer Output Layer Hidden Layers 10 of 86

11 Fully-Connected DNN Graph Weight Neuron Input Vector Output Classes 11 of 86

12 Fully-Connected DNN Graph SIMD Window Input Vector Output Classes 12 of 86

13 Fully-Connected DNN Graph Weights Input Vector Output Classes 13 of 86

14 Fully-Connected DNN Graph Input Vector Output Classes 14 of 86

15 Fully-Connected DNN Graph Input Vector Output Classes 15 of 86

16 Fully-Connected DNN Graph Input Vector Output Classes 16 of 86

17 Fully-Connected DNN Graph Input Vector Output Classes 17 of 86

18 Fully-Connected DNN Graph Input Vector Output Classes 18 of 86

19 Fully-Connected DNN Graph Input Vector Output Classes 19 of 86

20 Fully-Connected DNN Graph Input Vector Output Classes 20 of 86

21 Fully-Connected DNN Graph Input Vector Output Classes 21 of 86

22 Balancing Efficiency and Bandwidth Data"Reads"(Norm.)" 1" 0.8" 0.6" 0.4" 0.2" 0" No Data Reuse 1" 4" 8" 16" 32" SIMD"Width"(Words)" Parallelism increases throughput and data reuse 30" 20" 10" 0" Weight"Reads"/"Cycle" 22 of 86

23 Balancing Efficiency and Bandwidth Data"Reads"(Norm.)" 1" 0.8" 0.6" 0.4" 0.2" 0" No Data Reuse Bandwidth too high 1" 4" 8" 16" 32" SIMD"Width"(Words)" Parallelism increases throughput and data reuse But also increases Memory Bandwidth demands 30" 20" 10" 0" Weight"Reads"/"Cycle" 23 of 86

24 Balancing Efficiency and Bandwidth Data"Reads"(Norm.)" 1" 0.8" 0.6" 0.4" 0.2" 0" No Data Reuse 1" 4" 8" 16" 32" SIMD"Width"(Words)" Parallelism increases throughput and data reuse But also increases Memory Bandwidth demands 128b/cycle Bandwidth too high 8- Way SIMD is efficient at reasonable memory BW 10x Ac5va5on reuse, with 128b AXI channel 30" 20" 10" 0" Weight"Reads"/"Cycle" 24 of 86

25 DNN ENGINE Micro-Architecture Host Processor Data In CFG Data Out Data Read IPBUF XBUF Writeback NBUF W- MEM DMA Weight Load MAC Datapath Activation 25 of 86

26 DNN ENGINE Micro-Architecture Host Processor Data In CFG Data Out Data Read IPBUF XBUF Writeback NBUF W- MEM DMA Weight Load MAC Datapath Activation Host Processor loads configura5on and input data 26 of 86

27 DNN ENGINE Micro-Architecture Host Processor Data In CFG Data Out Data Read IPBUF XBUF Writeback NBUF W- MEM DMA Weight Load MAC Datapath Activation Accumulate products of Ac5va5on and Weights 27 of 86

28 DNN ENGINE Micro-Architecture Host Processor Data In CFG Data Out Data Read IPBUF XBUF Writeback NBUF W- MEM DMA Weight Load MAC Datapath Activation Add bias, apply ReLU ac5va5on and writeback 28 of 86

29 DNN ENGINE Micro-Architecture Host Processor Data In CFG Data Out Data Read IPBUF XBUF Writeback NBUF W- MEM DMA Weight Load MAC Datapath Activation IRQ to host, which retrieves predic5on data 29 of 86

30 Outline Mo5va5on DNN Engine Parallelism Sparsity Resilience Measurement Results Summary 30 of 86

31 Exploiting Sparse Data in DNNs Input Vector Output Classes 31 of 86

32 Exploiting Sparse Data in DNNs Input Vector Output Classes 32 of 86

33 Exploiting Sparse Data in DNNs Input Vector Output Classes 33 of 86

34 Exploiting Sparse Data in DNNs Input Vector Output Classes 34 of 86

35 Exploiting Sparse Data in DNNs Input Vector Output Classes 35 of 86

36 Exploiting Sparse Data in DNNs Input Vector Output Classes 36 of 86

37 Exploiting Sparse Data in DNNs Input Vector Output Classes 37 of 86

38 Exploiting Sparse Data in DNNs Input Vector Output Classes 38 of 86

39 Exploiting Sparse Data in DNNs Input Vector Output Classes 39 of 86

40 Exploiting Sparse Data in DNNs Input Vector Output Classes 40 of 86

41 Exploiting Sparse Data in DNNs Input Vector Output Classes 41 of 86

42 Exploiting Sparse Data in DNNs Input Vector Output Classes 42 of 86

43 DNN ENGINE Micro-Architecture Host Processor Data In CFG Data Out Data Read IPBUF XBUF Writeback NBUF W- MEM DMA Weight Load MAC Datapath Activation 43 of 86

44 DNN ENGINE Micro-Architecture Host Processor Data In CFG Data Out Data Read IPBUF XBUF Writeback NBUF W- MEM SKIP Index DMA Weight Load MAC Datapath Activation 44 of 86

45 Outline Mo5va5on DNN Engine Parallelism Sparsity Resilience Measurement Results Summary 45 of 86

46 Razor Hardware Accelerators CLK process safety voltage temp coupling jizer ageing 46 of 86

47 Razor Hardware Accelerators CLK process safety voltage temp coupling jizer ageing Occasionally allow real cri5cal paths to fail 5ming Explicitly check for late- arriving signals Adapt Vdd/Fclk to track near- zero error- rate 47 of 86

48 Razor Hardware Accelerators CLK process safety voltage temp coupling jizer ageing Occasionally allow real cri5cal paths to fail 5ming Explicitly check for late- arriving signals Adapt Vdd/Fclk to track near- zero error- rate Must ensure 5ming errors do not affect system Roll- back and replay [Das et al., CICC 13] Algorithmic error tolerance [Whatmough et al., ISSCC 13] 48 of 86

49 DNN Engine Micro-Architecture Host Processor Data In CFG Data Out Data Read IPBUF XBUF RZ Writeback NBUF W- MEM RZ SKIP Index DMA Weight Load MAC Datapath Activation Two Razor stages: W Memory and MAC Datapath 49 of 86

50 Memory: Algorithm-Level Tolerance Count%(a.u.)% Two s Compliment (TC) -1: : !0.3%!0.2%!0.1% 0.0% 0.1% 0.2% 0.3% Weight%Value% Weights are zero- mean Gaussians (regulariza5on) Small magnitude, random sign (worst- case in TC) 50 of 86

51 Memory: Algorithm-Level Tolerance Count%(a.u.)%!0.3%!0.2%!0.1% 0.0% 0.1% 0.2% 0.3% Weight%Value% Two s Compliment (TC) -1: : Sign- Magnitude (SM) -1: : Weights are zero- mean Gaussians (regulariza5on) Small magnitude, random sign (worst- case in TC) Sign- Magnitude (SM) reduces switching in weights Lowers switched cap (power) and bit error rate 51 of 86

52 Datapath: Circuit-Level Tolerance X RZ Y W RZ Algorithmic error tolerance in datapath is poor Errors are persistent in accumulator Circuit- level error tolerance using 5me borrowing Single- sided cri5cal path at accumulator register 52 of 86

53 Datapath: Circuit-Level Tolerance X W RZ Cri5cal Path RZ Y Algorithmic error tolerance in datapath is poor Errors are persistent in accumulator Circuit- level error tolerance using 5me borrowing Single- sided cri5cal path at accumulator register 53 of 86

54 Datapath: Circuit-Level Tolerance Slack X W RZ Cri5cal Path RZ Y Algorithmic error tolerance in datapath is poor Errors are persistent in accumulator Circuit- level error tolerance using 5me borrowing Single- sided cri5cal path at accumulator register 54 of 86

55 Outline Mo5va5on DNN Engine Parallelism Sparsity Resilience Measurement Results Summary 55 of 86

56 28nm SoC for IoT Applications 28nm SoC Test Chip Cortex- M0 Subsystem Accelerator Subsystem OSC HCLK ARM Cortex- M0 M S V DCO DCO FCLK USB Scan GPIO RTC USB Low- BW Peripherals Timers Wdog UARTs BIST S S S V SOC 32b APB UART SCAN GPIO SYSCTL I- MEM 64KB D- MEM 64KB Bridge S M S M M S S S S V SOC 32b AHB S ASYNC DNN Engine W- MEM 1MB M V ACC!Accelerator Logic V MEMP!SRAM Periphery V MEMC!SRAM Core S 128b AXI 56 of 86

57 Razor Voltage Scaling V DD Scaling (F CLK = 667MHz) Timing Violation Rate TC SM Sign- off Power (mw) FC- DNN (98.36% MNIST) 57 of 86

58 Razor Voltage Scaling V DD Scaling (F CLK = 667MHz) Timing Violation Rate PoFF 30% Power Reduc5on TC SM Sign- off Power (mw) Opera5on at PoFF (770mV) lowers power by 30% 58 of 86

59 Razor Voltage Scaling V DD Scaling (F CLK = 667MHz) Timing Violation Rate Zero Margin Margin PoFF 30% Power Reduc5on TC SM Sign- off Power (mw) Margin for fast varia5ons down to 715mV 59 of 86

60 Timing Error Tolerance Summary Timing Violation Rate % Accuracy Memory Datapath Combined > 1,000,000 X TC SM BM TC SM TB TC ALL Aggregate 5ming viola5on rate greater than of 86

61 Normalized Energy/Pred Energy/Throughput Summary 98.36% Accuracy - 30% Energy Throughput Normalized Pred/Sec TC SM 0.9V 667MHz Sign-off 8b 61 of 86

62 Normalized Energy/Pred Energy/Throughput Summary 98.36% Accuracy - 30% - 30% +4X - 4X Energy Throughput Normalized Pred/Sec TC SM 8b AVS SKIP 0.9V 667MHz Sign-off 0.77V 667MHz PoFF 62 of 86

63 Normalized Energy/Pred Energy/Throughput Summary 98.36% Accuracy - 30% - 30% +4X - 4X Energy Throughput 260nJ Normalized Pred/Sec TC SM 8b AVS SKIP AVS 0.9V 667MHz Sign-off 0.77V 667MHz PoFF 0.715V 667MHz 63 of 86

64 Energy/Throughput Summary Normalized Energy/Pred % Accuracy Energy Throughput - 30% +50% 260nJ +4X - 4X Normalized Pred/Sec TC SM 8b AVS SKIP AVS AFS SKIP 0.9V 667MHz Sign-off 0.77V 667MHz PoFF 0.715V 667MHz 0.9V 1GHz PoFF 64 of 86

65 Energy/Throughput Summary Normalized Energy/Pred % Accuracy Energy Throughput - 30% +50% 260nJ +4X - 4X +20% 570nJ Normalized Pred/Sec TC SM 8b AVS SKIP AVS AFS SKIP AFS 0.9V 667MHz Sign-off 0.77V 667MHz PoFF 0.715V 667MHz 0.9V 1GHz PoFF 0.9V 1.2GHz 65 of 86

66 Summary and Conclusion DNN Engine A Shared Programmable Classifier Parallelism: 10X Data 128b/cycle BW Sparsity: +4X Throughput, - 4X Energy Resilience: +50% Throughput or - 30% Energy w/razor Energy Efficient DNN GHz Timing Error Tolerance Pe 98.36% (MNIST) International Solid-State Circuits Conference with >0.1 Timing Error Rate Tolerance for IoT Applications 66 of 86

67 67 of 86