White Rabbit Applications for Data Acquisition Systems

Size: px

Start display at page:

Download "White Rabbit Applications for Data Acquisition Systems"

Jonathan Wiggins
5 years ago
Views:

1 White Rabbit Applications for Data Acquisition Systems Dimitris Lampridis CERN BE-CO Hardware and Timing section DAQ Meeting, IN2P3, 01 June 2016 Dimitris Lampridis WR in DAQ 1/36

2 1 White Rabbit 2 WR Demo 3 WR for Distributed DAQ 4 Adding WR to your Design 5 Conclusions Dimitris Lampridis WR in DAQ 2/36

3 Outline 1 White Rabbit 2 WR Demo 3 WR for Distributed DAQ 4 Adding WR to your Design 5 Conclusions Dimitris Lampridis WR in DAQ 3/36

4 What is White Rabbit? A protocol to synchronize nodes in a large-scale network with sub-ns accuracy Open Hardware and Open Software with commercial support International collaboration Dimitris Lampridis WR in DAQ 4/36

5 Why we use Open Hardware? Commercial Non-commercial Open Winning combination. Best of both worlds. Whole support burden falls on developers. Not scalable. Proprietary Vendor lock-in. Dedicated non-reusable projects. Get a design just the way we want it Peer review and design re-use Healthier relationship with companies Dimitris Lampridis WR in DAQ 5/36

6 White Rabbit: an extension of Ethernet Standard Ethernet network Ethernet features (VLAN) & protocols (SNMP) Dimitris Lampridis WR in DAQ 6/36

7 White Rabbit: an extension of Ethernet Standard Ethernet network Ethernet features (VLAN) & protocols (SNMP) High accuracy synchronization Reliable and low-latency Control Data Dimitris Lampridis WR in DAQ 6/36

8 White Rabbit Switch Central element of WR network 18 port gigabit Ethernet switch with WR features Optical transceivers: up to 10 km, single-mode fiber Fully open design, commercially available Dimitris Lampridis WR in DAQ 7/36

9 White Rabbit Nodes Carrier boards in PCI-Express, VME, PXIe Equipped with a WR port and FMC connector(s) Mezzanines use the WR clock signal and timing interface All sources available in the OHWR: Dimitris Lampridis WR in DAQ 8/36

10 White Rabbit technology Based on Gigabit Ethernet over fiber IEEE-1588 (PTP) protocol Dimitris Lampridis WR in DAQ 9/36

11 White Rabbit technology Based on Gigabit Ethernet over fiber IEEE-1588 (PTP) protocol Enhanced with Layer 1 syntonization Digital Dual Mixer Time Difference (DDMTD) Link delay model Dimitris Lampridis WR in DAQ 9/36

12 Precision Time Protocol (IEEE 1588) Frame-based synchronization protocol Like NTP but in hardware Simple calculations: link delay ms δ ms = (t 4 t 1 ) (t 3 t 2 ) 2 clock offset ms = t 2 t 1 + δ ms Dimitris Lampridis WR in DAQ 10/36

13 Precision Time Protocol (IEEE 1588) Frame-based synchronization protocol Like NTP but in hardware Simple calculations: link delay ms δ ms = (t 4 t 1 ) (t 3 t 2 ) 2 clock offset ms = t 2 t 1 + δ ms Can be further improved assumes symmetry of medium all nodes have free-running oscillators frequency drift compensation vs. message exchange traffic Dimitris Lampridis WR in DAQ 10/36

White Rabbit WR Demo WR for Distributed DAQ Adding WR to your Design Conclusions Layer 1 Syntonization All network devices use the same physical layer clock.

14 White Rabbit WR Demo WR for Distributed DAQ Adding WR to your Design Conclusions Layer 1 Syntonization All network devices use the same physical layer clock. Clock is encoded in the Ethernet carrier and recovered by the receiver chip. Clock is looped back, phase detection allows sub-ns delay measurement. Dimitris Lampridis WR in DAQ 11/36

15 Digital Dual Mixer Time Difference DDMTD Used for precise phase measurements Outputs are at much lower frequencies, easier to measure Dimitris Lampridis WR in DAQ 12/36

16 Link delay model WR Master WR Slave WR gear Δ RXM Δ TXM Δ TXM δ SM ε S ε M Δ TXS Δ RXS WR gear δ MS static hardware delays: TXM, RXM, TXS, RXS semi-static hardware delays: ɛ M, ɛ S fiber asymmetry coefficient: α = δ MS δ SM δ SM Dimitris Lampridis WR in DAQ 13/36

17 White Rabbit application examples CERN and GSI Dimitris Lampridis WR in DAQ 14/36

18 White Rabbit application examples CERN and GSI HiSCORE: Gamma&Cosmic-Ray experiment Dimitris Lampridis WR in DAQ 14/36

19 White Rabbit application examples CERN and GSI HiSCORE: Gamma&Cosmic-Ray experiment The Large High Altitude Air Shower Observatory Dimitris Lampridis WR in DAQ 14/36

20 White Rabbit application examples CERN and GSI HiSCORE: Gamma&Cosmic-Ray experiment The Large High Altitude Air Shower Observatory MIKES: Centre for metrology and accreditation Dimitris Lampridis WR in DAQ 14/36

White Rabbit application examples CERN and GSI HiSCORE: Gamma&Cosmic-Ray experiment The Large High Altitude Air Shower Observatory MIKES: Centre for metrology and

21 White Rabbit application examples CERN and GSI HiSCORE: Gamma&Cosmic-Ray experiment The Large High Altitude Air Shower Observatory MIKES: Centre for metrology and accreditation KM3NET: European deep-sea research infrastructure More WR users: Dimitris Lampridis WR in DAQ 14/36

22 Outline 1 White Rabbit 2 WR Demo 3 WR for Distributed DAQ 4 Adding WR to your Design 5 Conclusions Dimitris Lampridis WR in DAQ 15/36

23 WR Demo Demo in progress... Dimitris Lampridis WR in DAQ 16/36

24 Outline 1 White Rabbit 2 WR Demo 3 WR for Distributed DAQ 4 Adding WR to your Design 5 Conclusions Dimitris Lampridis WR in DAQ 17/36

25 Purpose Provide a communication protocol for distributed instrumentation over WR Dimitris Lampridis WR in DAQ 18/36

26 Motivation OASIS: Open Analog Signals Information System Distributed oscilloscope 1000s of signals 100s of triggers Unidirectional Hard-wired Dimitris Lampridis WR in DAQ 19/36

27 Existing Solution: LXI Nearest existing solution is LXI Designed for instrumentation Works over Ethernet Plug & Play Has extensions for synchronisation, timestamping and message exchanging Dimitris Lampridis WR in DAQ 20/36

28 WRXI White Rabbit extensions for Instrumentation A communication protocol for distributed instrumentation over a White Rabbit (WR) network Inspired by LXI Leverages the high accuracy and precise synchronisation offered by WR Augments WR with complex event scheduling, timestamping and real-time message exchanging across the network Designed in an application-agnostic way, so that it can be adopted and re-used by others Fully open design and implementation Dimitris Lampridis WR in DAQ 21/36

29 Vision Design a new protocol for instrumentation Flexible Robust Scalable Re-usable Sustainable Fully open On top of WR Node Host Controller WR Switch Node WR Switch Node Non-WRXI Node The network will be built on top of WR switches, with distributed instrumentation nodes, under the supervision and control of a host controller. The host controller can be linked to an external network. Non-WRXI instrumentation can be attached to special nodes (eg. GPIB bridges, external trigger generators, etc.) Dimitris Lampridis WR in DAQ 22/36

30 WRXI Example 1 Host Controller WR Switch FMC-ADC WR Switch FMC-TDC Pulse Generator FMC-DEL Non-WRXI Digitiser Dimitris Lampridis WR in DAQ 23/36

31 WRXI Example 1 1 FMC-TDC: generate message #1 upon reception of external TTL pulse Host Controller WR Switch FMC-ADC WR Switch FMC-TDC Pulse Generator FMC-DEL Non-WRXI Digitiser Dimitris Lampridis WR in DAQ 23/36

32 WRXI Example 1 1 FMC-TDC: generate message #1 upon reception of external TTL pulse 2 FMC-ADC: get message #1 and arm Host Controller WR Switch FMC-ADC WR Switch FMC-TDC Pulse Generator FMC-DEL Non-WRXI Digitiser Dimitris Lampridis WR in DAQ 23/36

33 WRXI Example 1 1 FMC-TDC: generate message #1 upon reception of external TTL pulse 2 FMC-ADC: get message #1 and arm 3 FMC-ADC: generate message #2 on trigger FMC-ADC Host Controller WR Switch WR Switch FMC-TDC Pulse Generator FMC-DEL Non-WRXI Digitiser Dimitris Lampridis WR in DAQ 23/36

34 WRXI Example 1 1 FMC-TDC: generate message #1 upon reception of external TTL pulse 2 FMC-ADC: get message #1 and arm 3 FMC-ADC: generate message #2 on trigger 4 FMC-DEL: get message #2 and generate pulse FMC-ADC Host Controller WR Switch FMC-TDC WR Switch FMC-DEL Pulse Generator Non-WRXI Digitiser Dimitris Lampridis WR in DAQ 23/36

35 WRXI Example 1 1 FMC-TDC: generate message #1 upon reception of external TTL pulse 2 FMC-ADC: get message #1 and arm 3 FMC-ADC: generate message #2 on trigger 4 FMC-DEL: get message #2 and generate pulse 5 Execute FMC-ADC #1 Host Controller WR Switch FMC-TDC WR Switch #2 FMC-DEL Pulse Generator Non-WRXI Digitiser Dimitris Lampridis WR in DAQ 23/36

36 WRXI Example 1 1 FMC-TDC: generate message #1 upon reception of external TTL pulse 2 FMC-ADC: get message #1 and arm 3 FMC-ADC: generate message #2 on trigger 4 FMC-DEL: get message #2 and generate pulse 5 Execute 6 Retrieve data FMC-ADC Host Controller WR Switch FMC-TDC Pulse Generator WR Switch FMC-DEL Non-WRXI Digitiser Dimitris Lampridis WR in DAQ 23/36

37 WRXI Example 2 Host Controller WR Switch FMC-ADC1 WR Switch FMC-TDC FMC-ADC2 Pulse Generator Dimitris Lampridis WR in DAQ 24/36

38 WRXI Example 2 1 FMC-TDC: record pulse, generate message #1 Host Controller WR Switch FMC-ADC1 WR Switch FMC-TDC FMC-ADC2 Pulse Generator Dimitris Lampridis WR in DAQ 24/36

39 WRXI Example 2 1 FMC-TDC: record pulse, generate message #1 2 FMC-ADC1: in free-running mode, get message #1 and trigger FMC-ADC1 Host Controller WR Switch WR Switch FMC-TDC FMC-ADC2 Pulse Generator Dimitris Lampridis WR in DAQ 24/36

40 WRXI Example 2 1 FMC-TDC: record pulse, generate message #1 2 FMC-ADC1: in free-running mode, get message #1 and trigger 3 FMC-ADC2: in free-running mode, get message #1 and trigger FMC-ADC1 Host Controller WR Switch WR Switch FMC-TDC FMC-ADC2 Pulse Generator Dimitris Lampridis WR in DAQ 24/36

41 WRXI Example 2 1 FMC-TDC: record pulse, generate message #1 2 FMC-ADC1: in free-running mode, get message #1 and trigger 3 FMC-ADC2: in free-running mode, get message #1 and trigger 4 Execute FMC-ADC1 #1 Host Controller WR Switch FMC-TDC WR Switch #1 FMC-ADC2 Pulse Generator Dimitris Lampridis WR in DAQ 24/36

42 WRXI Example 2 1 FMC-TDC: record pulse, generate message #1 2 FMC-ADC1: in free-running mode, get message #1 and trigger 3 FMC-ADC2: in free-running mode, get message #1 and trigger 4 Execute 5 Rewind and retrieve data FMC-ADC1 Host Controller WR Switch FMC-TDC WR Switch FMC-ADC2 Pulse Generator Dimitris Lampridis WR in DAQ 24/36

43 Outline 1 White Rabbit 2 WR Demo 3 WR for Distributed DAQ 4 Adding WR to your Design 5 Conclusions Dimitris Lampridis WR in DAQ 25/36

44 Many possibilities Make use of one of the provided carriers and selection of mezzanines Include an HDL core in your design Use a standalone WR node implementation Dimitris Lampridis WR in DAQ 26/36

45 WR PTP Core - overview WRPC Clocks PHY i/f WR Fabric Misc. Timecode i/f HDL core with soft CPU Ethernet MAC with WR features WR implementation for the nodes Dimitris Lampridis WR in DAQ 27/36

46 WR PTP Core - inside SFP PHY (GTP, GTX,...) WR PTP Core UART Endpoint Fabric redirector Pipelined WB MAC I/F Flash / EEPROM Tunable oscillators Periph SoftPLL 1-PPS Wishbone crossbar Lattice Mico32 mini-nic RAM Control Wishbone I/F Dimitris Lampridis WR in DAQ 28/36

47 WR PTP Core - clocks 125 MHz reference clock 62.5 MHz DDMTD clock system clock ( ref. clock) aux clocks Dimitris Lampridis WR in DAQ 29/36

p n WR PTP Core - how to integrate Spartan 6 FPGA SFP TD+ TD- RD+ RDmod-def(0) mod-def(1) mod-def(2) TxDisable EEPROM sfp_txp_o sfp_txn_o sfp_rxp_i sfp_rxn_i wr_gtp_phy_spartan6 PHY i/f sfp_sda_i/o

48 p n WR PTP Core - how to integrate Spartan 6 FPGA SFP TD+ TD- RD+ RDmod-def(0) mod-def(1) mod-def(2) TxDisable EEPROM sfp_txp_o sfp_txn_o sfp_rxp_i sfp_rxn_i wr_gtp_phy_spartan6 PHY i/f sfp_sda_i/o sfp_scl_i/o sfp_det_i sda_i/o scl_i/o WRPC rst_n_i clk_sys_i clk_ref_i dac_dpll_load_p1_o dac_dpll_data_o dac_hpll_load_p1_o dac_hpll_data_o clk_dmtd_i 62.5MHz 62.5MHz PLL_BASE spec_serial_dac_arb dac_cs_n_o(0) PLL_BASE IBUFGDS dac_clr_n_o dac_sclk_o dac_din_o dac_cs_n_o(1) BUFG AD5662 DAC DAC AD5662 VM53S VCXO VCXO LF VCXO MHz CDCM61004 CLK GEN 125MHz DS18B20 owr_en_o owr_i uart_rxd_i uart_txd_o WRF Source user-defined module WRF Sink WRF Sink WRF Source Timecode i/f 1-PPS Dimitris Lampridis WR in DAQ 30/36

49 WR PTP Core - resource utilization Xilinx Spartan-6, XC6SLX45T-3FGG484 Dimitris Lampridis WR in DAQ 31/36

50 Standalone WR node 1 Dimitris Lampridis WR in DAQ 32/36

51 Standalone WR node 2 Dimitris Lampridis WR in DAQ 33/36

52 Outline 1 White Rabbit 2 WR Demo 3 WR for Distributed DAQ 4 Adding WR to your Design 5 Conclusions Dimitris Lampridis WR in DAQ 34/36

53 Summary Open (H/W & S/W) Dimitris Lampridis WR in DAQ 35/36

54 Summary Open (H/W & S/W) Commercial support Dimitris Lampridis WR in DAQ 35/36

55 Summary Open (H/W & S/W) Commercial support More applications than ever expected Dimitris Lampridis WR in DAQ 35/36

56 Summary Open (H/W & S/W) Commercial support More applications than ever expected A versatile solution for general control and data acquisition Dimitris Lampridis WR in DAQ 35/36

57 Summary Open (H/W & S/W) Commercial support More applications than ever expected A versatile solution for general control and data acquisition Standard-compatible and standard-extending Dimitris Lampridis WR in DAQ 35/36

58 Summary Open (H/W & S/W) Commercial support More applications than ever expected A versatile solution for general control and data acquisition Standard-compatible and standard-extending Active participation in IEEE1588 revision process Dimitris Lampridis WR in DAQ 35/36

59 Join the development! Dimitris Lampridis WR in DAQ 36/36

Session Objectives. At the end of the session, the participants should: Understand advantages of BFD implementation on S9700

Session Objectives. At the end of the session, the participants should: Understand advantages of BFD implementation on S9700 BFD Features Session Objectives At the end of the session, the participants should: Understand advantages of BFD implementation on S9700 Understand when to use BFD on S9700 1 Contents BFD introduction