Quality control of data streams at ANFOG

Transcription

1 Quality control of data streams at ANFOG Christine Hanson & Claire Gourcuff Australian National Facility for Ocean Gliders (ANFOG), School of Environmental Systems Engineering (SESE) and UWA Oceans Institute

2 Measured data/variables Slocum gliders Seagliders Conductivity (salinity) Temperature Depth Dissolved oxygen 4 channel downwelling irradiance (on G2 Slocum gliders) Optical backscatter (650 or 700 nm) Chlorophyll a fluorescence (695 nm) CDOM fluorescence (460 nm) Depth averaged currents (estimated based on variation between dead reckoned position and GPS position)

3 ANFOG Workflow

4 Automatic QC Impossible date test Valid after 2007 and before present Impossible location test Valid within range of latitude 90 to 90, longitude 180 to 180 Deepest pressure test Max of 220 m for Slocum, 1100 m for Seaglider

5 Automatic QC Parameter (units) Valid Min, Valid Max Range CNDC (S m 1 ) 0, 60 0, 60 PSAL (psu) 2, 41 2, 41 TEMP ( C) 2, 40 2, 40 reduce min? DOXY (kg m 3 ) 0, 16 0, 16 FLU2 (mg m 3 ) 0, 50 0, 20 (?) for discussion CDOM (ppb) 0, 386 0, 55 (?) for discussion VBSC (m 1 sr 1 ) 0, , (?) for discussion OCR (uw cm 2 nm 1 ) 0, 1000 Valid min/valid max from ARGO (where possible) and sensor min/max output Range test relative to Australian waters, must accommodate all expected extremes in regions of deployment Should these values be the same across all IMOS facilities?

6 Automatic QC Spike & gradient tests From ARGO definitions A difference between sequential measurements, where one measurement is quite different than adjacent ones, is a spike in both size and gradient Spike test value = V2 (V3 + V1)/2 (V3 V1) / 2 Gradient test value = V2 (V3 + V1)/2 TEMP PSAL Flagged when exceeding: 6.0 C (< 500 db) 2.0 C (> 500 db) 0.9 PSU (< 500 db) 0.3 PSU (> 500 db) DOXY 4.0 kg m 3 FLU2 1.1 mg m 3 CDOM 1.0 ppb VBSC m 1 sr 1 where V2 is the measurement being tested as a spike, and V1 and V3 are the values above and below Values refined from ARGO numbers based on examination of glider datasets TEMP PSAL Flagged when exceeding: 9.0 C (< 500 db) 3.0 C (> 500 db) 1.5 PSU (< 500 db) 0.5 PSU (> 500 db)

7 Manual QC Subjective visual inspection of data values by an operator Aim to reduce the subjective nature of this QC as much as possible although will still have an influence To avoid delays, not mandatory before real time distribution Examples of when this additional QC is applied: Glider sitting on the seabed Noise due to bio fouling or other reasons Sensor failure or mechanical problems

8 Visual QC Examine plots of each variable to check for anomalies

9 Check for reasons that may explain anomalous points At end of manual QC of physical variables, check consistency of flags for TEMP/CNDC/PSAL

10 Manually set a locally acceptable data range

11 Then remove remaining spikes

12 CDOM patterns for discussion CDOM increase in daylight hours and at surface (< 0.5 m)?

13 Surface spikes in fluorescence data

14 Saturation of backscatter sensor

15 Pre and post deployment optical instrument checks Using solid standards to check for fouling or sensor malfunction as per Earp et al. Incorporate sensor photos and these recovery sensor tests into metadata or other

16 Flag bad data resulting from bio fouling Examining various parameters and ratios to assist with flagging of data affected by bio fouling Coefficient of variation

17 Optical instrument checks using liquid standards Using liquid standards to check for consistency across platforms small and large calibration chambers Potential differences between Slocum and Seaglider, currently double checking

18 Questions/items for discussion Pressure not currently kept as a variable in netcdf files Latitude_quality_control & Longitude_quality_control redundant to have both kept as variables, should we replace with Position_QC If position flagged bad, should also flag rest of data bad? Not currently done at ANFOG Should we consider Bad (QC = 4) only for automatic QC tests, and use another flag for manual/subjective QC ( probably bad?)

19 Dissolved oxygen corrected for time lag

20 NSW Seaglider Deployments 123 days 696 dives 2,704 kilometres East Australian Current Investigating interactions of the East Australian Current (EAC) with the shelf waters Examine vertical structure of EAC eddies