Homogenization of monthly and daily climatological time series

Size: px

Start display at page:

Download "Homogenization of monthly and daily climatological time series"

Gloria Adelia McGee
5 years ago
Views:

1 Homogenization of monthly and daily climatological time series Petr Štěpánek Pavel Zahradníček Czech Hydrometeorological Institute, Czech Republic 1 st Joint DMCSEE-JRS Workshop on Drought Monitoring, Ljubljana, Slovenia, Sept. 2009

2 Climatological studies Measuring and collecting data

3 Climatological studies Measuring data Data quality control and 0.8 Homogenization

4 Climatological studies Measuring data Homogenization Data Analysis

5 Processing before any data analysis Software AnClim, ProClimDB

6 Data Quality Control Finding Outliers Two main approaches: Using limits derived from interquartile ranges (time series) -2.0 comparing values to values of -4.0 neighbouring stations (spatial analysis) Quality control Homogenization Data Analysis

7 Homogenization Change of measuring conditions inhomogeneities

8 Inhomogeneity Detection Absolute Homogeneity Testing 3,0 2,0 Praha - Klementinum 1,0 0,0-1,0-2,0-3,0-4,

9 Inhomogeneity Detection Absolute Homogeneity Testing Relative Homogeneity Testing (differences with reference series) 1,5 1,0 0,5 0,0-0,5-1,0-1,5-2,0 Diference Praha - Klementinum a průměrovaná řada okolí ČR

10 Relative Homogeneity Testing Creating reference Series

11 Relative Homogeneity Testing Creating reference Series Tests of homogeneity

12 Relative Homogeneity Testing Creating reference Series Tests of homogeneity Assessing homogeneity - Metadata - Physically justified - undoubted inhomogeneity

13 Relative Homogeneity Testing Creating reference Series Tests of homogeneity Assessing homogeneity - Metadata - Physically justified - undoubted inhomogeneity Adjusting Series

14 Inhomogeneity Detecting by SNHT (p=0.05, 950 series) Inhomogeneities detected/ % Error /years > Amount of change in level / C

15 Assessing Homogeneity Problems: most of metadata incomplete we depend upon statistical tests results

16 Assessing Homogeneity Problems: most of metadata incomplete we depend upon statistical tests results uncertainty in test results - right inhomogeneity detection is problematic (for smaller amount of change)

17 Proposed solution To get as many test results for each candidate series as possible Ensemble approach - processing of big amount of test results for each individual series

18 Adventages of the Ensemble approach we know relevance (probability) of each inhomogeneity we can easily assess quality of measurements for series as a whole

19 How to increase number of test results Monthly, Seasonal and How to increase Data Processing number of test Annual results Averages Quality Control - Outliers Interquartile Range Comparing to Neighbours Combining Near Stations Homogeneity Testing Alexandersson test Bivariate Test t-test Mann-Whitney-Pettit from Correlations Reference Series from Distances Several Iterations Hom. Assessment Probability Adjusting Data Filling Miss. Values Days, Months, seasons, year

Creating Reference Series for monthly, daily data (each month individually) weighted/unweighted mean from neighbouring stations criterions used for stations selection (or combination of it): best

20 Creating Reference Series for monthly, daily data (each month individually) weighted/unweighted mean from neighbouring stations criterions used for stations selection (or combination of it): best correlated / nearest neighbours (correlations from the first differenced series) limit correlation, limit distance limit difference in altitudes neighbouring stations series should be standardized to test series AVG and / or STD (temperature - elevation, precipitation - variance) - missing data are not so big problem then

21 Example: Proposed list of stations used for creating reference series ID_1 ID_2 BEGIN END LENGREMARK CORREL DISTANCE ALT_1 ALT_2 B1BLAT st. (l: B1HLUK y. comm.p B1VELV y. comm.p B1STRZ y. comm.p B1UHBR y. comm.p B1RADE y. comm.p Selection according to correlations, distances and altitudes B1BOJK st. (l: B1STRN y. comm.p B1STHR y. comm.p B1LUHA y. comm.p B1VIZO y. comm.p B1UHBR y. comm.p B1BRBY st. (l: B1BOJK y. comm.p O3ZDEC y. comm.p O3HUSL y. comm.p B1HLHO y. comm.p B1STHR y. comm.p B1BUCH st. (l: B1STME y. comm.p B2KYJO y. comm.p B2KORC y. comm.p B1BZEN y. comm.p B1NAPA y. comm.p

22 Relative homogeneity testing Available statistical tests: Alexandersson SNHT Bivariate test of Maronna and Yohai Mann Whitney Pettit test t-test Easterling and Peterson test Vincent method 20 year parts of the daily series (40 for monthly series with 10 years overlap), in SNHT splitting into subperiods in position of detected significant changepoint (30-40 years per one inhomogeneity)

23 Homogeneity Tests Alexandersson s SNHT Alexandersson Standart Normal Homogeneity Test (Single shift test) Reference series: k q i = Y /{[ ρ X Y / X j ]/ ρ } i j= 1 k 2 j ji k j= 1 2 j q i = Yi { ρ j [ X ji X j + Y ]/ ρ j } j= 1 2 k j= Null and alternative hypothesis: H 0 : z i N(0,1), i {1,..,n}. H 1 : z i N(μ 1,1), i {1,..,a}, z i N(μ 2,1), i {a+1,..,n}, for 1? a < n a μ 1? μ 2. z i = (q i - q )/s q, z i N(0,1) Test statistic: 2 2 T 0 = max {Ta} 1 a< n 1 = max { az1 + ( n a) z 2} 1 a< n 1 where z a 1 1 = z i, ( z 1? μ a 1), Quality control í = 1 n z 1 Homogenization 2 = z i, ( z 2? μ ( n a) 2). í = a+ 1 Data Analysis

24 Homogeneity Tests Bivariate Test of Maronna and Yohai Bivariate Test Null and alternative hypothesis: H 0 : vectors {x i,y i } bivariate normal distributed N(μ x, μ y, σ 2 x, σ 2 y, ρ) H 1 : pro 0<i 0 <n a d? 0 - N(μ x, μ y, σ 2 x, σ 2 y, ρ) pro i? i 0 N(μ x, μ y+d, σ 2 x, σ 2 y, ρ) pro i > i 0. Test statistic: T 0 = max{t } i<n i i where: X i = 1 / i x j, Y i = 1 / i y j, X = X n, Y = Yn n j= 1 i j= S x = ( x j X ), S y = ( y j Y), S xy = ( x j X )( y j Y), j = 1 n j= 1 2 F i = S x ( X i X ) ni /( n i), i<n, D i = S x ( Y Yi ) S xy( X X i ) n /[( n i) Fi ], 2 i 2 xy T i =[ i( n i) D F ]/( S S S ) i x y n j= 1

25 Homogeneity Tests Two-phase linear regression (Vincent Technique, Easterling and Peterson test) Easterling and Peterson Test statistic: U = [(RSS 1 -RSS 2 )/3]/[RSS 2 /(n-4)]? F(3,n-4) t-test: differences of levels before and after a discontinuity

26 Homogeneity assessment Various outputs created for better inhomogeneities assessment Combining results with information from metadata whenever possible Decision about undoubted inhomogeneities (without metadata) coincidence of test results

27 Homogeneity assessment Output example: Station Čáslav, 3rd segment, , n=40 Test Ref I II III IV V VI VII VIII IX X XI XII Win Spr Sum Aut Year A avg A 1930 A corr A A dist A B avg B 1922 B corr B B 1937 B dist B V corr V V 1937 V dist V 1918

28 Homogeneity assessment, Output II example: Begin End Length InHomogen eity Number % detected inhom % possible inhom End Missin g Summed numbers of detections for individual years

29 Homogeneity assessment combining several outputs (sums of detections in individual years, metadata, graphs of differences/ratios, ) ID ELYEAR_BEGINEND YEAR_COUNY_POSSIBL YEAMIS X_BEGIN_DX_END_DATX_X_L L ABREMARKC C_ x B1BOJK01 x # # Bchange B1BOJK01 x # # obs V B B1BYSH01 x ? B1BYSH01 x ? B1BYSH01 x ? B1HLHO01 x B1HOLE01 x B1KROM01 x xb1rade01x # # Rchange B1RADE01 x # # obs JoB x B1RYCH01 x # # Vchange B1RYCH01 x # # obs MB xx? B1STRZ01 x B1STRZ01 x B1UHBR01 x # # Uchange B1UHBR01 x # # obs JoB x B1UHBR01 x # # Uchange B1UHBR01 x # # obs JoB B1VELI01 x ? B1VELI01 x ? B1VKLO01 x x B1VYSK01 x # # Vchange B1VYSK01 x # # obs V B B2BOSK01_rx B2BREC01 x B2BRUM01 x # # Bchange B2BRUM01 x # # obs MB

30 Adjusting monthly data using reference series based on correlations adjustment: from differences/ratios 20 years before and after a change, monhtly smoothing monthly adjustments (low-pass filter for adjacent values) I II III IV V VI VII VIII IX X XI XII

31 Example: Adjusting values - evaluation

32 Iterative homogeneity testing several iteration of testing and results evaluation several iterations of homogeneity testing and series adjusting (3 iterations should be sufficient) question of homogeneity of reference series is thus solved: possible inhomogeneities should be eliminated by using averages of several neighbouring stations if this is not true: in next iteration neighbours should be already homogenized

33 Filling missing values Before homogenization: influence on right inhomogeneity detection After homogenization: more precise - data are not influenced by possible shifts in the series Dependence of tested series on reference series

34 ACTION COST-ES0601: Advances in homogenisation methods of climate series: an integrated approach (HOME) 03/05/ End date: 02/05/2011, Year: 3 Inventory of existing detection and correction methods Compilation of a benchmark dataset with (un)known inhomogeneities Selection, comparison and evaluation of existing detection and correction (including those not traditionally used in climatology) Objective analysis of advantages and/or disadvantages of existing methods (benchmark) Investigation in further improvements of methods Documentation of practical recommendations Presentation and release of a new common method

35 Quality control on example of series in the Czech Republic

36 Data Quality Control Finding Outliers Two main approaches: Using limits derived from interquartile ranges (time series) -2.0 comparing values to values of -4.0 neighbouring stations (spatial analysis)

37 Example: Comparing base station to its neighbours ID YEAR MONTST_BASE REMARK ST_1 ST_2 ST_3 ST_4 ST_5 Rat1_STND Rat2_STND Rat3_STND Rat4_STND Rat5_STND CDF_MAX No_sign. B1BLAT Altitudes,lim B1HLUK01 st_1, distan 6.8 B1VELV01 st_2, distan 8.9 B1STRZ01 st_3, distan 10.4 B1BZEN01 st_4, distan 12.2 B1RADE01 st_5, distan 13.3 B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT B1BLAT

38 Example of outputs for outliers assessment Suspicious values Expected value Neighbour stations values List of neighbours Altitudes and distances of neighbours

39 Quality control Run for period , daily data (measured values in observation hours) All stations (200 climatological stations, 800 precipitation stations) All meteorological elements (T, TMA, TMI, TPM, SRA, SCE, SNO, E, RV, H, F) parameters set individually Historical records will follow now

40 Air temperature, number of outliers 1961 from station-days number of outliers , T air temperature at obs. hour, TMA daily maximum temp., TMI daily min. temp., TPM daily ground minimum temp T_07:00 T_14:00 T_21:00 T_AVG TMA TMI TPM

41 Air temperature, number of outliers 1961 from station-days number of outliers , Air temperature at obs. hour, AVG daily average temp. 250 Temperature 07:00 14:00 21:00 AVG Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

42 Air temperature, number of outliers 1961 from station-days number of outliers , TMA daily maximum temp., TMI daily min. temp., TPM daily ground minimum temp. 250 Max, min temperature, ground minimum TMA TMI TPM v Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

43 Air temperature, number of outliers , Number of outliers per one station (all observation hours, AVG) temperature

44 Spatial distribution of precipitation stations period stations mean minimum distance: 7.5 km

45 Problematic detections (heavy rainfall)

46 Problematic detections (heavy rainfall), Radar information

47 Precipitation, number of outliers , from station-days precipitation Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

48 Precipitation, number of outliers , Number of outliers per one station precipitation

49 Homogenization of the series in the Czech Republic # Prague # Brno

50 Example: CZ, air temperature (200 stations, ) Number of significant inhomogeneities before and after homogenization (p=0.05) 2500 Before After I II III IV V VI VII VIII IX X XI XII Before After 0 Win Spr Sum Aut Year

51 Amount of adjustments for homogenised series (absolute values) - median C I II III IV V VI VII VIII IX X XI XII Correlation coefficients between candidate and reference series before and after homogenization (median) 1.00 Before After I II III IV V VI VII VIII IX X XI XII

52 Example: CZ, precipitation (800 stations, ) 4 tests, 4 reference series, 12 months + 4 seasons and year Number of detected inhomogeneities (significant) 6000 Number of detections I II III IV V VI VII VIII IX X XI XII Month

53 Amount of change (ratios standardized to be >1.0), precipitation (reference series calculation based on correlations) Amount of change (standardized) I II III IV V VI VII VIII IX X XI XII Boxplots: -Median - Upper and lower quartiles (for 589 testes series) Correlation improvement Correlation increase I II III IV V VI VII VIII IX X XI XII

54 Inhomogeneities in summer versus in winter, Air temperature Change of measuring conditions at the station (relocation etc.) is manifested in the series mainly in summer in winter: active surface role is diminished, prevailng circulation factors, in summer: active surface role increases, prevailing radiation factors

55 Inhomogeneities in summer versus in winter, Precipitation Change of measuring conditions at the station (relocation etc.) is manifested in the series mainly in winter in winter: errors of measurement (solid precipitation -wind, )

56 Homogenization Final remarksr emarks, recommendations 1/2 data quality control before homogenization is of very importance (if it is not part of it) Using series of observation hours (complementarily to daily AVG) is highly recommended (different manifestation of breaks) be aware of annual cycle of inhomogeneities, adjustments, to know behavior of spatial correlations (of element being processed) to be able to create reference series of sufficient quality

57 Homogenization Final remarksr emarks, recommendations 2/2 Because of Noise in the time series it makes sense: - Ensemble approach to homogenization (combining information from different statistical tests, time frames, overlapping periods, reference series, meteorological elements, ) - more information for inhomogeneities assessment higher quality of homogenization in case metadata are incomplete

58 Software used for data processing LoadData - application for downloading data from central database (e.g. Oracle) ProClimDB software for processing whole dataset (finding outliers, combining series, creating reference series, preparing data for homogeneity testing, extreme value analysis, RCM outputs validation, correction, ) AnClim software for homogeneity testing limahom.eu

59 AnClim software AnClim software

60 ProClimDB software ProcData software

61 Download data from database (e.g. Oracle) (LoadData) Quality control (ProClimDB) Homogenization (ProClimDB/AnClim) Technical series and grid points calculation (ProClimDB)

62 Statistical analysis SPI, Validation of RCM outputs Extreme value analysis Correction of RCM outputs Spatial analysis (connection ProClimDB - ArcView) Further tools: (connection ProClimDB - R)

COMPARISON OF FIXED & VARIABLE RATES (25 YEARS) CHARTERED BANK ADMINISTERED INTEREST RATES - PRIME BUSINESS*

COMPARISON OF FIXED & VARIABLE RATES (25 YEARS) Fixed Rates Variable Rates FIXED RATES OF THE PAST 25 YEARS AVERAGE RESIDENTIAL MORTGAGE LENDING RATE - 5 YEAR* (Per cent) Year Jan Feb Mar Apr May Jun Jul