Calibration and Transferability of Accident Prediction Models for Urban Intersections

Transcription

1 Calibration and Transferability of Accident Prediction Models for Urban Intersections Bhagwant Persaud Department of Civil Engineering Ryerson University 350 Victoria Street, Toronto M5B2K3 Phone: ; Fax ; Dominique Lord Center for Transportation Safety Texas Transportation Institute Texas A&M University System 3135 TAMU College Station, TX Phone: ; Joseph Palmisano ITRANS Consulting, Inc. 100 York Blvd., Richmond Hill, Ontario Canada, L4B 1J8 ABSTRACT Accident prediction models, also known as safety performance functions, have several important uses in modern day safety analysis. Unfortunately, calibration of these models is not straightforward. This paper documents a research effort that demonstrates the complexity of calibrating these models for urban intersections. These complexities relate to the specification of the functional form, the accommodation of the peculiarities of accident data, and to the transferability of models to other jurisdictions. Toronto data are used to estimate models for 3- and 4-legged signalized and unsignalized intersections. Then the performance of these models is compared to models for Vancouver and California that were recalibrated for Toronto using a procedure recently proposed for the application in the Interactive Highway Safety Design Model (IHSDM). The results of this transferability test are mixed, suggesting that a single calibration factor as is currently specified in the IHSDM procedure may be inappropriate and that a disaggregation by traffic volume might be preferable.

2 BACKGROUND AND INTRODUCTION The purpose of this paper is to document a research effort that demonstrates the complexity of calibrating macroscopic accident prediction models for urban intersections. These models, also known as safety performance functions, relate the annual accident experience of an entity to its characteristics, most importantly, its traffic volume. They have several uses in safety analysis. For example, they can be used to examine how the actual accident experience of a specific intersection compares to the expected safety performance of similar intersections. These models are also used in the empirical Bayes method to, in effect, smooth the random fluctuation in accident counts in estimating the number of accidents expected in the long run at a specific intersection (1). This expected value is used in before-after studies and can be used in the screening of potentially hazardous intersections for treatment (2). The importance of accident prediction models (APMs) makes it imperative that they be properly calibrated. Calibrating such models is not straightforward. First high quality data is required for a large enough sample of entities and accidents, and there needs to be a facility to easily link traffic, accident and road characteristic files that are usually separately maintained. Second, to obtain large enough samples of accidents, several years of data are used, which amounts to repeated observations in modelling accidents per year, a reality that is in conflict with the assumption in conventional regression modelling of independence of observations; this difficulty is magnified by temporal trends in accident counts. Third, the specification of the mathematical form is not a trivial task. Finally, the actual process of model calibration is complicated by the fact that accident counts are discrete and non negative and therefore are not in accord with the important assumption in conventional regression modelling of a normally distributed dependent variable. The complexities of calibrating accident prediction models perhaps explain why, despite their value, there is a relatively small selection of these models available for urban intersections. Of note are two theoretically sound efforts of late, by Rodriguez & Sayed (3) who calibrated models for unsignalized intersections in British Columbia and Bauer & Harwood (4) who used California data to estimate models for both signalized and unsignalized intersections. The difficulty of developing new models makes it unlikely that each jurisdiction, particularly smaller ones, will be able to develop their own models. Instead it would be desirable if models calibrated for one jurisdiction in one period of time could be applied for a different period in the same or another jurisdiction. It seems reasonable to adopt this approach since traffic volumes are the major contributing factor in explaining accident occurrence and the nature of this interaction is unlikely to vary among jurisdictions with reasonably similar driver, road and vehicle characteristics. This is the approach that is adopted in a recent Federal Highway Administration (FHWA) report (5) that outlines a procedure for recalibrating base models for application in a jurisdiction of interest. This recalibrating is largely required because of differences in the level of accident reporting over time and across jurisdictions. This paper serves two basic purposes. First, it presents urban intersections accident prediction models recently calibrated for Toronto, and which would be of general interest to the research and practitioner communities. In so doing it addresses the model estimation and transferability issues raised above. For the latter, the procedure in the FHWA report (5) is applied to recalibrate the British Columbia and California models (3, 4) for Toronto conditions. The prediction accuracy of these artificially calibrated models is assessed relative to that for models directly calibrated from Toronto data. The temporal transferability models, although of interest, is outside the scope of this paper. However, some insights on this issue have been provided in a recent paper (6). THE TORONTO DATABASE Table 1 indicates the basic characteristics of the Toronto data used in the calibration of the accident prediction models. It is evident that this dataset is quite rich in terms of numbers of intersections and accidents. 2

3 Intersection Class & Number in sample TABLE 1: Characteristics of data used to estimate Toronto models Accident Accidents Severity Min-max in year 6 Year total Major Road Entering AADT (min-max) (6 year average) Minor Road Entering AADT (min-max) (6 year average) Signalized 4-legged (868) Total Injury PDO ,989 16,339 38,650 5,305-72, ,644 Signalized 3-legged (250) Total Injury PDO ,214 2,074 5,140 1,660-75, ,923 Unsignalized 4- legged (59) Total Injury PDO , ,857-53, ,638 Unsignalized 3- legged (277) Total Injury PDO , ,218 8,869-52, ,909 MODELLING APPROACH, ISSUES AND RESULTS It was desired to estimate models for three and four-legged signalized and unsignalized intersections and, for each of these four intersection types, separate models for injury (fatal + non-fatal) and all accident severities combined (injury plus property damage only). The model forms were selected after conducting exploratory analyses on the data using the ID method proposed by Hauer & Bamfo (7). Basically, the method consists of creating an Empirical Integral Function (EIF) for each covariate by placing intersections into a series of bins in increasing order of the covariate. For a given intersection, the left boundary of the bin is located halfway between the covariate value for that intersection and the value for the previous one. The right boundary is located halfway between the value for the current intersection and the next one. The bin height is the number of accidents that occurred at that intersection. Hence, the value of the EIF at the right boundary of the current bin is the sum of all bin heights, from that for the lowest covariate value up to that boundary. For instance, consider the first three intersections on a list ordered by total entering vehicles. Suppose these experience 5,000, 10,000, and 15,000 total entering vehicles per day and 10, 12, and 15 accidents respectively. For the second bin, the left and right boundaries are equal to 7,500 (midway between 5,000 and 10,000) and 12,500 respectively and the height of the EIF at the right boundary is equal to 22 (=10+12). This would create a set of coordinates (12,500; 22) for the EIF graph. If several intersections were to have the same entering volume, then the EIF would become vertical and increase by the total number of accidents at those intersections. The goal of the method is to compare the EIF graph thus created with pre-established cumulative probability graphs of well-known functions (power, gamma, polynomial, etc.) in order to indicate the most appropriate relationship between the dependent variable and the candidate covariates. The plots on the right side of Figure 1 show the cumulative probability graphs for the power and gamma functions while Figures 2 and 3 show the EIF graphs for the major and minor road entering AADT for total accidents at 4-legged signalized intersections in the Toronto database. The shape of the graphs in Figures 2 and 3 is closest to the shape in Figure 1 for the gamma function, which is therefore the form of the relationship that seems appropriate. The actual functional form and the shape of the graph relating accidents to the covariates are indicated by the plots on the left hand side of Figure 1. 3

4 FIGURE 1 Corresponding f(x) and F(x) of power and Gamma functions The results of the exploratory analysis revealed that for all intersection classes, the relationship between accidents and each covariate could be described either by the Power function or the Gamma function. To determine which of these functions was more appropriate, each relationship was investigated in turn. The selected model forms are shown in equations 1 to 3. The results of the model fitting process were also used to confirm the validity of these selections. F 1 = Power, F 2 = Gamma: E{Κ} = α F 1 β1 F 2 β2 e (β3f2) 1) F 1 = Gamma, F 2 = Power: E{Κ} = α F 1 β1 F 2 β2 e (β4f1) 2) F 1 = Power, F 2 = Power: where, E{Κ} = α F 1 β1 F 2 β2 3) E{Κ} = the expected annual number of accidents for the period 1990 to 1995; F 1, F 2 = entering AADTs of the major and minor roads, averaged over the period 1990 to 1995; α, β 1, β 2, β 3, β 4 = coefficients to be estimated. 4

5 60000 Cumulative Accidents Major Road Entering AADT (F1) Figure 2: EIF for major road entering AADT for total accidents at 4-legged signalized intersections Cumulative Accidents Minor Road Entering AADT (F2) Figure 3: EIF for minor road entering AADT for total accidents at 4-legged signalized intersections The models were estimated with Genstat 5, Version 4.1 (8) which allows for the specification of a Negative Binomial (NB) error structure. The NB has been shown to more accurately describe the distribution of traffic accidents between sites (9-12) than the normal distribution assumed in conventional regression modelling. The overdispersion parameter γ of the negative binomial distribution, which is estimated iteratively in the model calibration process, is such that VAR{Κ} = E{Κ} 2 /γ 4) Since the variance decreases as γ increases, the value of γ can also be used to compare the goodness of fit of various models fitted to the same data, in that the larger the value of γ, the smaller would be the variance and the better the model. As a goodness of fit index, γ should suffice for accident prediction models, for which the question is: what is the model prediction estimate and how accurately is it known. Other measures are relevant, but not the traditional R 2 measure, according to Miaou (13), who has used simulation to demonstrate the pitfalls 5

6 of using this measure in accident prediction models. That research has suggested the use of R 2 α, a dispersion parameter based R 2, which is calculated as follows: R 2 α = 1 (γ min /γ) 5) Where γ is the over-dispersion parameter for the calibrated model and γ min is the smallest over-dispersion parameter possible, a value that is obtained by having no covariates in the model (by assuming that all sites have an identical accident prediction estimate that is equal to the mean over all sites). Since several years of data were used for the model estimation, the temporal correlation in the data needed to be accounted for, especially in the light of the year to year variation, or trend, in accident counts that is due to factors that change every year (14, 15). For this purpose, a general equations estimating (GEE) procedure (16) was used. Specifics of the application, for which the built-in GEE procedure of Genstat was utilized, are described by Lord & Persaud (6). Further details of the modeling can be found in (17). The estimated models are summarized in Table 2, along with the t-statistics and goodness of fit measures R 2 α and γ. Note that values of ln(") are provided; this is because GENSTAT fits a linearized form of Equations 1 to 3. In all cases except two ($2 for injury accidents at unsignalized 4-legged and $3 for injury accidents at signalized 4-legged) the coefficients are significant at the 5% level (t > 1.96). TABLE 2: Estimates of coefficients (and t-statistics) for Toronto APMs (accidents/year) Parameters Signalized 4-legged Signalized 3-legged Unsignalized 4-legged Unsignalized 3-legged All Injury All Injury All Injury All Injury # of intersections Accidents 54,989 16,339 7,214 2, Model Form Eq. 1 Eq. 1 Eq. 3 Eq. 3 Eq. 2 Eq. 3 Eq. 3 Eq. 1 LN(") (14.28) $ (13.02) $ (13.16) $3 8.92E-6 (2.24) (15.18) (12.44) (11.52) 6.94E-6 (1.64) (11.88) (10.16) (13.21) (11.76) (10.14) (8.19) (4.59) (3.63) (3.07) $4-2.29E-4 (2.04) (2.92) (3.31) (1.42) (4.91) (3.03) (10.22) (3.64) (2.88) (7.84) -8.90E-5 (2.15) ( R 2 α For each of these the model forms indicated by the other equations as well as others were also investigated. The additional ones included one proposed by Brüde et al. (18), in which the ratio of the minor to the total entering AADT was included in the model. In the end, the examination of the t-statistics and the values of R 2 α and γ were used to confirm that the selected forms were indeed the best. Another method used in this confirmation was the

7 cumulative residuals (CURE) method, as proposed by Hauer & Bamfo (7) in which the cumulative residuals (the difference between the actual and fitted values for each intersection) were plotted in increasing order for each covariate separately. The graph shows how well the model fits the data with respect to each individual covariate. Figure 4 illustrates the CURE plot for the covariate F 1 for the model for injury accidents at 4-legged signalized intersections. The indication is that the fit is very good for this covariate in that the cumulative residuals oscillate around the value of 0 and lie between the two standard deviation boundaries. Additional CURE plots are presented later in the paper in assessing the transferability of accident prediction models and in reference (17). Figure 4: CURE plot for injury accidents at 4-legged signalized intersections TRANSFERABILITY OF ACCIDENT PREDICTION MODELS Harwood et al. (5) present a calibration procedure for use in the IHSDM to adapt accident prediction models calibrated for one jurisdiction to suit the data of another jurisdiction. That report recommends that even a jurisdiction whose data were used to calibrate a model should consider applying the calibration procedure every 2 to 3 years because of changes in safety conditions over time. Fundamentally, accidents and model variable values for a sample of intersections are obtained for the new jurisdiction. A calibration factor is then obtained as the total number of accidents for the sample divided by the sum of the predicted accidents from the original base model. The model for the new jurisdiction is simply the original base model multiplied by the calibration factor. To test this procedure, we used Toronto intersection data as the sample for a new jurisdiction. We then calibrated recently published intersection models (3,4) for Toronto conditions. As a test of the calibration procedure, predictions from the Toronto models so calibrated were then compared to predictions from the models presented earlier in this paper. Harwood et al. (5) recommend that the sample for the new jurisdiction be such that the distribution of traffic volumes is similar to that in the data used for the original calibration. We were not able to accommodate this recommendation for the test that we conducted because the distribution of traffic volumes in the calibration data was unavailable. It was felt that this should not create a deterrent since the calibration factor, at least in the procedure as proposed, is independent of traffic volume and is applied to all 7

8 intersections regardless of the distribution of traffic volumes in the jurisdiction. Therefore, it is of interest to examine if the procedure would work for the likely situation where the distributions of traffic volumes in two jurisdictions are different. The models in Sayed & Rodriguez (3) pertain to accidents of all severities combined at urban unsignalized intersections in Vancouver, British Columbia. The Bauer & Harwood models (4) were for both signalized and unsignalized intersections in urban areas and were based on California data. These models had several variables for which the values were not available for the Toronto data. For these variables we used average values for California to derive models that could be applied to the Toronto data. Since traffic volumes are know to explain most of the variation in accident experience at intersections, it is expected that these derived models would in effect be similar to models calibrated for California data with AADT as the only variables. The model forms and parameter estimates for the models tested are shown in Table 3 along with the calibration factor estimated with the Harwood et al. procedure (5). In applying the Vancouver and California models to Toronto, the estimates from those models were multiplied by the appropriate calibration factor. Two visual tests were performed to see how well the recalibrated models work when compared to models calibrated directly from the Toronto data. First we plotted predictions from the three sets of models for selected minor AADTs. These are shown in Figures 5 to 8. Second, we examined plots of the cumulative residuals, in accordance with the CURE method suggested by Hauer & Bamfo (7). A sample of these plots is shown in Figures 9 and 10. We also estimated root mean squared prediction errors for the three sets of models. These are shown in Table 3. Table 3: Model forms, parameter estimates and calibration factors for California and BC models Parameter Jurisdiction Unsignalized 3-legged (All severities) Model Form " $1 $2 Calibration Factor Root mean squared error of prediction 8 Unsignalized 4-legged (All severities) Signalized 4-legged (Injury) Signalized 4-legged (All severities) California Equation 3 Equation 3 Equation 3 Equation 3 Vancouver Equation 3 Equation 3 Equation 3 Equation 3 Toronto Equation 3 Equation 2 Equation 1 Equation 1 California Vancouver n/a n/a Toronto California Vancouver n/a n/a Toronto California Vancouver n/a n/a Toronto California Vancouver n/a n/a Toronto California Vancouver n/a n/a Toronto

9 Accidents Per Year Toronto - F2=20000 California - F2=20000 California - F2=2000 Toronto - F2= Major Road Entering AADT (F1) Figure 5: APMs calibrated or recalibrated for Toronto 4-legged signalized intersections (all severities) for selected minor road entering AADTs (F2) Accidents Per Year Toronto - F2=20000 California - F2=20000 California - F2=2000 Toronto -F2= Major road entering AADT(F1) Figure 6: APMs calibrated or recalibrated for Toronto 4-legged signalized intersection injury accidents for selected minor road entering AADTs (F2) 9

10 Accidents Per Year Vancouver F2=6000 California F2=6000 Toronto F2=6000 California F2=1000 Toronto F2=1000 Vacouver F2= Major Road Entering AADT (F1) Figure 7: APMs calibrated and recalibrated for Toronto unsignalized 4-legged intersections (all severities) for selected minor road entering AADTs (F2) 7 Accidents Per Year Toronto F2=3000 Vancouver F2=3000 California F2=3000 California F2=500 Toronto F2=500 Vancouver F2= Major road entering AADT (F1) Figure 8: APMs calibrated or recalibrated for Toronto unsignalized 3-legged intersections (all severities) for selected minor road entering AADTs (F2) 10

11 Cumulative Residuals California Toronto Vancouver Major road entering AADT (F1) Figure 9: Cumulative residuals for major road AADT for 3-legged unsignalized intersection models Cumulative Residuals Vancouver Toronto California Minor road entering AADT (F2) Figure 10: Cumulative residuals for minor road AADT for 3-legged unsignalized intersection models 11

12 DISCUSSION The parameter estimates in Table 2 and the accident prediction plots in Figures 5-8 indicate that the models calibrated directly for the Toronto data are all quite reasonable in that the general shape of the graphs is consistent with that for other published models and that the value of the parameters ( and R 2 α are relatively high compared to that for similar modeling efforts by others. That the AADT exponents are positive and less than 1 is in accord with conventional wisdom as well as the results of similar research efforts. The CURE plots, a sample of which is shown in Figures 4, 9 and 10, all indicated that the fits for the directly calibrated models were generally reasonable over the entire range of the major and minor road AADTs. The results of the model transferability procedure tests are mixed. Figures 8 to 10 indicates that it would be reasonable to assume that the model calibrated for 3-legged unsignalized intersections in Vancouver can be recalibrated for application in Toronto using the Harwood et al. (5) procedure. That the root mean squared error value for the recalibrated model is similar to that for the original Toronto model would appear to support this assumption. The same might be said for California models for unsignalized 4-legged intersections. However, the California models for signalized intersections and for unsignalized 3-legged intersections do not appear to fare as well, generally predicting more accidents than the Toronto models when the minor road AADT is low. Similarly, the Vancouver unsignalized intersection models, which predicted well for 3-legged intersections, predict quite different accident frequencies than the Toronto 4-legged models, particularly for higher minor road AADTs. The quality of a model is undoubtedly an important determinant of how well it will transfer to other jurisdictions. This may partly explain why the B.C. 3-legged intersection model, with an over dispersion parameter γ of 2.34, fits the Toronto data better than the one for 4-legged intersections, for which the value of this parameter is smaller (2.17). These values were not available for the California models since the original data were not available to re-estimate them for the derived models. It can also be said that the model transfer procedure works best when the model form and AADT exponents for the other jurisdictions are similar to those calibrated for Toronto. This similarity is an assumption that is implicit in model transfer procedure. Since this assumption does not appear to hold in general, it would be desirable to modify the procedure to estimate calibration factors for the different traffic volume strata as opposed to the single factor proposed in Harwood et al. (5). Since stratification of the recalibration sample by traffic volume is already a recommended part of the procedure, this refinement could be relatively straightforward to implement, providing there are sufficient intersections in each strata. ACKNOWLEDGEMENT The research was partly supported by an operating grant from the Natural Sciences and Engineering Council of Canada (NSERC). This support is greatly appreciated as is the assistance of the city of Toronto Transportation Department who provided the data. REFERENCES 1. Hauer, E. Observational Before-After Studies in Road Safety: Estimating the Effect of Highway and Traffic Engineering Measures on Road Safety. Pergamon, Elsevier Science Ltd, Oxford, U.K., Persaud B.N., Lyon C. and T. Nguyen. Empirical Bayes procedure for ranking sites for safety investigation by potential for safety improvement. Transportation Research Record 1665, pp.7-12, (1999). 3. Rodriguez, L.-P., and Sayed, T. Accident Prediction Models for Urban Unsignalized Intersections in British Columbia. In Transportation Research Record 1665, TRB, National Research Council, Washington, D.C., pp ,

13 4. Bauer, K.M., and Harwood, D.W. Statistical Models of At-Grade Intersections - Addendum. FHWA-RD , U.S. Department of Transportation, March Harwood, D.W., Council, F.M., Hauer, E., et al. Prediction of the Expected Safety Performance of Rural Two- Lane Highways. FHWA-RD , U.S. Department of Transportation, December Lord, D., Persaud, B. Accident Prediction Models with and without Trend: Application of the Generalized Estimating Equation. In Transportation Research Record 1717, TRB, National Research Council, Washington, D.C., pp , Hauer, E., and Bamfo, J. Two Tools for Finding What Function links the dependent Variable to the Explanatory Variables. In Proceedings of the ICTCT 1997 Conference, Lund, Sweden., Payne, R.W., Lane, P.W., et al. Genstat 5: Release 3. Clarendon Press, Oxford, U.K., Hauer, E. Empirical Bayes Approach to the Estimation of Unsafety : the Multivariate Regression Method. Accident Analysis & Prevention, Vol. 24, No. 5, pp , Kulmala, R. Safety at Rural Three- and Four-Arm Junctions: Development and Applications of Accident Prediction Models. VTT Publications 233, Technical Research Centre of Finland, Finland, Nicholson, A., and Turner, S. Estimating Accidents in a Road Network. In Proceedings of Roads 96 Conference, Part 5, New Zealand, pp , Poch, M., and Mannering, F.L. Negative Binomial Analysis of Intersection Accident Frequencies. Journal of Transportation Engineering, Vol. 122, No. 2, pp , Miaou, et al. Pitfalls of Using R 2 to Evaluate Goodness of Fit of Accident Prediction Models. Transportation Research Record 1542, pp. 6-13, Maher, M., and Summersgill, I. A Comprehensive Methodology for the Fitting of Predictive Accident Models. Accident Analysis & Prevention, Vol. 28, No. 3, pp , Mountain, L., M.J. Maher, and B. Fawaz. The Influence of Trend on estimates of Accidents at Junctions, Accident Analysis & Prevention, Vol. 30, No. 5, pp , Liang, K.-Y., Zeger, S.L. Longitudinal Data Analysis Using Generalized Linear Models. Biometrika, Vol. 73, pp , Lord D. The Prediction of Accidents on Digital Networks: Characteristics and Issues Related to the Application of Accident Prediction Models. Ph.D. dissertation, University of Toronto, Brüde, U., Larsson, J., and Hegman, K.-O. Design of Major Urban Junctions-Accident Prediction Models and Empirical Comparison, VTI, Linköping, Sweden,

14

15 1