Air Force Calibration Interval Analysis of Test, Measurement and Diagnostic Equipment (TMDE) Based on Maintenance Data Collection (MDC) Speaker: Scott Brockway AFMETCAL 813 Irving-Wick Dr. W. Bldg 2 Heath Oh, 43056-1199 (740) 788-5146 Fax (740) 788-5021 Co-Author: Tom Gorman Air Force Metrology and Calibration Program Abstract This paper will present the way the Air Force calculates its calibration intervals based on data collected through various MDC systems. Calibration intervals require periodic adjustment to ensure the Air Force reliability goal for measurement standards is maintained. The 85 percent end-of-period goal provides the Air Force with accurate TMDE to support all weapons systems. Intervals are adjusted utilizing a combination of statistical analysis based on the maintenance data submitted and the technical expertise of the TMDE s responsible technical content manager (TCM). The recommended interval changes are reviewed before approval to ensure there aren t any factors present, which could invalidate data responsible for causing the changes. The Air Force currently uses a trio of MDC combinations (Calibration Condition Received, Action Taken, Calibration Condition Returned) enabling Precision Measurement Equipment Laboratory s (PMEL) identification of needed local information while at the same time identifying all pass/fail scenarios for use during calibration interval analysis. The Air Force technique makes use of the most current processes, configurations, specifications and standards by utilizing a current group of data tied to the interval itself and the negative exponential distribution identified in the NCSLI RP-1. 1. Introduction. The Air Force supports over 620,000 pieces of TMDE which generates over 560,000 calibration actions annually. The MDC data is acquired and analyzed for the purpose of maintaining reliability and adjusting calibration intervals for TMDE used by the Air Force and indexed in Technical Order (T.O) 33K-1-100-2 and Calibration Measurement Summaries (CMS). The Air Force Precision Measurement Equipment Laboratories (PMELs) currently record MDC data in 3 different management information systems (MIS). The PMEL Automated Management System (PAMS) is used by 105 locations, the Facilities and Equipment Maintenance System (FEMS) is used at 3 locations and LabMet is used by 2 sites. Having three unique systems collecting the MDC data adds to the complexity of performing the calibration analysis. Once the data is collected into a single database it is manipulated with a series of manual and automated techniques to produce a listing of TMDE that are candidates for a calibration interval change.
There are three separate calibration interval analyses performed each year. Twice a year, in April and October, a regular calibration interval analysis for TMDE is performed. The timing of the analysis is in sync with the scheduled publication dates of the Air Force Calibration Index T.O. 33K-1-100-2. Any TMDE item with 20 or more MDC actions within its current calibration interval time frame from the end of the reporting period is initially considered a candidate for change in calibration interval during a regular analysis. Additionally, an analysis of low volume items is performed once a year in August or at a designated point between publications of T.O. 33K-1-100-2. The analysis of low volume items involves a data screen to look for less than 20 MDC actions during the most recent calibration interval analysis period, no calibration interval changes over a 5 year period, and for a total of 30 or more MDC actions over the current group plus the historical group (5 years). The low volume item calibration interval analysis is for items with Air Force quantities too low to achieve statistical significance during routine calibration interval analysis. 2. Calibration Interval Analysis Process. 2.1 Extracting and collating data Once the MDC data has been collected from the three systems it is imported into a Microsoft Access database. MDC data files are now in one table with a common data structure and are ready for processing. The MDC data is then filtered to remove all records not used in the calibration interval analysis process; these include all maintenance actions that the management information system tracks for the local laboratory managers but do not have anything to do with calibration intervals. 2.2 Determining the recommended calibration interval during a regular analysis The MDC data is broken into two groups, a current group and a historical group. The current group consists of data gathered during the current calibration interval of the item. For example, an item on a 12 month calibration interval would have a current group of MDC data based on the last 12 months of MDC data. The historical group consists of the last 5 years of MDC data minus its current calibration interval. So an item with a 12 month calibration interval would have a historical group which consists of MDC data greater than 12 months old but less than 5 years old. Any TMDE item with 20 or more MDC actions within its current calibration interval group will be initially considered a candidate for change in calibration interval during a regular analysis. Items with less than 30 actions in its current group will be noted as not having as robust statistical significance as those having 30 or more. To determine if a MDC record is a Pass or Fail for analysis purposes a trio of codes is used; Calibration Condition Received, Action Taken and Calibration Condition Returned. The fields Calibration Condition Received and Calibration Condition Returned were added to separate the calibration condition from the repair condition of an item. The valid combinations of codes that are used for calibration interval analysis purposes are listed in Table 1. ly only the LabMet MIS forces a technician to enter a valid combination of codes. The next releases of PAMS and FEMS will also have this capability which will greatly improve the data integrity in the process.
Calibration Condition Received Action Taken Calibration Condition Returned Pass / Fail NO ADJUSTMENT Pass TEST/INSPECT NON AF Calibration Pass NO ADJUSTMENT Pass REPAIRED Pass TEST/INSPECT NON AF Calibration Pass ADJUSTED Pass REPAIRED Fail REPAIRED ADJUSTED Fail ADJUSTED REPAIRED NOT CALIBRATED Fail BENCH CHECKED- NRTS - CONDEMNED NOT CALIBRATED Fail REPAIRED UNKNOWN or Not applicable REPAIRED Fail UNKNOWN or Not applicable REPAIRED REPAIRED Fail Table 1 Using the negative exponential method below the recommended calibration interval is determined: R(t) = exp(-kt) where k is defined as the out of tolerance rate, R is the recommended calibration interval and t is time Let R* be the TMDE reliability and I equal the current interval. Failure rate parameter defined as: k = - (1/I) ln R* If the TMDE reliability target is r* (AFMETCAL = 85) the interval is calculated as: Interval = - (1/k) ln r* = I (ln r*/ln R*) Recommended calibration interval = current calibration interval ((ln(.85)/ln(r*)) R* = (good calibrations/total calibrations)
An example using actual data is listed in Table 2. It should be noted that this item is relatively new to the inventory which explains the low number of historical records. At this point items with less than a +/- 10 percent change in the new recommended calibration interval will be considered a random fluctuation and dropped from the candidates list for change in calibration interval. For the remaining items an algorithm is used to provide recommended calibration intervals ranging from 3 to 60 months in multiples of three months as well as the raw exponential recommendation. Raw Exponential Recommendation, = current cal interval((ln(.85)/ln(c urrent Pass/ Raw Exponential Recommendation, Historical = current cal interval((ln(.85)/ln(his torical Pass/Historical Total)) PART NUMBER WUC CAL INT Pass Total Historical Pass Historical Total Total)) Recommended Interval (Multiple of 3) A812-100-4 YZC03 8 23 25 11 12 15.59 15 14.94 Table 2 2.3 Determining the recommended calibration interval during a low volume analysis In order to look at the reliability data for items with a low number of MDC actions, a low volume calibration interval analysis was created. The analysis of low volume items will involve a data screen to look for TMDE with less than 20 MDC actions during the most recent calibration interval analysis period, no calibration interval changes over a 5 year period, and a total of 30 or more MDC actions over the current group plus the historical group (5 years). The calculation for the recommended calibration interval is the same as the regular analysis except that the current group and historical groups are combined into one group. This low volume group is subject to a pass rate of greater than 90% or less than 80% (over the 5 year period) to allow for flexibility due to a loss of statistical significance. A report is then generated just as in the regular analysis and the remaining steps for the low volume calibration interval analysis are the same as the regular analysis. 2.4 Making the final calibration interval determination After all the numbers are crunched and a report similar to Table 2 above is generated it is up the Measurement Area s Technical Content Manager (TCM) to make the actual change to the calibration interval. TMDE items having a recent increase of calibration interval are generally not considered ready for a calibration interval change until a time frame equal to 1½ times, in months, of their new calibration interval. Also, TMDE items having a recent decrease of calibration interval are generally not considered ready for a calibration interval change until a time frame equal to the new calibration interval. USAF recommended calibration intervals range from 3 to 60 months. It is also recommended to set intervals in multiples of 3, due to the prevalence of calibration systems containing multiple instruments. When making a final calibration interval determination, the TCM may adjust the calibration interval in 1 month increments for intervals less than 9 months. An increase of a calibration interval should not be more than 2 times its current calibration interval and a decrease
of calibration interval should not be less than ½ of its current calibration interval. There are cases when a particular piece of TMDE has a calibration interval that is set by regulatory statute, such as radiation dosimeters. In these situations the TCM must use the documented calibration interval requirement. Once the TCM s have made the final determinations for all MDC report candidates and their supervisor has approved all updates, the changes are incorporated into T.O. 33K-1-100-2 or applicable CMS database. Changing a calibration interval out of cycle is limited as much as possible; however there are times when the TCM believes there is a problem with the current calibration interval. For these cases an out-of-cycle calibration interval analysis is performed. Out-of-cycle calibration interval changes must be approved by an engineering supervisor. Initial calibration intervals are established using available technical data, as well as reviewing previously established calibration intervals for similar type equipment. When an interval cannot be established using either method, a 12-month interval is used. 3. Summary Calibration interval analysis is an important part of the Air Force Metrology and Calibration Program, whose mission is to ensure Air Force systems and equipment are safe, accurate, uniform, reliable and traceable to National Standards through the Air Force Primary Standards Laboratory. The Air Force calibration interval analysis process goal is to ensure that an 85 percent end-of-period goal is achieved to the best of its ability with the available MDC data. The combination of using the negative exponential statistical method identified in the RP-1 to obtain a recommended calibration interval with the expertise of the TCM making the final determination has served the Air Force well for many years. The most recent calibration interval analysis conducted by AFMETCAL in April 2008 changed the calibration intervals of 64,672 TMDE items, decreasing the Air Force calibration workload by 1,818 calibrations per year. There is a current plan to consolidate the three different MIS into one system for the entire Air Force by the year 2015. This would streamline the initial tasks of gathering and collating the MDC data. It should also help with data integrity and further optimize the use of calibration interval analysis in the Air Force.