Accreditation of laboratories for the performance evaluation of CMMs according to the ISO 10360-2 A. Balsamo a, R. Mugno b, E. Audrito a, D. Corona a, G.E. D'Errico a, P. Pedone a INRIM Italian National Research Institute of Metrology a Division of Mechanics, Precision Engineering Programme b SIT Servizio di Taratura in Italia a.balsamo@inrim.it 1
Outline Background and motivation The SIT/Tec-016/09 document Intent Overview on the issues covered Experimental auditing by comparisons History Set up Results Conclusions 2
Background The ISO 10360-2:2001 is the most widely used ISO standard for CMMs It is about the performance verification of CMMs in acceptance and reverification testing It is intended to test the size measurement and probing performance of a CMM The size test is based on the repeated measurement of calibrated standards of size and the evaluation of the error of indication, E The probing test is based on the form measurement of a nearly perfect sphere and the evaluation of the probing error, P 3
Does the ISO 10360-2 provide traceability? The traceability of CMM measurements is a complex issue: At the scientific level (still debated) To achieve in practice (often requires expert professionals) There is lack of official documents on this issue (some are available, e.g. the ISO/TS 15530 series) The ISO 10360-2 is about testing only For acceptance at purchase For reverification afterwards, e.g. for quality assurance purposes However, the ISO 10360-2 is a minimum requirement for, and a good approximation to, traceability of CMM measurements Its practice is to be encouraged in industry 4
Accreditation of ISO 10360-2 testing Increasing demand of ISO 10360-2 testing in industry, often provided by professionals (CMM manufacturers or service providers) In this enlarging market, not all actors have got adequate competence to guarantee a good service A need exists for third party accreditation of the competence For a company to rely on its tested CMMs For formal documentation, e.g. in Quality Systems 5
SIT: accreditation in Italy The SIT (Servizio di Taratura in Italia) is the Italian official body for calibration accreditation according to the ISO/IEC 17025 The SIT was at the forefront in the accreditation of the ISO 10360-2 testing, starting 1998 Over these years, there was no ISO standard available on how to evaluate the test uncertainty; this was resolved in 2006 with the ISO/ TS 23165 Meantime, the accredited laboratories went on their own ways, resulting in non uniform accredited services In 2005, when the ISO/TS 23165 was about to be published, the need for a general reorganisation of this field of accreditation was clear 6
An SIT joint Working Group In 2005 a joint working group was established, made of representatives of Interested accredited laboratories SIT INRIM The goal was the preparation of a document: sound in science, practically viable, promoting fair competition among labs The document SIT/Tec-016/09 was published in 2009 (details in the written paper) 7
SIT/Tec-016/09 Accreditamento SIT di verifiche di prestazioni di CMM secondo la UNI EN ISO 10360-2:2005 Requisiti tecnici SIT accreditation of CMM performance verification as per ISO 10360-2:2001 Technical requirements 8
Intent and content The issues covered are Not interpretations of the ISO 10360-2 About specifics of implementing the ISO 10360-2 by accredited laboratories Stemming from years of practical experience; a list of unresolved issues had been compiled over time and resolved here A major issue was the experimental auditing of competence Before, only drills were run in the presence of an SIT assessor, but the actual measurement results were disregarded This was not in line with the SIT general and recognised approach, which values actual results most 9
An accredited lab is a third party The ISO/IEC 17025 clearly states that [4.1.5] The laboratory shall d) have policies and procedures to avoid involvement in any activities that would diminish confidence in its competence, impartiality, judgement or operational integrity; [4.1.4 Note 2] If the laboratory wishes to be recognized as a third-party laboratory, it should be able to demonstrate that it is impartial and that it and its personnel are free from any undue commercial, financial and other pressures which might influence their technical judgement. The third-party testing or calibration laboratory should not engage in any activities that may endanger the trust in its independence of judgement and integrity in relation to its testing or calibration activities. 10
while the ISO 10360-2 provides two parties In an ISO 10360-2 test there are provision for Two parties, in case of acceptance testing One party, in case of reverification testing Never there are for a third party Typical case is that of a CMM manufacturer which pursues accreditation for acceptance testing (first party testing) This is no different from existing cases of e.g. accredited gauge block manufacturers which sell calibrated gauges; it is just more delicate to handle 11
Countermeasures and allowances SIT exerts close surveillance as to the independence (e.g. organisation chart) The test outcome in a certificate may be all three cases: conformance, non conformance and uncertainty zone; no default decision rule as per ISO 14253-1 All procedural choices at the time of testing set by the ISO 10360-2 must be left to the customer; in case he is thinks not to be competent enough, the accredited lab must choose, according to an accredited written procedure A posteriori reassignment of MPE s based on actual test data is allowed, with limitations (mutual agreement, no violations of contracts and procedures, 25% error allowance for equity) 12
Material standards of size The ISO 10360-2 requires the standards to be calibrated Standards are subject to damage and ageing due to external transportation In addition, they are required to be accompanied by a data logger at all times to monitor their thermal history ad interim reverified on a regular basis, e.g. against equivalent equipment 13
Adjustment followed by testing Service providers are often contracted primarily for CMM maintenance, which may include adjustment of the parametric errors; the test just follows afterwards When this is done with the same standards used for testing, shadowing of some errors may occur At that temperature the CMM works fine, maybe not at others A viable solution to this problem was not found The only remedy is to raise awareness on this problem 14
Reference environment Labs are accredited to stated uncertainties They are officially filed and customers may base their choice on this The uncertainty is often dominated by the CTE, α u E ( α ) = L t 20 C u( α ) The average temperature, t, depends on the customer s site; when the accreditation is granted, it must be guessed Setting a conventional environmental temperature improves uniformity and fair competition 15
Reference environment /2 A single reference temperature though does not disclose the ability of the lab to deal with this, i.e. how accurately they know the CTE Two reference temperatures are set: 20 C (no effect) 25 C (large effect) Example accreditation (up to L = 1000 mm): U(α) = ± 1.5 10-6 U(α) = ± 0.2 10-6 K -1 K -1 U(E) = 0.4 µm at 20 C U(E) = 0.4 µm at 20 C 16
Experimental auditing Usually in accreditation, auditing is made by means of experimental comparisons Here, the measurand is a CMM performance, but a CMM cannot be circulated: need to convene at a same CMM Unlike done before, the base to pass/fail the audit must be the actual measurement results More details in the second part of this presentation 17
Multiple operator and multiple seat labs Particularly large companies may have more than one operator, and more than one seat Only the competence of a lab leader is usually assessed by the SIT In case of the ISO 10360-2 test, operators do not work under direct leader s control, as usual All operators are required to prove competence (courses, experience) and must be formally appointed by the leader All are assessed experimentally, either by the SIT or by the leader Multiple seat labs operate as single seat, i.e. same Quality System, leader having full control over all seats. 18
CMM software Several SW interfaces and languages exist in the market SIT lab operators may be familiar with some, not with all The flow of relevant operations in the part programme (e.g. probe qualification, alignment) are documented in the accredited written procedure Three alternative ways of developing the actual part programme are allowed Pick up from a repository (when compatible) Development on the spot (when the operator is familiar) With customer s support for low-level implementation under direction of the SIT lab operator (otherwise) 19
Experimental auditing of labs to prove competence 20
Motivation How reproducible the ISO 10360-2 test is was never evaluated experimentally How well the ISO/TS 23165 uncertainty budgeting describes practical implementations was never evaluated experimentally To align this field of accreditation to all others 21
Goal of the intercomparison To test how good the ISO/TS 23165 is in giving guidance in test uncertainty budgeting To find a scientifically sound and practically feasible way of experimentally proving candidate laboratories capabilities of performing the ISO 10360-2 test 22
The development A first trial was made in 2005 (Florence) 5 participant labs Preliminary, too short, all together Poor machine and environment were chosen, mistake Failure of the exercise, but lessons were learned A second trial was made later in 2005 (Venice) 6 participant labs Still too short, a lab at a time Better machine and environment, but still not good enough Failure again A third trial was made in 2007 (at the INRIM, Turin) 9 participant labs Half a day each (a full week overall) High accuracy CMM and lab environment Success and demonstration of the approach 23
Organisation The test was carried out on a same CMM Each lab brought its own equipment (size standards and supports, test sphere, thermometers) to simulate actual testing on the field A full week was the agreed time budget for the test; each lab was given half a day and allowed to bring its equipment the day before for acclimatisation The exercise was done under surveillance of, and with the support of, INRIM personnel 24
Lesson learned from previous failures The measurand is in fact a CMM performance; it is essential that this is kept stable over the comparison Same CMM Highly repeatable high accuracy CMM, to minimise accidental errors Very good environment, to minimise drifts due to a varying temperature As short exercise as possible, again to minimise drifts Same location in the CMM volume, to minimise proximity errors due to residual geometrical errors 25
Actual implementation A Leitz PMM-C 12107 at the INRIM was used; each lab input its own CTE One lab at a time, to minimise thermal load of the room Well conditioned lab, temperature in the range (20 ± 0.3) C Ideally, all labs measure the same points on a same line in the CMM volume Single line only to meet the time budget (instead of 7 as per the ISO 10360-2) x z y 26
Choice of nominal lengths Different standards (step gauges, gauge blocks) As close nominal lengths as possible Nominal length 1 / mm 20 25 30 Nominal length 2 / mm 100 125 130 140 Nominal length 3 / mm 200 210 220 225 Nominal length 4 / mm 300 310 325 Nominal length 5 / mm 500 510 525 Nominal length 6 / mm 800 810 820 825 27
How stable is the measurand? In spite of any effort, the measurand is not invariant the variations must be assessed and accounted for in the evaluation of the comparison results Drift A same INRIM s (uncalibrated) step gauge was fixtured aside and measured after each lab, to evaluate the actual drift (over a week) Proximity errors (due to variations of residual geometrical errors) Different standards were used getting exactly the same points is impossible At the exercise end, the INRIM s gauge was measured repeatedly in slightly different positions and orientations, to mimic the labs variability about the target points 28
29
30
31
32
Drift and repeatability Ripetibilità e deriva CMM 0,80 0,60 scarti / µm 0,40 0,20 0,00-0,20-0,40 0 100 200 300 400 500 600 700 800 900 Deriva 5.1 Deriva 5.2 Deriva 5.3 Deriva 7.1 Deriva 7.2 Deriva 7.3 Deriva 8.1 Deriva 8.2 Deriva 8.3 Deriva 9.1 Deriva 9.2 Deriva 9.3 + σ σ -0,60 L / mm 33
Proximity effects 0,50 0,40 0,30 0,20 scarti / µ 0,10 0,00-0,10-0,20 0 100 200 300 400 500 600 700 800 900 Posiz. 1 Posiz. 2 Posiz. 3 Posiz. 4 Posiz. 5 Posiz. 6 Posiz. 7 + σ σ -0,30-0,40-0,50-0,60 L / mm 34
Evaluation of results No reference value available; INRIM cannot do significantly better than others to provide a reference The simple mean was chosen as reference value, with no outlier elimination Each lab followed its own usual procedure and evaluated its test uncertainty according to the ISO/TS 23165 Additional comparison uncertainty added to account for drift and proximity 35
Results from the labs / I 2,50 2,00 L = 20 mm - 30 mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 2,50 2,00 L = 100 mm - 140 mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 1,50 1,50 1,00 1,00 0,50 E / µm 0,50 0,00-0,50 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9 E / µm 0,00-0,50-1,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9-1,50-1,00-2,00-1,50-2,50-2,00-3,00 36
Results from the labs / II 3,00 2,00 L = 200 mm - 225 mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 3,00 2,00 L = 300 mm - 325 mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 1,00 1,00 E / µm 0,00-1,00 0,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9-1,00 E / µm -2,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9-2,00-3,00-3,00-4,00-4,00-5,00 37
Results from the labs / III 4,00 3,00 L = 500 mm - 525 mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 5,00 4,00 L = 800 mm - 825 mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 3,00 2,00 2,00 1,00 1,00 E / µm 0,00-1,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9 E / µm 0,00-1,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9-2,00-2,00-3,00-3,00-4,00-4,00-5,00-5,00-6,00 38
Probing error, P 2,50 Errore di tastatura P P Ref. + U(ref) Ref. Ref. - U(ref) 2,00 1,50 P / µm 1,00 0,50 0,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9 39
Probing error, P This part of the exercise was given less preparation and resulted not well organised The MPE P = 0.6 µm is overestimated by the labs, while the CMM proved to be OK The labs were not fully acquainted with the software used, and may have made mistakes It looks like the labs using the slimmest test sphere stems overestimated most, possibly due to sphere rock induced by significant contact force of the proportional probe Definitely, this must be repeated with greater care (failure but confidence that it can be dominated with greater care) 40
Conclusions /1 The accreditation of ISO 10360-2 testing has been offered in Italy since 1998 Since 2006, the standard package is complete The need was felt to reorganise the accreditation in this field, to align it to others A joined working group was established and finalised a normative SIT document addressing accreditation specific issues One of the most difficult (and expensive) to solve was the experimental auditing of the lab competence, based on a comparison on a same CMM 41
Conclusions /2 The feasibility of performing such comparison has been demonstrated The ISO/TS 23165 seems to work well in most cases The results are satisfactory in most cases, and demonstrate that the application of the ISO 10360-2 is all but trivial Based on this experience, the exercise will be repeated, giving labs more time (maybe a day each), with a greater care given to the P test SIT is now ready to carry out a comparison for true accreditation purposes (no more drills) 42
Thank you for your attention! 43