Accreditation of laboratories for the performance evaluation of CMMs according to the ISO

Transcription

1 Accreditation of laboratories for the performance evaluation of CMMs according to the ISO A. Balsamo a, R. Mugno b, E. Audrito a, D. Corona a, G.E. D'Errico a, P. Pedone a INRIM Italian National Research Institute of Metrology a Division of Mechanics, Precision Engineering Programme b SIT Servizio di Taratura in Italia a.balsamo@inrim.it 1

2 Outline Background and motivation The SIT/Tec-016/09 document Intent Overview on the issues covered Experimental auditing by comparisons History Set up Results Conclusions 2

3 Background The ISO :2001 is the most widely used ISO standard for CMMs It is about the performance verification of CMMs in acceptance and reverification testing It is intended to test the size measurement and probing performance of a CMM The size test is based on the repeated measurement of calibrated standards of size and the evaluation of the error of indication, E The probing test is based on the form measurement of a nearly perfect sphere and the evaluation of the probing error, P 3

4 Does the ISO provide traceability? The traceability of CMM measurements is a complex issue: At the scientific level (still debated) To achieve in practice (often requires expert professionals) There is lack of official documents on this issue (some are available, e.g. the ISO/TS series) The ISO is about testing only For acceptance at purchase For reverification afterwards, e.g. for quality assurance purposes However, the ISO is a minimum requirement for, and a good approximation to, traceability of CMM measurements Its practice is to be encouraged in industry 4

5 Accreditation of ISO testing Increasing demand of ISO testing in industry, often provided by professionals (CMM manufacturers or service providers) In this enlarging market, not all actors have got adequate competence to guarantee a good service A need exists for third party accreditation of the competence For a company to rely on its tested CMMs For formal documentation, e.g. in Quality Systems 5

6 SIT: accreditation in Italy The SIT (Servizio di Taratura in Italia) is the Italian official body for calibration accreditation according to the ISO/IEC The SIT was at the forefront in the accreditation of the ISO testing, starting 1998 Over these years, there was no ISO standard available on how to evaluate the test uncertainty; this was resolved in 2006 with the ISO/ TS Meantime, the accredited laboratories went on their own ways, resulting in non uniform accredited services In 2005, when the ISO/TS was about to be published, the need for a general reorganisation of this field of accreditation was clear 6

7 An SIT joint Working Group In 2005 a joint working group was established, made of representatives of Interested accredited laboratories SIT INRIM The goal was the preparation of a document: sound in science, practically viable, promoting fair competition among labs The document SIT/Tec-016/09 was published in 2009 (details in the written paper) 7

8 SIT/Tec-016/09 Accreditamento SIT di verifiche di prestazioni di CMM secondo la UNI EN ISO :2005 Requisiti tecnici SIT accreditation of CMM performance verification as per ISO :2001 Technical requirements 8

9 Intent and content The issues covered are Not interpretations of the ISO About specifics of implementing the ISO by accredited laboratories Stemming from years of practical experience; a list of unresolved issues had been compiled over time and resolved here A major issue was the experimental auditing of competence Before, only drills were run in the presence of an SIT assessor, but the actual measurement results were disregarded This was not in line with the SIT general and recognised approach, which values actual results most 9

10 An accredited lab is a third party The ISO/IEC clearly states that [4.1.5] The laboratory shall d) have policies and procedures to avoid involvement in any activities that would diminish confidence in its competence, impartiality, judgement or operational integrity; [4.1.4 Note 2] If the laboratory wishes to be recognized as a third-party laboratory, it should be able to demonstrate that it is impartial and that it and its personnel are free from any undue commercial, financial and other pressures which might influence their technical judgement. The third-party testing or calibration laboratory should not engage in any activities that may endanger the trust in its independence of judgement and integrity in relation to its testing or calibration activities. 10

11 while the ISO provides two parties In an ISO test there are provision for Two parties, in case of acceptance testing One party, in case of reverification testing Never there are for a third party Typical case is that of a CMM manufacturer which pursues accreditation for acceptance testing (first party testing) This is no different from existing cases of e.g. accredited gauge block manufacturers which sell calibrated gauges; it is just more delicate to handle 11

12 Countermeasures and allowances SIT exerts close surveillance as to the independence (e.g. organisation chart) The test outcome in a certificate may be all three cases: conformance, non conformance and uncertainty zone; no default decision rule as per ISO All procedural choices at the time of testing set by the ISO must be left to the customer; in case he is thinks not to be competent enough, the accredited lab must choose, according to an accredited written procedure A posteriori reassignment of MPE s based on actual test data is allowed, with limitations (mutual agreement, no violations of contracts and procedures, 25% error allowance for equity) 12

13 Material standards of size The ISO requires the standards to be calibrated Standards are subject to damage and ageing due to external transportation In addition, they are required to be accompanied by a data logger at all times to monitor their thermal history ad interim reverified on a regular basis, e.g. against equivalent equipment 13

14 Adjustment followed by testing Service providers are often contracted primarily for CMM maintenance, which may include adjustment of the parametric errors; the test just follows afterwards When this is done with the same standards used for testing, shadowing of some errors may occur At that temperature the CMM works fine, maybe not at others A viable solution to this problem was not found The only remedy is to raise awareness on this problem 14

15 Reference environment Labs are accredited to stated uncertainties They are officially filed and customers may base their choice on this The uncertainty is often dominated by the CTE, α u E ( α ) = L t 20 C u( α ) The average temperature, t, depends on the customer s site; when the accreditation is granted, it must be guessed Setting a conventional environmental temperature improves uniformity and fair competition 15

16 Reference environment /2 A single reference temperature though does not disclose the ability of the lab to deal with this, i.e. how accurately they know the CTE Two reference temperatures are set: 20 C (no effect) 25 C (large effect) Example accreditation (up to L = 1000 mm): U(α) = ± U(α) = ± K -1 K -1 U(E) = 0.4 µm at 20 C U(E) = 0.4 µm at 20 C 16

17 Experimental auditing Usually in accreditation, auditing is made by means of experimental comparisons Here, the measurand is a CMM performance, but a CMM cannot be circulated: need to convene at a same CMM Unlike done before, the base to pass/fail the audit must be the actual measurement results More details in the second part of this presentation 17

18 Multiple operator and multiple seat labs Particularly large companies may have more than one operator, and more than one seat Only the competence of a lab leader is usually assessed by the SIT In case of the ISO test, operators do not work under direct leader s control, as usual All operators are required to prove competence (courses, experience) and must be formally appointed by the leader All are assessed experimentally, either by the SIT or by the leader Multiple seat labs operate as single seat, i.e. same Quality System, leader having full control over all seats. 18

19 CMM software Several SW interfaces and languages exist in the market SIT lab operators may be familiar with some, not with all The flow of relevant operations in the part programme (e.g. probe qualification, alignment) are documented in the accredited written procedure Three alternative ways of developing the actual part programme are allowed Pick up from a repository (when compatible) Development on the spot (when the operator is familiar) With customer s support for low-level implementation under direction of the SIT lab operator (otherwise) 19

20 Experimental auditing of labs to prove competence 20

21 Motivation How reproducible the ISO test is was never evaluated experimentally How well the ISO/TS uncertainty budgeting describes practical implementations was never evaluated experimentally To align this field of accreditation to all others 21

22 Goal of the intercomparison To test how good the ISO/TS is in giving guidance in test uncertainty budgeting To find a scientifically sound and practically feasible way of experimentally proving candidate laboratories capabilities of performing the ISO test 22

23 The development A first trial was made in 2005 (Florence) 5 participant labs Preliminary, too short, all together Poor machine and environment were chosen, mistake Failure of the exercise, but lessons were learned A second trial was made later in 2005 (Venice) 6 participant labs Still too short, a lab at a time Better machine and environment, but still not good enough Failure again A third trial was made in 2007 (at the INRIM, Turin) 9 participant labs Half a day each (a full week overall) High accuracy CMM and lab environment Success and demonstration of the approach 23

24 Organisation The test was carried out on a same CMM Each lab brought its own equipment (size standards and supports, test sphere, thermometers) to simulate actual testing on the field A full week was the agreed time budget for the test; each lab was given half a day and allowed to bring its equipment the day before for acclimatisation The exercise was done under surveillance of, and with the support of, INRIM personnel 24

25 Lesson learned from previous failures The measurand is in fact a CMM performance; it is essential that this is kept stable over the comparison Same CMM Highly repeatable high accuracy CMM, to minimise accidental errors Very good environment, to minimise drifts due to a varying temperature As short exercise as possible, again to minimise drifts Same location in the CMM volume, to minimise proximity errors due to residual geometrical errors 25

26 Actual implementation A Leitz PMM-C at the INRIM was used; each lab input its own CTE One lab at a time, to minimise thermal load of the room Well conditioned lab, temperature in the range (20 ± 0.3) C Ideally, all labs measure the same points on a same line in the CMM volume Single line only to meet the time budget (instead of 7 as per the ISO ) x z y 26

27 Choice of nominal lengths Different standards (step gauges, gauge blocks) As close nominal lengths as possible Nominal length 1 / mm Nominal length 2 / mm Nominal length 3 / mm Nominal length 4 / mm Nominal length 5 / mm Nominal length 6 / mm

28 How stable is the measurand? In spite of any effort, the measurand is not invariant the variations must be assessed and accounted for in the evaluation of the comparison results Drift A same INRIM s (uncalibrated) step gauge was fixtured aside and measured after each lab, to evaluate the actual drift (over a week) Proximity errors (due to variations of residual geometrical errors) Different standards were used getting exactly the same points is impossible At the exercise end, the INRIM s gauge was measured repeatedly in slightly different positions and orientations, to mimic the labs variability about the target points 28

29 29

30 30

31 31

32 32

33 Drift and repeatability Ripetibilità e deriva CMM 0,80 0,60 scarti / µm 0,40 0,20 0,00-0,20-0, Deriva 5.1 Deriva 5.2 Deriva 5.3 Deriva 7.1 Deriva 7.2 Deriva 7.3 Deriva 8.1 Deriva 8.2 Deriva 8.3 Deriva 9.1 Deriva 9.2 Deriva σ σ -0,60 L / mm 33

34 Proximity effects 0,50 0,40 0,30 0,20 scarti / µ 0,10 0,00-0,10-0, Posiz. 1 Posiz. 2 Posiz. 3 Posiz. 4 Posiz. 5 Posiz. 6 Posiz. 7 + σ σ -0,30-0,40-0,50-0,60 L / mm 34

35 Evaluation of results No reference value available; INRIM cannot do significantly better than others to provide a reference The simple mean was chosen as reference value, with no outlier elimination Each lab followed its own usual procedure and evaluated its test uncertainty according to the ISO/TS Additional comparison uncertainty added to account for drift and proximity 35

36 Results from the labs / I 2,50 2,00 L = 20 mm - 30 mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 2,50 2,00 L = 100 mm mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 1,50 1,50 1,00 1,00 0,50 E / µm 0,50 0,00-0,50 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9 E / µm 0,00-0,50-1,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9-1,50-1,00-2,00-1,50-2,50-2,00-3,00 36

37 Results from the labs / II 3,00 2,00 L = 200 mm mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 3,00 2,00 L = 300 mm mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 1,00 1,00 E / µm 0,00-1,00 0,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9-1,00 E / µm -2,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9-2,00-3,00-3,00-4,00-4,00-5,00 37

38 Results from the labs / III 4,00 3,00 L = 500 mm mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 5,00 4,00 L = 800 mm mm Ripet. 1 Ripet. 2 Ripet. 3 + Uref - Uref + Umedia - Umedia 3,00 2,00 2,00 1,00 1,00 E / µm 0,00-1,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9 E / µm 0,00-1,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9-2,00-2,00-3,00-3,00-4,00-4,00-5,00-5,00-6,00 38

39 Probing error, P 2,50 Errore di tastatura P P Ref. + U(ref) Ref. Ref. - U(ref) 2,00 1,50 P / µm 1,00 0,50 0,00 Centro 1 Centro 2 Centro 3 Centro 4 Centro 5 Centro 6 Centro 7 Centro 8 Centro 9 39

40 Probing error, P This part of the exercise was given less preparation and resulted not well organised The MPE P = 0.6 µm is overestimated by the labs, while the CMM proved to be OK The labs were not fully acquainted with the software used, and may have made mistakes It looks like the labs using the slimmest test sphere stems overestimated most, possibly due to sphere rock induced by significant contact force of the proportional probe Definitely, this must be repeated with greater care (failure but confidence that it can be dominated with greater care) 40

41 Conclusions /1 The accreditation of ISO testing has been offered in Italy since 1998 Since 2006, the standard package is complete The need was felt to reorganise the accreditation in this field, to align it to others A joined working group was established and finalised a normative SIT document addressing accreditation specific issues One of the most difficult (and expensive) to solve was the experimental auditing of the lab competence, based on a comparison on a same CMM 41

42 Conclusions /2 The feasibility of performing such comparison has been demonstrated The ISO/TS seems to work well in most cases The results are satisfactory in most cases, and demonstrate that the application of the ISO is all but trivial Based on this experience, the exercise will be repeated, giving labs more time (maybe a day each), with a greater care given to the P test SIT is now ready to carry out a comparison for true accreditation purposes (no more drills) 42

43 Thank you for your attention! 43