DOE Golfer Experiment A Design of Experiments Report Travis Anderson Jake Munger Deshun Xu 11/11/2008
INTRODUCTION We used Response Surface Methodology to optimize a golf putter. A face centered Central Composite Design was used with a number of input factors. A target was selected at random, and we used our model to predict which settings were needed to hit the golf ball as close as possible to the target. METHODS AND ANALYSIS Independent variables The independent variables that we thought were important are listed in Table 1. The variables marked with the * are the variables that we included in the design. The ball could have had an impact but we decided to assume that the balls would similar and that they would probably not be significant. Table 1 Independent Variables Considered *Putter length *Head weight *Arc angle *Location *Putter Ball Dependant Variables The dependant variable that was measured was the distance the ball traveled when putted. The units used to measure the distance were centimeters. Response Surface Design The five factor face centered central composite design was used to model the response surface. The central composite design was chosen because it allowed for a half factorial design to be run first. If curvature was not found to be important then alpha point did not need to be tested which would save twelve runs. A five factor face centered central composite design also has fewer runs than the five factor Box Behnken design. The experiment was run in two blocks. The first block was the half factorial test which found curvature to be significant. Because curvature was found to be significant the second block with the face centered alpha points was ran. Appendix A shows the coded X s for the face centered composite design. Appendix A also shows the random run order. The exception to the random run order is that the center points were spread uniformly throughout the block. Because we were using a face centered design we knew
that every run was feasible so it was not necessary to cut down the factors in order to accommodate any infeasible points. Model calculation To calculate the response surface model a regression analysis was run on the data collected. Table 2 outlines the significant factors. Table 2 Significant model factors for the full model Coefficients St. Error t Stat P-value Length X1 98.667 8.462 11.660 0.000 Head weight X2 35.167 8.462 4.156 0.001 Angle X3 207.500 8.462 24.521 0.000 Location 40.500 8.462 4.786 0.000 X1X3 64.938 8.975 7.235 0.000 X2X3 28.813 8.975 3.210 0.005 X32 69.854 22.878 3.053 0.008 Intercept 291.278 9.865 29.526 0.000 Model Verification To check the model we set the weight and length and calculated the angle to achieve a set distance. We found that the model was fairly accurate at higher distances but not very accurate at lower distances. We calculated the lack of fit, and found that the model had significant lack of fit. Although location was found to be significant, there was no way that we could control the location or include every possible location in the model so we decided to create a simpler model using just the length, head weight and angle. The regression analysis is summarized in Table 3. Table 3 Three factor model Factor Coefficients St. Error t Stat P-value Intercept 289.51 13.51 21.43 0.00 Length X1 98.67 11.72 8.42 0.00 Head weight X2 35.17 11.72 3.00 0.01 Angle X3 207.50 11.72 17.70 0.00 X12-49.66 29.41-1.69 0.11 X22-16.16 29.41-0.55 0.59 X32 59.84 29.41 2.03 0.054 X1X2 18.31 12.44 1.47 0.16 X1X3 64.94 12.44 5.22 0.00 X2X3 28.81 12.44 2.32 0.03 X1X2X3 12.81 12.44 1.03 0.31
We took some more sample runs and found that this new model was little more accurate at the shorter distances but that it still was not very accurate at longer distances. If the X 3 2 was included in the model it improved the model at longer distances but made the model worse at shorter distances. Therefore we decided to use the simplest model that we know would work. We knew that we could reach all of the distances if we used the long length and the heavy weight. Therefore we kept the weight and length at their high values and collected data at three locations for the low, mid and high angle values. From this we fit a quadratic equation that is based only on angle as shown in equation 1. In this equation the angle is measured in degrees and the distance is cm. This new simpler model was able to more accurately predict the distance the golf ball would travel at both low and high angles. Distance = 0.0667(angle) 2 + 4.235(angle) 81.679 (1) RESULTS Using the simplified model and a target distance of 750 cm an angle of 84 degrees was found to be the angle needed to reach to target. The calibration put resulted in the ball traveling the correct distance from the putter but it had a little curve which made the total distance from the target rather large. The putter was re aimed for the test runs. Table 4 outlines the results of the four test runs. The mean distance from the target was 35.9 cm and the standard deviation was 9.06 cm which resulted in a score of 37. Table 4 Test run results Run Distance 1 37.4 2 36 3 46 4 24 LESSONS LEARNED Simplicity We learned many lessons in doing this lab. First off, we learned the importance and the benefit of keeping your model as simple as possible. When we first started out, we were going to test six factors (putter length, head weight, angle, location, putter, and ball). We scratched out the ball and created our initial design and performed the experiment with the other five factors. Once we obtained our model, we discovered there was significant lack of fit, rendering our model inadequate. We decided to try to simplify our model and changed it to include only three factors (putter length, angle, and head weight). We used the runs we did initially at different locations as replication points to help us better estimate the noise and build more robustness into the model. This time, evidence suggested we did not have lack
of fit. We did a few verification runs, and discovered our model, while adequate, still wasn t very accurate. We then determined if our goal in doing the experiment was to hit a target with a ball, we could even further simplify our model. We fixed the putter length and the head weight, and performed a new set of experiments in various locations varying just the angle. This gave us an adequate and much more accurate model! It is important to keep the model as simple as possible. Not only does it make the data collection and analysis much easier, it (at least in this case) also gives a better representation of what s really happening. Rather than try to account for every possible variable, keep it simple! Use your calibration shot to adjust your model for things like location and putter and ball, but keep your model simple! Variation As soon as we began performing the experiment, we discovered there was a LOT of variation (sometimes as much as 50%!) in our measurements! So we tried to eliminate and/or control the sources of variation. We started using a square edge to perpendiculate the arm holding the putter with the base. We created a run procedure. We used the same operator and the same measurer each time. We placed the ball to be hit in the same place each time. The angle was measured the same way each time, and the putter was released in the same manner each time. We used landmarks to assure we were using precisely the same location each time. We were surprised to see how much of a difference these little things made! If we were to do this experiment again, we would try harder to identify the causes of variation. We would consider other possible sources of variation (such as curvature in the floor, carpet wear, measurement variation, putter arm wobbling, etc.). We would do a variation analysis on these factors to determine which one(s) cause the most variation, and then take further steps and precautions to eliminate or better control those particular sources of variation. Planning We learned the importance of planning your experiment in advance. In this area, we did some things really well, and some things we could have done much better. We created design (a five factor, facecentered Central Composite Design) and created our spreadsheet to record the data. We put in some dummy data to assure we could run the analysis correctly. We setup the spreadsheet to test for curvature on our factorial points, letting us know real time whether or not we would have to run the star points. It was all pretty slick. Except We did not do a pilot study; we simply assumed we knew enough about the process to determine which factors might be significant. A pilot study would have helped us tremendously by eliminating some of those factors (and perhaps bringing our attention to others). This potentially could have helped us
create a much simpler design right from the beginning. We also were clueless as to what values we should use for the high and low factorial levels, and a pilot study would have given us an idea of the operating range of the putter. Lack of Fit We learned a great deal about lack of fit principles. When we first obtained our model, we didn t do a lack of fit test. We were so excited to finally have our model created that we ran off to verify our model with some test prediction runs. And our model was awful! Was that because there was too much noise in the process? Or did our model surface not correctly represent the response? We later performed a lack of fit test and discovered in our full five factor design, there was significant lack of fit! Our model did not fit the data very well at all. Disappointed, we had to modify our model. We create some models that were better at longer distances and others that were more accurate at shorter distances. Our final model did not have lack of fit at either long or short distances. Now we were finally ready to test it out with some prediction runs!