Simulaion Validaion Mehods J. PELLETTIERE* Federal Aviaion Adminisraion, Washingon DC Absrac Modeling and simulaion is increasingly being used o represen occupan behavior, boh of human subjecs and of Anhropomorphic Tes Devices. Fuure rends are owards he applicaion of cerificaion by analysis whereby a produc, such as an aircraf sea, can be parially cerified hrough analyical means ha evaluaes he occupan responses o dynamic evens. If cerificaion by analysis is o be uilized, hen an objecive mehod of deermining model validiy is needed so ha he cerificaion auhoriy can make an objecive deerminaion of he model accepabiliy. Jus using exper opinion could inroduce bias and lead o inconsisen resuls depending on he paricular engineer or office consuled. This objecive mehod mus be robus and able o discriminae differen feaures of he resuling es daa such as magniude, phase, and pulse shape. Anoher consideraion is in developing he baseline comparison daa. In any esing, here is some inheren uncerainy and variaion in he measured responses. If jus an average is calculaed, he resul may no be represenaive of he rue response as some feaures may ge smoohed ou. Likewise, calculaing sandard deviaions around his average has similar issues. Each individual response should be reaed as a separae case. A mehod derived from Sprague and Geers has been developed and applied o occupan models of sea simulaions. Using he es daa, accepable error levels can be calculaed ha represen he uncerainy and provide consisen and objecive resuls. Keywords: Validaion mehods and merics, Cerificaion by analysis, Occupan injury, Aircraf sea. Inroducion Injury research is crucial o proecing hose who are exposed o dynamic environmens, such as a pilo during an ejecion or an occupan in an auomoive crash or aviaion acciden. Figure FAA Dynamic Sea Tess The FAA has a number of sandards and regulaions ha are designed o proec occupans in he even of a crash. As par of hese regulaions, dynamic esing and occupan injury assessmen have been required for seas in newly cerified aircraf since he adopion of 4 CFR 5.56 and similar regulaions in pars 3 and 5, 7, and 9 (USC Tile 4). There are wo ess ha mus be conduced. Tes is a primarily verical impac es wih a minimum impac velociy of 35 fps wih peak acceleraion of 4 G s and an impac angle of 30 degrees off verical. Tes is a fronal es wih a minimum impac velociy of 44 fps wih peak acceleraion of 6 G s and an impac angle of 0 degrees yaw. Boh ess have limis on he rise ime wih associaed injury crieria ha mus be me before a sea is cerified for use in aviaion. Oher aircraf caegories have similar requiremens. The injury merics include limis on lumbar and leg loads, head injury crierion (HIC), shoulder srap loads when used, and he requiremen ha bels remain in place. For complee deails, please see he applicable regulaions. While compliance wih hese regulaions is ypically me hrough dynamic *Corresponding auhor. Email: joseph.pelleiere@faa.gov
esing, he regulaions as saed do allow for compuer modeling and simulaion (M&S) in suppor of a dynamic es program. Secion 5.56(b) saes ha Each sea ype design approved for crew or passenger occupancy during akeoff and landing mus successfully complee dynamics ess or be demonsraed by raional analysis based on dynamic ess of a similar ype sea I is he demonsraion by raional analysis ha led o he developmen of AC 0-46 (FAA 003) for use in par 3, 5, 7, 9 airplanes and roorcrafs, and he abiliy o use M&S in cerificaion programs. AC 0-46 idenifies hree main groups o which he use of M&S may be applicable: applicans including he sea as par of he aircraf ype design, sea manufacurers, and sea insallers. Two possible uses for M&S in cerificaion programs include: o esablish he criical sea insallaion or configuraion which will be esed, poenially reducing he number of required cerificaion ess; or o demonsrae compliance of changes made o an exising cerified sea sysem. Compuer M&S is no limied o jus he cerificaion of he sea. The uses of M&S in aircraf sea design and evaluaion are numerous and begin in he early phases of any developmen program. Compuer aided engineering ools are common and lead o he direc developmen of M&S of prooype designs. The use of M&S here allows radeoffs, evaluaion of injury risks, invesigaion of poenial failure areas, and he selecion of successful design parameers. The use of M&S in hese early phases will help develop he es plans o successfully validae a sysem. In addiion o AC 0-46, SAE has recenly approved ARP 5765, Analyical Mehods for Aircraf Sea Design and Evaluaion (SAE 0). ARP 5765 was creaed o define specificaions for a virual Anhropormophic Tes Device (v-atd) suiable for aviaion impac es simulaions, o provide procedures for validaion of boh lumped mass and deailed finie elemen sea models, and o convey he curren bes pracices o assis engineering analyss in developing efficien, accurae models. Par of he purpose of his ARP was o provide addiional deail ha was no conained in he AC 0-46 and provide deails on pracices o develop a model ha can be used in he cerificaion process. The firs version of ARP 5765 includes bes pracices for boh esing and modeling and provides a deailed descripion of he requiremens for a v-atd o suppor dynamic sea simulaions. Any occupan model, including a v- ATD, mus undergo a proper validaion process ha is applicable o is inended use wih any limiaions adequaely described (Pelleiere and Knox, 0, Pelleiere and Moorcrof 0a). While analysis may be possibly applied in cerificaion projecs, hisorically, cerificaion has been accomplished hrough physical esing. One of he issues ha mus be addressed is he fac ha cerificaion esing is deerminisic. The resuls of a paricular es eiher pass or fail all of he requiremens levied upon i. If a es is considered a failure, hen hose resuls would no be forwarded o he regulaing auhoriy as par of he cerificaion package. This selecion of es daa limis cerificaion es daa o only being usable as a poin validaion since i only represens a single possibiliy of wha migh happen o a seaing sysem and only under accepable condiions. Anoher issue is ha he indusry considers every es a cerificaion es. This consideraion eliminaes he impeus o conduc repeaed esing o develop confidence inervals in he resuling daa. This is beer explained by he fac ha once a sea passes a paricular es configuraion, here is lile incenive o repea his es. If a es is repeaed and i subsequenly resuls in a failure, hen he sysem is no cerified. Simply pu, once a sea passes he poin es condiion, from an indusry sandpoin, i is bes o le i be and move on o he nex configuraion. Anoher issue wih repeaed esing is he added coss associaed wih conducing addiional ess. Wihou clear benefis esablished and consisency across he indusry, any paricular manufacurer would view his as an increased burden upon heir paricular company. An addiional cos burden ha hey already have conended wih sems from he coss ha migh be associaed wih addiional insrumenaion requiremens ha oherwise may be necessary for conducing esing o suppor model validaion. A cerificaion es will be conduced wih only he insrumenaion necessary o gaher he daa o make a pass or fail deerminaion in accordance wih he regulaions. Model validaion, however, may levy addiional daa requiremens ha will place addiional esing coss ono he indusry.. Cerificaion by Analysis Before cerificaion by analysis (CBA) can be successfully applied in meeing he safey requiremens for approval of new producs, here are several issues ha mus be addressed. These include he definiion of requiremens, predicion of failures, quanificaion of he uncerainy presen, and he applicaion of one-sided pass/fail crieria (Pelleiere and Moorcrof 0b, Pelleiere and Moorcrof 03). I is he saemen in 5.56(b) ha allows he use of dynamic analysis in he cerificaion of seaing sysems. This paragraph also denoes he raceabiliy o dynamic esing leading o he need o validae any model ha is proposed. In order o validae he model, specific merics are needed o
assess he qualiy of he model and is applicabiliy for fuure use. In developing SAE ARP 5765 a process was developed o compare boh magniude error and shape error. This descripion of his process addresses boh he quanificaion of error and he applicaion of he pass-fail crieria as described in previous work... Error merics The magniude error can be expressed as an absolue error which is given by Error, (Eq. ) or a relaive error: Peak Tes Peak Sim Peak Tes Peak Sim Error *00%,(Eq. ) Peak Tes where he ypes of daa can be force, momen, acceleraion, or velociy. The shape error can be evaluaed using he Sprague and Geers comprehensive error (Sprague and Geers 004, Moorcrof 007). Given wo ime hisories of equal lengh, measured m() and compued c(), he following ime inegrals are defined: I ( ) m ( ) d (Eq. 3) mm I ( ) c ( ) d (Eq. 4) cc I ( ) m( ) c( ) d (Eq. 5) mc The magniude error, biased owards he es, is hen defined as: M I / I (Eq. 6) cc mm The phase error is defined as: P cos ( I / I I ) (Eq. 7) mc mm cc The comprehensive error is defined as: C M P (Eq. 8) Due o he relaive simpliciy of he error meric, i can be implemened ino a spreadshee program wih lile loss of accuracy. The inegrals can be approximaed by summaions using he rapezoidal mehod... Example scenarios These merics allow he comparison of es and simulaion daa biased oward he es. However, in order for hem o be used in he validaion process, i mus be deermined wha levels of errors are accepable for he various daa channels. To achieve his, four differen scenarios for a forward facing dummy were developed and esed.. -poin bel wihou a oe sop. Inpu acceleraion is 6 G, 3.4 m/s (44 f/s) defined in Par 5.56 for he horizonal es condiion.. 60 degree pich es wih a -poin bel. Inpu acceleraion is 9 G, 9.45 m/s (3 f/s) defined in Par 3.56 for he combined horizonal-verical es condiion. 3. 3-poin bel. Adjus shoulder bel o produce 3.75 mm (.5 in) of iniial slack. Inpu acceleraion pulse is G,.8 m/s (4 f/s) defined in Par 3.56 for he horizonal es condiion. The geomery of he 3-Poin resrain sysem shall be such ha he shoulder bel o lap bel aachmen poin is 0.6 mm (4 in) o he righ of he ATD cenerline. 4. 4-poin bel. Adjus shoulder bel o produce 3.75 mm (.5 in) of iniial slack. Inpu acceleraion Pulse is G,.8 m/s (4 f/s) defined in Par 3.56 for he horizonal es condiion. All four scenarios were developed o be represenaive of ypical aircraf configuraions ha would be esed for various seaing sysems. For each scenario here are a number of parameers ha should be measured during he ess. I should be noed ha hese parameers may differ slighly from hose ha would be colleced during ypical cerificaion ess. Scenario is a ypical configuraion for ranspor caegory aircraf where he resrain sysem is of a -poin ype. Here, he key is he amoun of dummy forward flexion as i loads ino he resrain sysem. For breviy, only informaion peraining o scenario wih he FAA Hybrid III will be discussed, bu he mehodology was expanded o all four scenarios and included he 3
Sea Pan Load (lb) Hybrid II ATD as well. This mehodology can be generally applied o any siuaion where seing he accepance crieria for he required meric is necessary. The firs sep was o conduc repeaed esing of a leas hree ess. The sea was mocked up as a rigid sea in a ypical aircraf configuraion. A rigid sea was used wih he goal o evaluae he he variaions of ATD response. Afer he esing was complee and he daa gahered, he wo error merics for magniude and shape were calculaed amongs he esing pairs (Figure ). By doing his, he esing variabiliy and he measuremen uncerainy involved in he physical sysem are demonsraed. The reason for gahering hese daa is ha any simulaion canno be expeced o perform any beer han a paricular es. For insance, if he es o es variabiliy is 0%, hen i would be unreasonable o expec simulaion variabiliy o be wihin 0%. Figure All FAA Hybrid III responses o scenario.3. Resuling errors While here are many channels of daa colleced in a es, no all of hem are imporan when validaing a model. Some of hem (e.g., Upper Neck Fy and he Sea Back Fz) have boh large errors and large es variaions. Anoher consideraion is he imporance of a variable o he configuraion. In a forward facing es, he lumbar load is no an imporan parameer aslike i is for a es wih a verical componen. As such, i could be ignored in Scenario even hough i has low errors and low variabiliy. The imporan poin is o choose hose variables ha are perinen o he specific scenario and have conrollable variaions. For scenario he chosen parameers are hose relaed o he forward flail of he ATD, he loading of he sea pan (Figure 3) and resrain sysem and he acceleraion of he sled (Table ). I should be poined ou ha he ess were conduced o exercise conrol over he response and o minimize any esing variabiliy. From his lis he average of he variabiliy of he error merics observed in he es daa can be calculaed (Figures 4 and 5). 000 0-000 -000-3000 -4000-5000 -6000 0 0. 0. 0.3 0734-6 0734-7 0734-8 Figure 3 Example response daa Time (msec) Table Lis of seleced variables o measure in a Scenario es Measuremen Variable Sled Ax Righ, Lef Lap bel load Sea Pan, Fx, Fz,My Head CGx, CGz posiion Hpoin x, z posiion Knee x, z, posiion Ankle x, z posiion Head Angle Pelvis Angle 4
Figure 4 FAA Hybrid III load responses o scenario Figure 5 FAA Hybrid III posiion responses o scenario These anicipaed average and variaion errors were hen used o se he olerance levels a which he simulaion mus agree wih he es daa (Table ). Table Maximum allowable errors Measuremen Variable Magniude Error Shape Error Righ Lap bel 0% 5% Load Lef Lap bel load 0% 5% Sea Pan Fx N/A 0% Sea Pan Fz 5% 0% Sea Pan My 0% 0% Head CGx.7 mm (0.5 in) 0% Head CGz N/A 0% Hpoin x posiion.7 mm (0.5 in) 0% Hpoin z posiion.7 mm (0.5 in) 0% Knee x posiion.7 mm (0.5 in) 0% Knee z posiion N/A 0% Ankle x posiion N/A 5% Ankle z posiion N/A 0% Head Angle N/A 0% Pelvis Angle 7 o 0% A couple of specifics ha are imporan in he lis of seleced errors. Firs, i is apparen ha no every variable is seleced a a 0% level. For insance, he magniude error for he Sea Pan Fz is se o a 5% level. This is a direc resul of he esing showing approximaely 0% error in repeaed esing. Second, no every variable ha can possibly be measured is seleced for he validaion se. For insance, sea pan Fy is also measured, bu in a forward facing es, i is of lile consequence. Las, no each variable is seleced for boh magniude and shape error calculaions. The ankles had very low magniude errors and low shape errors. Ensuring ha he overall shape of he response decides wha are he imporan parameers o invesigae. 3. Discussion Specific error merics and allowable values for a v- ATD using in simulaions of sea sysems have been developed. These merics are scenario - specific. If a differen simulaion scenario is o be used, a process mus be conduced o deermine he proper se of measuremen variables o invesigae and wha hose acceped error values should be. 5
Because of his, i is he process ha is imporan and no he final values. I was demonsraed ha jus using an error limi of 0% is no always he mos appropriae. There may be siuaions where he acceped error could be lower or higher. One cavea here, he accepance values for he simulaions were no se a jus he minimum error values in he es daa. Raher hey were adjused upwards slighly hrough several differen opions. These included seing he error minimum 0% o remain in accordance wih he exising AC 0-46, rounding up o avoid cases of uneven values, and incorporaing an addiional facor afer verifying wih curren simulaion capabiliies. Using hese merics o assess he validiy of a simulaion resuls in four disinc oucomes. These include: () he simulaion meeing he acceped errors for boh magniude and shape; () he simulaion failing o mee he acceped errors for boh magniude and shape; (3 and 4) he simulaion only meeing he acceped errors for eiher he magniude or shape. The firs wo possibiliies are sraighforward in ha would direcly resul in a pass or fail. However, only meeing one condiion for he las wo possibiliies may or may no be considered a pass or fail. In he example provided here, he final conclusion was ha no every parameer had o mee a condiion for which passing boh was required. I may be necessary o revisi he accepable values if i is never possible o mee he required condiions. This could be a resul of no enough es daa iniially available o develop he acceped errors. For insance, in he presened example, he error values were generaed from only hree ess, no enough o develop saisical confidence for which hese values were hen adjused upwards. I should be poined ou ha meeing he validaion crieria presened here are no he only crieria for validaing a v-atd for use in aircraf seaing. There are oher checks ha mus be made as well, including assessing he mass and geomeric properies, evaluaing he sub-componens, and evaluaing he shape of he pelvis. As presened in SAE ARP 5765, a v-atd should demonsrae compliance wih all four scenarios, however if a v- ATD is only o be used in specific siuaions, hen i is possible o only validae i o he scenario of ineres and have a v-atd ha is only condiionally complian. FAA AC 0-46 was developed as a response o he public Law HR 000 under secion 757, Sreamlining Sea and Resrain Sysem Cerificaion Process and Dynamic Tesing Requiremens (Pub. L. No 06-000, 000). The AC was par of he plan ha was promoing alernaive mehods of compliance, which in his case include using M&S in he cerificaion process. Since FAA AC 0-46 was iniially drafed, he sae of he ar in M&S has drasically changed. As such i is currenly undergoing a revision process o describe how a model should be validaed if i is o be used as par of CBA. One shorcoming in he curren guidance is ha i describes a limi of 0% accepance. This blanke 0% sems from he fac here is lile repeaed esing on real sea sysems. Once a sea is esed as a pass, here is lile moivaion o rees i again. I will ake he indusry o conduc hese addiional ess o documen wha he acual esing variabiliy would be in order o jusify any changes o hese values. 4. Conclusion Cerificaion by analysis coninues o be pursued by he aircraf sea indusry. A componen in his endeavor is he validaion of he required occupan model or v-atd. A mehodology has been adoped ha uses es daa o assess he variabiliy and combines he variabiliy wih Sprague and Geers error merics o deermine he validaion of a model. While he specific example of a v-atd uilized in aircraf seaing simulaions was discussed, he mehodology would be generally applicable o oher cases of assessing he validiy of digial human models. Disclaimer The findings and conclusions in his paper are he opinions of he auhors and should no be consrued o represen any agency deerminaion or policy. References Code of Federal Regulaions, Tile 4 Par 5 (4 CFR Par 5) Airworhiness Sandards: Transpor Caegory Aircraf. FAA AC 0-46, 003. Mehodology for Dynamic Sea Cerificaion by Analysis for use in Pars 3, 5, 7, 9 Airplanes and Roorcrafs. Moorcrof D, 007. "Selecion of Validaion Merics for Aviaion Sea Models" The Fifh Triennial Inernaional Fire & Cabin Safey Research Conference, Oc, 007. Pelleiere J and Knox T, 0. Occupan Model Validaion, in Advances in Applied Digial Human Modeling, ed. Duffy VG, CRC Press. Pelleiere J and Moorcrof D, 0a. Occupan Calibraion and Validaion Mehods, nd Inernaional Conference on Applied Digial Human Modeling Inernaional conference, July, 0. 6
Pelleiere J and Moorcrof D, 0b. Implemenaion of Verificaion and Validaion in Cerificaion by Analysis from a Regulaory Perspecive, 50 h Annual SAFE Symposium, Oc 0. Pelleiere J and Moorcrof D, 03. Aircraf Sea Cerificaion by Analysis from a Regulaory Perspecive, American Helicoper Sociey 69h Annual Forum, Phoenix AZ, -3 May 03. SAE ARP 5765, 0. Analyical Mehods for Aircraf Sea Design and Evaluaion, SAE Inernaional, Warrendale, PA. Sprague MA and Geers TL, 004. A Specral- Elemen Mehod for Modeling Caviaion in Transien Fluid-Srucure Ineracion, Inernaional Journal for Numerical Mehods in Engineering, 60 (5), 467-499, 004. Wendell H. Ford Aviaion Invesmen and Reform Ac for he s Cenury, Pub. L. No. 06-000, 000. 7