30 April 2 May 2018 ICC Sydney Unlocking the Future through Systems Engineering. sete2018.com.au. Captain Ross Grafton

Transcription

1 30 April 2 May 2018 ICC Sydney Unlocking the Future through Systems Engineering sete2018.com.au Captain Ross Grafton

2 Lessons Learned in Applying Efficient Test Design and Analysis Techniques to Aircraft Survivability Assessments Joint Electronic Warfare Operational Support Unit Captain Ross Grafton Flight Lieutenant Gregory Brown Ms Zivana Novakovic University of New South Wales Canberra Dr Keith Joiner

3 Lessons Learned in Applying Efficient Test Design and Analysis Techniques to Aircraft Survivability Assessments Design of Experiments (DOE) Making aircraft countermeasures test and evaluation more efficient

4 Scope Aircraft survivability assessment Lessons learned through application of DOE 1. Efficient test design 2. Selection of most useful input factors 3. Evaluation of binary outcomes 4. Treatment of invalid test points 5. Consideration of uncertain responses Application to ongoing work

5 Survivability chain Stay away Aircraft Suppress threats Avoid detection Electronic Warfare Self-Protection -threat warning -countermeasures Prevent engagement Avoid hit Tolerate damage Threat system

6 Survivability assessment Why? Protect people and assets Assess risk Inform decision makers Who? + = Joint Electronic Warfare Operational Support Unit DOE How? Simulated threat systems Limited flight trials Modelling and simulation environment DOE

7 Legacy vs DOE approach Aspect (L=5) (0, 45, 90, 135, 180 ) Constants (C) Aircraft type Countermeasures Test envelope DOE Input (X) Airspeed (L=3) (Hover, Med, High) Range (L=3) (Near, Mid, Far) Altitude (L=2) (Low, High) Countermeasures effectiveness Noise (N) Simulated threat system Decoy (Y/N) Output (Y) (Binary) Aircrew, Cloud, Sun, Temperature, Airborne particles, Wind, Seeker tolerances, Countermeasure strength variation, Countermeasure duration variation Legacy Constraints Altitude Range Airspeed Trial logistics Entrenched attitudes Statistical rigour User expectations Limited DOE experience

8 Test planning Choice of software tools Five level input: rdexpert(lite) TM Binary output: DOE Pro XL TM? Combine with legacy test points Risk reduction measure Improved DOE-only orthogonality Reduced reps per run Indicative test matrix Source Test No. A : Aspect B : Airspeed C : Range D : Altitude DOE Hover Near Low Med Mid High High Far Low Hover Near High 5 90 Hover Mid Low 6 90 Med Far High 7 45 Hover Far High 8 45 Med Near Low 9 45 High Mid High 10 0 High Near High High Far Low High Near Low 13 0 Hover Mid Low 14 0 Med Far Low Med Near High Med Mid High Legacy Hover Mid Low (must do) High Mid Low Hover Mid Low High Mid Low 5 90 Hover Mid Low High Mid Low Hover Mid Low High Mid Low 13 0 Hover Mid Low 24 0 High Mid Low

9 Lesson 1 Efficient test design Application of DOE allowed: Expanded test envelope. Comparable overall trial effort. Increased confidence levels.

10 Altitude Selection of test factors Determining test envelopes Legacy methodology Factor impacts not quantified Subject matter expert opinion DOE sometimes reinforced opinions, sometimes debunked myths Multi-collinearity Scenario dependent Think Atmospheric effects example Range Slant range significance examples (range and altitude interaction) Air density effects Screen Think again Slant range sufficient Other factors at work

11 Lesson 2 Selection of most useful input factors Application of DOE resulted in: Identification of redundant input factors. Reconsideration of legacy test methods. Smarter selection of future test points/envelopes (see lesson 3).

12 Exceeds 0 to 1 Binary outcome systems (Using multiple response linear regression) (Using multiple response binary logistic regression) Low R 2 Very weak model Saturated with 1s (few 0s) Form of: y = b o + b 1 x 0.2 < R 2 U < 0.4 Moderate strength Most outcomes decoy/no decoy - poor linear regression results. DOE PRO XL TM marginal means can still show general trends. Binary logistic regression using Quantum XL TM. Form of: p = e (b o+b 1 x)

13 Test cricket analogy Similar edge characteristics to aircraft survivability scenarios Unclassified data Complex 12 factor quantitative and categorical inputs 16 x 2-way interactions assessed 6 x 3-way interactions assessed Win = 1, Loss/draw = 0 Factor Name Const A Toss (Loss/Win) B Home Ground (No/Yes) C Century (0,1,2,3) D Double (0,1) E Bat longer than 1 day (No, Yes) F Declare (No, Yes) G Score over 400 (No, Yes) H Runouts (0,1,2) I Caught Behind (0,1,2,3,4,5,6,7) J Stumpings (0,1,2) K 5 Wicket (0,1) L Bowlers used (3,4,5,6,7,8)

14 Linear vs logistic regression Regression type Linear Binary logistic Significant factors Red: p<0.05 Blue: 0.05 p<0.1 Validation on 2017 Ashes test series 2-level Linear Regression Logistic Regression Model strength, significance, goodness of fit Test 1st 2nd 3rd 4th Venue Brisbane Adelaide Perth Melbourne Who Bats First England Australia England Australia 1st Innings Result 10/302 8/442(Dec) 10/403 10/327 Actual Outcome England lose Aust win Aust win Draw Prediction Interpretation Eng lose Aust win Uncertain Aust win Validation Probability Lose 73.4% Eng lose 0.8% Aust lose 79.9% Eng lose 25.4% Aust lose Probability Win 26.4% Eng win 99.2% Aust win 20.1% Eng win 74.6% Aust win Validation B Home Ground J Stumpings K 5 wicket L Bowlers used JK Stumpings * 5 wicket JL Stumpings * Bowlers used R 2 = (Weak) F, F p-value, lack of fit OK Software applied DOE PRO XL TM Quantum XL TM A Toss BJ Home ground * Stumpings E Bat longer than 1 day CK Century * 5 wicket AC Toss * Century KL 5 Wicket * Bowlers used AE Toss * Bat longer than 1 day BD Home ground * Double BG Home ground * Score over 400 R 2 U = (Moderate) G, G p-value, goodness of fit OK

15 Lesson 3 Evaluation of binary outcomes Application of DOE identified: Logistic regression is essential for countermeasure win/lose scenarios. Test envelope must adequately cover both outcomes.

16 Invalid test points Logistic regression more tolerant of missing data. Invalid reps due to equipment/coordination failure redo. Invalid runs may be operationally exploited. Avoid hit Assumes launch has occurred. Most countermeasure assessments. Better yet Prevent engagement Prevent lock and/or launch. Flight profile, signature, jamming? Unknown aircraft / simulated threat vulnerability? DOE Legacy Two output response: Test success (1=Lock, 0=No lock) Outcome (1=Decoy, 0=No decoy)

17 Lesson 4 Treatment of invalid test points Application of DOE facilitated: Contingency to redo some reps. Identification of unexpected simulated threat vulnerabilities. An innovative approach to using trial no test data.

18 Edge cases Aircraft survivability example Test cricket example Validation on 2017 Ashes test series 3-level linear Regression [0,1,2] Multinominal regression R 2 U = 0.34 Uncertain response Decoy Uncertain behaviour No decoy Use multi-level (ordinal) logistic regression Cricket example Noise/test perturbations? Partial disruption? Decoy then reacquire? Small sample size, plus only one draw in 2017! Test 1st 2nd 3rd 4th Venue Brisbane Adelaide Perth Melbourne Who Bats First England Australia England Australia 1st Innings Result 10/302 8/442(Dec) 10/403 10/327 Actual Outcome England lose Aust win Aust win Draw Prediction Interpretation Eng lose or draw Aust win Draw Aust win or draw Validation Probability Losing 90.2% Eng lose 7.7% Aust lose 3.9% Eng lose 2.5% Aust lose Probability Draw 0% draw 0% draw 30.6% draw 14% draw Probability Win 9.8% Eng win 92.3% Aust win 65% Eng win 83.4% Aust win Validation

19 Lesson 5 Consideration of uncertain responses Application of DOE requires: Use of multi-level (ordinal) logistic regression. A reasonable proportion of uncertain outcomes.

20 Summary of lessons learned DOE-aided survivability assessments 1. Efficient test design 2. Selection of most useful input factors 3. Evaluation of binary outcomes 4. Treatment of invalid test points 5. Consideration of uncertain responses

21 Application to ongoing work Year and beyond Phase Test case Confidence building Rollout Full implementation Application 1 trial 4 trials (mixed legacy & DOE in critical cases) 2 historical analyses Regression type Linear Linear Binary logistic Training Advanced T&E techniques, UNSW Canberra Advanced T&E techniques, UNSW Canberra Ad-hoc binary logistic regression All trials must consider DOE In-depth historical analyses Countermeasure optimisation Linear Binary logistic Ordinal logistic Advanced T&E techniques, UNSW Canberra Design for Six Sigma, Air Academy Associates

22 Joint Electronic Warfare Operational Support Unit Captain Ross Grafton Flight Lieutenant Gregory Brown Ms Zivana Novakovic University of New South Wales Canberra Dr Keith Joiner Aircraft imagery from