FIRST Game Analysis
Agenda 2 sessions 13:30-15:00 Game Analysis 15:00-15:30 Ask A Head Referee
Who are we? Anne Shade 22nd year in FIRST Former FIRST Chesapeake Regional Director Former 2377 General Manager and Coach Former Mechanical and Systems Engineer, Leidos Steve Shade 21nst year in FIRST FIRST Chesapeake Head Referee Former 2377 Lead Engineer and Coach Senior Engineer Modeling and Simulation, JHU Applied Physics Laboratory
Background Knowledge Basic Understanding of FRC Games Basic Understanding of Statistics
Common Themes Write it down Post it up Questions What do you want to accomplish? Who is responsible? How are you going to accomplish it? Repeat
Making Decisions Every team should have a method to make decisions Popular vote is not always practical for robot building purposes, often misses the quiet contributors We like Decision Matrices for comparing ideas Voting can be used to break ties
Session Overview Assumptions Establish team goals Create event goals Form match goals Data Review Review statistical models Examine 2018, 2017, and 2016 data Compare results to models Game Analysis Understand how to play the game Determine how many points is needed to win a match Construct a game strategy to meet the point goal Implementation Select a strategy Evaluate options Strategic design Competition execution
Assumptions
Assumptions Your team wants to advance to the District Championship and World Championship Be a contributing alliance member Consistent and improving on-field performance A great robot can win matches, but two good robots with great strategy win more frequently
Big Picture Process Robot Requirements On Field Performance Robot Design Robot Construction Coaching Game & Team Strategy Scouting Robot Testing Prioritization Decision Making
District Ranking District Championship qualification is determined by points your team earns at first two Chesapeake district events plus team age correction FIRST Championship qualification includes points earned at Chesapeake District Championship District championship weighted 3x district events Qualifying Points Alliance Selection Order Playoff Performance Awards Event Ranking Points Chesapeake District Event 1 Chesapeake District Event 2 Chesapeake District Champ. Team Age Correction (1st & 2nd year teams) Total District Ranking Points
District Ranking Qualifying Points Alliance Selection Order Playoff Performance Awards Event Ranking Points Chesapeake District Event 1 Chesapeake District Event 2 Chesapeake District Champ. Team Age Correction (1st & 2nd year teams) Total District Ranking Points Must score points to maximize these categories!
Statistics Review and Historical Data
80/20 Rule (Pareto Principle) For many events, roughly 80% of the effects come from 20% of the causes In other words, if we can find the meaningful 20% of the game and make it great, we can contribute to achieve 80% of the points and win about 80% of the time Pareto works best with power distribution Scores in 2011 & 2012 followed power distribution Scores have become more normally distributed over time, but principles still apply
The Normal Distribution Distribution defined by mean (µ) and standard deviation (σ) calculated from raw data Probability of a win given a score can be found by calculating the area under the pdf. P(win score) = blue area Raw data can be modeled to reflect probability of score occuring. Σ P(all scores) = 1 Probability Density Function (pdf) The Cumulative Distribution Function (cdf) can be generated for by calculating the area under the pdf at each score. Looking for the score where the cdf = 0.8 (score = 0.85)
What are typical FRC scores? Depends on the game Depends on the event Our analysis ignores the effects of Foul points Fouls are included in recorded scores Fouls not included in Kickoff analysis Known error source Scores generally increase each week, Big steps at DCMP and CMP FIRST Chesapeake is not FIRST in Michigan, Indiana or Ontario, but districts are making a huge difference
Ranking Strategy Goal to win about 80% of your matches 12 matches x 0.8 = 9 to 10 wins per event Districts - Top 6 (estimated ranking) DCMP - Top 12 (estimated ranking) CMP - Top 24 (estimated ranking) Ignores bonus Ranking Points (e.g. 3 robot climb 2018, 4 Rotor Matches 2017)
2018 Game Overview
2018 Score Data
2018 Score Model - Normal Distribution
2018 Data Summary 2018 Probability of Win - CHS+CMP P(win score > 366 [kickoff guess]) = 0.71166 P(win score > 395 [80% win score]) = 0.79766 P(win score > 420 [match breakpoint]) = 0.85778 P(win score > 500 [max score]) = 0.96632 Kickoff Analysis 420 points breakpoint 366 points to win 80% matches 2018 Probability of Win - All FRC P(win score > 366 [kickoff guess]) = 0.76595 P(win score > 378 [80% win score]) = 0.79832 P(win score > 420 [match breakpoint]) = 0.88891 P(win score > 500 [max score]) = 0.97467 Fouls significantly inflate scores! 420 points -> 89% matches won 366 points -> 77% matches won Team Examples GDC District - 1731 (4) CHS DCMP - 5546 (5), 422 (11), 1599 (12) CMP ARC - 6328 (6), 2016 (32)
2017 Game Overview
2017 Score Data
2017 Score Model - Normal Distribution
2017 Data Summary 2017 Probability of Win - CHS+CMP P(win score > 265 [first kickoff guess]) = 0.49034 P(win score > 315 [week 1 update]) = 0.73283 P(win score > 333 [some fuel added]) = 0.8034 P(win score > 355 [4 rotor match]) = 0.87242 Kickoff Analysis 265 points initial guess 315 points with easier climb 2017 Probability of Win - All FRC P(win score > 265 [first kickoff guess]) = 0.71571 P(win score > 315 [week 1 update]) = 0.88876 P(win score > 333 [some fuel added]) = 0.92701 P(win score > 355 [4 rotor match]) = 0.95906 80% win point included just a little fuel. 315 points -> 89% matches won All FRC 315 points -> 73% matches won CHS+CMP Team Examples NMD District - 3941 (1) 240 max avg points CHS DCMP - 1731 (8) 311 max avg points CMP ARC - 2611 (5), 217 (6), 2590 (12)
2016 Game Overview
2016 Score Data
2016 Score Model - Normal Distribution
2016 Data Summary 2016 Probability of Win - CHS+CMP P(win score > 118 [80% CHS win rate]) = 0.79867 P(win score > 124 [kickoff guess]) = 0.84744 P(win score > 165 [typical max score]) = 0.98969 2016 Probability of Win - All FRC P(win score > 118 [80% CHS win rate]) = 0.92318 P(win score > 124 [kickoff guess]) = 0.94999 P(win score > 165 [typical max score]) = 0.99914 Kickoff Analysis 124 points 124 points -> 95% matches won All FRC 124 points -> 84% matches won CHS+CMP This is within 5% of the actual point for 80% win rate Team Examples CMD District - 836 (1) (only averaged 113) CHS DCMP - 422 (6) CMP GAL - 5006(18), 2468 (19)
P(Win??) > 0.8 How many points does my Alliance have to score to make this equation true? How do you figure this out at Kickoff?
Game Analysis
Overall Process Evaluate the Game Estimate Match Scores Develop Strategies Arena Scoring Opportunities Fouls Ranking Process Alliance Team Qualifying Playoffs Brainstorm methods to achieve the target match score WHAT will our robot do? Design Robot Develop Robot Requirements Select a Strategy Goal - Achieve Strategy Robot Requirements act as design guide Specify HOW robot will achieve strategy Demands Wishes Decision Matrix to select a strategy
Analyzing the Game READ THE RULES!!!!! Goal: Understand the game Examine every possible way to score points, no matter how obscure How much is each object/robot/action worth? Are there bonus points? Is there a maximum match score? Limited number of objects or do objects cycle? Examine every possible way to prevent opponents from scoring Stealing balls (2000), Capping robots (2004) Understand the ranking system Win-loss-tie, loser s score, own score plus double the loser s score, Coop points, bonus ranking points What are the key Fouls?
The Big Questions How many points does an alliance need to score to win a match? How many points does our team need to score to help our alliance win a match?
Strategy Development Examples 2011 (55-60 pts/match/team goal) 1 Ubertube top row and 2 top row Logos + 4th place minibot (58 points) 1 Ubertube top row and 1 top row Logo + 1st or 2nd place minibot (60 points) 1 Ubertube middle row and 2 middle row Logos + 1st place minibot (58 points) 2013 (~72 pts/match/team goal) 3 High Goal Auto Disc + 4 Pyramid Discs + 30 point Climb (68 points) 5 High Goal Auto Disc + 3 cycles @ 4 High Goal Discs + 10 point Climb (76 points) 3 High Goal Auto Disc + 20 High Goal Discs + 10 point Climb (88 points)
Strategy Development Examples 2014 (~165 pts/match goal) 15 Auto Mobility + 30 Auto Ball + 4 Cycles @ 20 Ball + 10 Truss (165 points) 15 Auto Mobility + 30 Auto Ball + 6 Cycles @ 10 Ball + 10 Truss (165 points) 15 Auto Mobility + 30 Auto Ball + 3 Cycles @ 40 Ball (165 points) 15 Auto Mobility + 0 Auto Ball + 5 Cycles @ 31 Ball (170 points) 2016 (~125 pts/match goal + 1 or 2 RP) 4 Auto Reach + 1 Auto Cross @ 10+ 10 Auto Boulder + 35 Cross + 5 Low Goals @ 2 + 4 High Goals @ 5 + 5 Challenge + 15 Scale (129 points) 1 Auto Reach + 2 Auto Cross @ 10 + 5 Auto Boulder + 8 Cross @ 5 + 4 Low Goals + 3 High Goals + 15 Challenge (120 points) 3 Auto Cross @ 10 + 5 Auto Boulder + 5 Cross @ 5 + 5 Low Goals + 3 High Goals + 15 Challenge (125 points) 3 Auto Cross @ 10 + 5 Auto Boulder + 7 Cross @ 5 + 9 Low Goals @ 2 + 10 Challenge + 15 Scale (113 points)
Strategy Development Examples 2017 (~150-200 pts/match goal) 5 Auto Mobility + 60 Auto Gear + 80 Rotors (5 Gear Cycles) + 50 Climb (195 points) 5 Auto Mobility + 0 Auto Gear + 120 Rotors (6 Gear Cycles) + 50 Climb (175 points) 5 Auto Mobility + 60 Auto Gear + 40 Rotors (4 Gear Cycles) + 50 Climb (155 points) 5 Auto Mobility + 60 Auto Gear + 0 Rotors (0 Gear Cycles) + 50 Climb + 40 kpa Fuel (155 points)
Strategy Development Examples (2018) Theoretical Alliance Max Actions Occurred Auto Teleo p Time Held 3 Red Alliance Total Actions Occurred 15 3 Time Held Blue Alliance Total Actions Occurred 15 3 Time Held Total Run 5 per robot Scale 2 per own 2 per sec 1 14 30 1 5 12 1 5 12 Switch 2 per own 2 per sec 1 14 30 1 13 28 1 13 28 Scale 1 per own 1 per sec 0 135 135 8 65 73 8 65 73 Switch 1 per own 1 per sec 0 135 135 2 135 137 2 135 137 Vault 5 per cube 9 45 9 45 9 Parking 5 per robot 0 0 Climbing 30 per robot 2 60 2 60 2 60 Levitate 30 per robot 1 30 1 30 1 30 Force 10 per use 0 0 1 10 1 10 Boost 10 per use 2 20 2 20 2 20 1-10 1-10 Dark Force -10 Total 500 0 420 15 45 0 420
Implementation
Next Steps Develop 3-6 Strategies Select 1 Strategy Define Robot Requirements
Game Strategy Thoughts Your team does not have to score all of the alliance points, just assist the alliance offense OFFENSIVE strategy should be priority Defensive maneuvers should not be primary strategy, but can be incorporated Best to drive and do one thing really well, than do many tasks averagely or worse Need to do the right thing well Example (2014) pickup and manipulate the ball well
If football is a game of inches robots is a game of seconds. Drive Have a purpose Practice makes perfect Acquire One touch and hold Don t let go until you want to Not enough resources, prototyping, time spent on this element 148 JVN Blog, 2363 Triple Helix Blog, 3847 Spectrum Blog Score Usually easier when acquire is robust (can t score without acquire) Too much time/resources spent here
Acquisition Accuracy Develop high accuracy, quick acting pickup mechanisms based on your abilities Scoring location is often secondary factor 2018 Switch/Vault Robot 2014 Low Goal Underused (31 points/cycle vs. 40 points/cycle) 2017 Gear/Hanging Robot
Cost-Benefit Analysis For each task you must compare the difficulty of accomplishment to the reward for doing so Balancing easier than scoring (2012) Small balls (2001) This is where the strategic value vs. coolness factor decision often pops up Vision tetra vs. hanging tetras (2005) Scoring Fuel vs. Gears (2017) The best tasks to perform are those which are relatively easy, yet provide big points Climbing the rope (2017) Remember denying your opponents 10 points is just as good as scoring 10 points (at least in terms of win/loss)
Prioritization Two separate lists Desired robot qualities Things like speed, power, agility, center of gravity Desired robot functionality The things you want your robot to be able to do Shoot balls, climb pyramid, traverse field Also consider necessary versus desired requirements Write these down and post so all can see At this point you can merge the two lists, and decide on a drive system and functionalities
Strategic Design Designing and building a cool robot is a lot of fun Designing and building a cool robot that does well in competition is even more fun Difficult to go through build without a goal Success in competition is clear choice It s not just about robots but there s a reason why the program is called FIRST Robotics COMPETITION Secondary objectives: aesthetics, design elegance, coolness, etc. Beware of the cool factor Can be fun, but sacrificing effectiveness hurts your partners
Simplicity & Golden Rules Golden Rule #1: Always build within your team s limits Evaluate your abilities and resources honestly and realistically Limits are defined by manpower, budget, experience Avoid building unnecessarily complex functions On the other hand, as you get more experienced start cautiously pushing a few boundaries Consider building limits/resources during the offseason instead of sacrificing performance Golden Rule #2: If a team has 30 units of robot and functions have maximum of 10 units capability, better to have 3 functions at 10/10 instead of 5 at 6/10 More important to do a few tasks well than a bunch of tasks mediocre
Once Built Re-evaluate strategy How close are you to your initial goals? Be realistic with what you can do Know the machine limits Watch how game is played at other events Game play evolves, very different each week Keep improving design Apply good ideas from what you watched Expand horizons Style of play is different in other parts of country
Competition Execution Scouting Goal is to collect useful data of actual performance to feed into match strategy Many resources available Effective Coaching Communication Negotiating and Consensus Building Leadership Visible team representative Teamwork with other Coaches during matches Alliance selection factor Robot Maintenance Checklists, Checklists, Checklists Preventive actions more effective than reactive fixes
Final Comments Read the rules!! Develop a clear, consistent strategy for how your robot will play the game Remember the Golden Rules The role of the coach cannot be understated Scouting is the easiest way to make your team more successful at competition Each FIRST match is like a high-speed game of chess: You need to have a well thought-out plan, but be prepared to counter your opponents moves Have fun!!