WISE STUDENT MONITORING, TUTORING AND ASSESSMENT SYSTEM T E C H N I C A L D E S C R I P T I O N

Transcription

1 T E C H N I C A L D E S C R I P T I O N WISE STUDENT MONITORING, TUTORING AND ASSESSMENT SYSTEM MPRI Simulatins Grup MPRI, a divisin f L3 Services Grup Inc West Califrnia Avenue Salt Lake City, Utah USA Phne Fax M-WISETECH

2 T E C H N I C A L D E S C R I P T I O N WISE Student Mnitring, Tutring & Assessment System Prepared by: MPRI Simulatins Grup MPRI, a divisin f L3 Services Grup Inc West Califrnia Avenue Salt Lake City, Utah USA Publicatin ID: 08-M-WISETECH Date: Nv 2008 NOTICE: This prpsal cntains data and infrmatin prprietary t MPRI that are maintained as trade secrets that are nt t be disclsed t third parties withut MPRI s apprval. This data and infrmatin shall nt be duplicated, used, r disclsed in whle r in part fr any purpse. The data and infrmatin subject t this restrictin are cntained in all pages and this Ntice shall be applicable t all such cntained data and infrmatin as if indicated n each page heref.

3 Table f Cntents 1 WISE Mnitring, Tutring and Assessment System 2 Intrductin Rule Generatin Prcess... 3 Prductin f rule definitins fr similar systems4 Rule definitins... 4 WISE Mnitring System... 5 Feedback messages... 5 Supplementary Material... 6 Operatr Prgress Mnitring... 7 WISE Training Curses... 8 Prvisin f Underpinning Knwledge... 9 Supprting WISE Rules... 9 Curse develpment facilities... 9 WISE Autmatic Cmpetence Assessment System Descriptin Assessing a Scre Setting up f the Assessment Mnitring f the assessment Benefits Use in Assessing Industry Standards WISE Dcumentatin M-WISETECH 1

4 WISE Mnitring, Tutring and Assessment System Intrductin The MPRI Liquid Carg Handling Simulatr and Engine Rm simulatrs can be prvided with student mnitring, feedback and assessment. This capability is prvided by the WISE sftware cmpnent. WISE is an intelligent mnitring and perfrmance assessment system, riginally develped specifically t allw the use f real time simulatin by individuals withut supervisin, it s functinality is been expanded bth t assist an instructr in the prcess f mnitring the activity f a large number f students undertaking a simulatin exercise and t prvide evaluatin f the perfrmance f the students engaged in the simulated activity. WISE is based upn the use f a specialized Expert System (a frm f artificial intelligence) t mnitr the activity f the student and the simulated mdel. The knwledge database used by the Expert System cmprises a large cllectin f rules entered by subject experts that are then cmbined tgether t define specific patterns f activity r results. The data gathered whilst the simulatin exercise is in prgress is then used t prvide the fllwing, depending upn the mde f peratin: Feedback directly t the student Highlighting f errrs r inapprpriate actins Advice regarding activities that have been r shuld be undertaken The highlighting f likely results f actins implemented when result falls utside f the expected nrm Access t supprt material t allw the student t identify the cause f errrs r reasn why feedback has been displayed, and then rectify the situatin accrdingly. Feedback t the instructr Full details f activity by individual students r grups Summary f key activity by individual students r grups Autmatic evaluatin f the perfrmance r cmpetence f the student r grup in undertaking the task(s) with the results prvided T the student T instructr The WISE Mnitring System has been especially develped fr the task and prvides the fllwing benefits when cmpared t traditinal mnitring systems based n the principals f Expert Systems namely: a) Allws direct input f rules by subject expert b) By remving the need fr an intermediary the knwledge database can be created much mre quickly and the number f errrs minimized. c) The number f rules required t be entered t prduce the same level f detail within the knwledge database are significantly reduced. 08-M-WISETECH 2

5 d) Rules defining the structure f the mdels being mnitred are pre cmpiled and hence d nt have t be included in the knwledge database. This leads t a cnsequent reductin in the inference requirements and hence a significant increase in the speed f evaluatin. A typical knwledge database fr a simulatin mdel may cmprise f rules input by the subject expert. This in turn will prduce a knwledge database fr prcessing cmprising f rules which in turn will mnitr variables n each prcess cycle. Fr a similar applicatin prduced using a traditinal Expert System the knwledge database wuld need t cmprise f 10,000 12,000 rules. Rule Generatin Prcess One f the key advantages prvided by WISE is the separatin f the rules frm the structure definitin f the bject being mnitred. Apart frm reducing the ttal number f rules that have t be included within the final knwledge database and hence cnsiderably reduces the required prcessing time, it als makes the prcess f entering the rules much simpler fr the subject expert, especially when the mnitring prcess is ging t be used n similar systems. Fr use n either the Lchs r Engine simulatrs the rules cmprise f the fllwing: Rules t cver the general peratinal principles and activities which will apply t mst situatins and peratins applicable t the mdel type being cnsidered Specific activities which will relate t the particular requirements f a specific vessel r engine Rules t track a students prgress when undertaking predefined exercises T prduce a knwledge database fr the peratin f a particular mdel the fllwing prcess is undertaken: General Rules (by expert)) Structure Definitin Structure SHIP - CARGO -- TANK --- 1P ---- VALVE SUCTION FLOPAR(?,4) SUCTION_FLOW VFLOW(4) STRIPPING FLOPAR(?,6) Specific rule - Kndatabase N. 1P carg tank suctin valve (FLOPAR(1,4) clsed N. 4S carg tank level decreasing AND N.4S suctin valve pen OR N. 4S lading valve pen 1) Using the input envirnment the subject expert will define the rules that he/she wishes t be tested. These rules are cnstructed at a general level, such as carg tank level filling. 08-M-WISETECH 3

6 The detail f the specific tank t be mnitred is nt included as part f the rule. As well as the rules t cver the general principles the subject expert may als add rules specific t a particular structure at this time. As the rules are cmmitted the general rule definitins are prduced. 2) Separately a structure tree is prduced defining the actual structure f the bject t be mnitred. The case shwn in the picture is fr a ships carg system but the definitin can be fr any structure such, as that f an engine system. This structure tree defines bth the individual items such as the number f tanks, valves, cntrls and cmpnents but als the assciated variable names used within the simulatin mdel that will be mnitred. 3) The general definitins are then cmbined with the specific structure tree using a Rule Generatr. The generatr takes in the general definitin and lks thrugh the structure and identifies all instances f the item defined in the general definitin. It then prduces a specific rule fr each instance lcated including the actual variable(s) that shuld be mnitred, resulting in rules specific fr the vessel t be mnitred. If there are general rules which d nt match the structure defined then these are autmatically ignred and a specific rule is nt prduced. Using the carg tank filling definitin and assuming the structure has the fllwing carg tanks defined:- N.1 Prt N.1 Stbd Slp The rule generatr will prduce the fllwing rules autmatically: N.1 Prt diesel il tank filling N.1 Stbd diesel il tank filling Slp tank filling Prductin f rule definitins fr similar systems This apprach allws a new knwledge database t be created fr a similar system withut the need t rewrite all the general rules. By changing the structure definitin t be suitable fr the new system (i.e. nly tw carg tanks instead f three) and by then rerunning the rule generatr a new set f rules will be prduced specific the new system. This prcess takes a matter f minutes. Rule definitins WISE allws definitins f the activities t be mnitred t be at the very simple level thrugh the determinatin f highly cmplex situatins. At the simplest level, a single rule may be used t check the value f a single parameter which may then be used t initiate a message such as: Vapriser utlet temp > 100 Hwever, a single rule can als be used t mnitr multiple parameters such as: Vapriser utlet temp > 100 And Vapriser utlet valve shut 08-M-WISETECH 4

7 Or Temp cntrl set t manual Fr the determinatin f mre cmplex situatins, rules can then be prduced which cmbine checks n ther rules and s n. This allws the subject expert r instructr t define very cmplex patterns cmbining checks f different aspects f the systems at the same time in the same way as a real instructr. If defined in an apprpriate way in additin t checking n values f parameters at a given pint in time it can be used t predict the cnsequence f an actin and advise the student r instructr accrdingly. Rule sets can be develped and extended t cncentrate n a particular activity in detail r track the prgress f an individual thrugh a particular exercise r scenari. Rule input facilities Facilities can be prvided if required that will allw instructrs t create their wn rules t assist them in mnitring specific exercises r student reactins. WISE Mnitring System When the WISE Mnitring system is started, the student is first required t lg n t the system and als select the apprpriate set f rules that will be used during the exercise. At the time f the student s initial lgn, a lg file fr the student is created. On the initial and every subsequent ccasin the student uses the system, the fllwing data will be recrded: Date and time f lgn Exercise rule set laded Every feedback message utput t the student during the exercise Date and time f lgff The mnitring system prcesses rules against the status f the mdel apprximately every secs. This perid has been determined as the best cmprmise between aviding prviding infrmatin cntinuusly leading t infrmatin verlad, and mnitring activity at sufficiently regular intervals t catch any errrs that may be made. Existing systems in use n the Liquid Carg Simulatr currently prcess in the regin f rules n each cycle depending upn the mdel being used. WISE perates in the backgrund whilst the simulatin is in prgress and will nly utput a feedback message if the need t d s has been identified. Feedback messages The rules within the system upn which it perates are primarily related t three areas: Training Safety Operatinal Efficiency All the rules have been develped with the help f subject experts and thse actively engaged in mnitring student perfrmance using the SAfeCarg and Engine Rm simulatrs within a classrm envirnment. 08-M-WISETECH 5

8 As indicated previusly the mnitring system regularly prcesses all the rules (currently in excess f 1800) against the status f the mdel, nrmally at time perids nt exceeding 20 secnds. Depending upn the status f the mdel at the time the system will utput apprpriate feedback messages t the peratr in the frm f highlighted messages in varius apprpriate categries including: 1) General Infrmatin 2) Warnings 3) Highlighting f Errrs The rules currently used within the system have been develped s that many simple and cmplex scenaris cvering all the systems assciated with carg peratins can be mnitred in sufficient detail t allw infrmatin t be prvided t an peratr s that they can identify the situatin being highlighted and take apprpriate actin withut utside interventin. Unlike many ther mnitring systems all the rules within the knwledge database are evaluated n each cycle, cnsequently whatever the student des is mnitred and des nt have t be pre defined by the instructr. The infrmatin that can be prvided in the messages is bviusly limited t indicating a prblem r the prvisin f basic advice. This alne may nt be sufficient t allw an peratr t cntinue with the exercise. Mre detailed infrmatin may be required. Supplementary Material T prvide the additinal infrmatin that may be required by the peratr, individual messages are linked directly int apprpriate sectins within an electrnic dcumentatin library. The peratr can access this infrmatin directly by clicking n the required message with the muse. If required, the student can freeze the simulatin whilst they read the material, and restart the simulatin nce they are satisfied with what they have t d t address the reasn fr the message. If the message is linked t a dcument, this will then be autmatically displayed in a brwser windw. The first dcument t be displayed prvides an explanatin as t why the message has been utput and a brief descriptin f the actins t be taken, if any. Links are then prvided t mre detailed infrmatin when it is available and apprpriate. The standard dcumentatin supplied prvides general guidance n peratinal prcedures, perating checklists and general assistance. Tgether, fr each message, they prvide sufficient infrmatin t allw the peratr t learn abut a particular aspect f the peratin and if necessary, vercme a prblem r crrect an errr, and allw them t cntinue with the exercise withut the need fr input frm an utside instructr r supervisr. 08-M-WISETECH 6

9 If required, the cntent and amunt f material can be expanded t include cmpany specific material such as: Operatinal guidelines r prcedures Cmpany checklists Specific equipment infrmatin Links can als be prvided int cmputer based training materials supplied by ther prviders. Operatr Prgress Mnitring The WISE mnitring system, as well as being used t prvide feedback t the peratr, can als be used t assist in the assessment f the perfrmance f the peratr. The fllwing facilities are prvided t prvide the required infrmatin: User Lg The name, date, time f cmmencing, and duratin f particular user lgn sessin are recrded t file. Message Lg All messages utput frm the system can be recrded t a file n disk fr each individual user. These may be examined later r utput t a printer t prvide a permanent perfrmance recrd. When cmbined with the exercise lg maintained by the SAfeCarg system it is pssible t clearly determine the activities f an peratr at a particular pint in time r thrughut an exercise. Rule Set Selectin WISE Carg is supplied with a standard set f rules that mnitr all the main peratins and many f the errrs that an peratr may make during such peratins in the areas f safety and peratinal efficiency. Further rule sets can easily be develped by MPRI r facilities can be prvided t allw the custmer t develp there wn rule sets. These rules may either 08-M-WISETECH 7

10 cncentrate n a particular activity in mre detail, r can be used t track the prgress f an peratr thrugh a particular exercise r scenari. The rule set t be used can be selected when the mnitring system is first activated. WISE Training Curses WISE was initially designed t allw students t undertake detailed training using the simulatrs withut the need fr an instructr r supervisr t be present. Whilst the WISE Expert System replaces the expertise f the instructr and can prvide the necessary mnitring f the students activities whilst the undertake an peratin, additinal facilities have t be available t prvide the student r t ensure they already have the required underpinning knwledge that the need befre they can undertake the peratins. T meet this requirement, we have develped curses in undertaking the carg peratins cncerned fr each f the mdels. Included in the curse are: a) Tw familiarizatin exercises which teach the student bth hw t lad and perate the simulatr, hw t lad and lg int the WISE mnitring system and hw t perate the varius equipment and facilities in the particular mdel cncerned b) A number f exercises undertaking specific stages f peratins. Fr example in the carg peratins, the exercises will guide them thugh undertaking mst f the peratins that can be expected t be undertaken in a dry dck t dry dck time frame. c) Reference sectin which prvides detailed answers t all the questins asked thrughut the curse d) Operatins manual fr the ship r equipment type Each exercise includes the fllwing: Aims and Objectives Instructins n hw t set up and prepare the simulatr Explanatin f the situatin at the start f the exercise 08-M-WISETECH 8

11 Questins t test and prvide the underpinning knwledge required t cmplete the task. Detailed instructins n what they shuld d thrughut the curse f the exercise Prvisin f Underpinning Knwledge The curses are prvided in a pdf frmat which allws them t be easily viewed n any PC. In additin interactive functins are added t the dcuments t allw the students t bth test themselves and ascertain the required underpinning knwledge if required. An example f a typical entry is as fllws: T ascertain a quick answer the student clicks n the? symbl. A brief summary f the answer is then displayed. If the student finds that they require mre detailed infrmatin they can then click n the Mre Inf buttn which takes them directly int the relevant sectin f the reference material where the full answer is prvided. When satisfied the student can then return t the riginal lcatin and cntinue wrking thrugh the exercise. Supprting WISE Rules T accmpany the curse material, specific rules are als generated in the WISE Mnitring system specific fr each exercise which track the student as they cmplete each f the requested tasks with the apprpriate entries being recrded int the students lf file. Curse develpment facilities As the curse are prduced using a pdf frmat it is a simple task fr instructrs t create their wn curse material in Wrd r anther suitable wrd prcessr. This is then cnverted int the required pdf frmat where the interactive facilities can be added. If requested detailed instructin n hw t d this can be prvided. 08-M-WISETECH 9

12 WISE Autmatic Cmpetence Assessment Using the infrmatin cllected by the WISE system the new Autmatic Cmpetence Assessment System will be able t prvide an immediate assessment f the cmpetence f the persn undertaking a particular task. The instructr will define which elements f a task shuld be marked in the psitive r negative. The system will then autmatically assign apprpriate scres the rules cncerned and as the task is prgressed will evaluate the scring withut any further input frm the instructr. Thrughut the task a graphical representatin f hw the student is perfrming is displayed and nce the task being assessed is cmpleted the system will indicate whether student has shwn that they are either: Cmpetent in undertaking a cmplete r part f a task Nt cmpetent and that further study is required Further evaluatin is required. The system is nw in the final stages f develpment and testing and is expected t be released Q System Descriptin When used in an assessment mde (e.g. t assess student s ability t undertake a task) the feedback system t the student is disabled (althugh the infrmatin can still be frwarded t the instructr if required) and hence the student is nt prmpted with any infrmatin and left t prceed accrding t ability. Using the infrmatin cllected by the WISE mnitring system previusly described, tgether with rules defined especially fr the assessment prcess f a particular activity WISE will cntinuusly mnitr the n ging activity as the task is cmpleted. The actual assessment f the student s ability is undertaken using pattern recgnitin techniques based n the implementatin f Bayes rule. Bayes rule is based n the use f prbability and is a recgnized mathematical rule that is used extensively by statisticians t assess evidence t supprt a particular hypthesis. In the case f WISE, the hypthesis is based n the identificatin f knwledge r ignrance. The implementatin f Bayes rule autmatically assigns an apprpriate scre t each f the rules being mnitred and during the curse f the assessment the sum f the scre frm all the rules is cntinuusly recalculated. This scre is cmpared directly with a set value which changes depending upn the activity and hence rules being mnitred. As well as the scring, WISE cntinuusly calculates a range f uncertainty beynd which passes are unquestinable and belw which fails are certain against which the students scre is cmpared. The system will then utput the result as either: Successful in cmpleting the task (Cmpetent) Unsuccessful at cmpleting the task (Incmpetent) Partially cmpleted the task successfully 08-M-WISETECH 10

13 Assessing a Scre An Area f Uncertainty Fr each test the system calculates the weights s that the best distinctin between Pass and Fail patterns can be made. Hwever, despite the attempt t calculate weights that shuld, in principle, prvide a clear-cut pass mark there is a prblem in that such a mark (see Why Marks dn t Add Up ) mre ften than nt des nt exist. There is usually an area f uncertainty abve which a Pass is certain and belw which a Fail is als certain. The area f uncertainty is determined frm the bservatin that all the Pass patterns will fall within a range f scres and that als all the Fail patterns will als fall within anther range f scres. The prblem arises when these tw areas verlap. The scre weights are adjusted s that the maximum scre can be specified in say There is n limit set n the scre less than that specified s it can reach a high negative number. The range f Pass and Fail scres are determined by running all the pssibilities f each set thrugh the scring system. There are tw ther values that are calculated. The first is the prbability f a Pass pattern which is calculated by: Prbability f Pass = Number f Pass Patterns / Number f Patterns and the secnd is the Mean Uninfrmed Guesser s Scre (MUGS) which is calculated using the functin: MUGS = 0.5 * Sum f the weights. and is based upn the assumptin that each switch r rule is independent and has an equal chance t be set n r ff (0 r 1). The initial scre is set at the mean f the uncertain range. This is because such a mark indicates n decisin can be made either way and this is the state f an untested examinee. It might be argued that the last mark given t a tested student shuld be the start cnditin since this is the knwledge abut the student s far. If the prbability f a pass pattern cmpared with all pssible patterns * scale is within this uncertain range then the test is cnsidered unsafe. An UNSAFE TEST warning is displayed when this value falls within the Pass range r abve the Fail range if there is n uncertain area. S the mre likely an examinee can guess a Pass pattern by a randm chice f a pattern frm the set f all pssible patterns the mre unsafe the test is. Similarly the less likely a guess may nt prduce a failure the mre unsafe the test is. This als assumes that all Pass patterns can be distinguished frm Fail patterns. This chice r prbability based scre des nt take int accunt the actual scre the examinee might get thrugh guesswrk. 08-M-WISETECH 11

14 Setting up f the Assessment T set up an assessment f a particular task r cmplete scenari the fllwing is required: Identificatin f which rules frm the rule library shuld be included in the assessment prcess Which f thse rules shuld be seen as a psitive, and thse that shuld be seen as a negative Relative imprtance f each f the rules in each grup against each ther, ie which f the rules will have a mre significant impact n the verall scre if they are tested as true. T allw this t be dne easily, the instructr is prvided with a simple interface via a Rule Generatin Tl which will allw them t: Quickly identify which variables are t be mnitred and the required criteria. (The instructr can access any variable within the simulated mdel t be used in a test) Develp rules fr use bth in prviding infrmatin t the student during nn-assessment exercises and purely fr assessment purpses. Cmbine the rules t prduce simple thrugh t very cmplex test cnditins Define the required assessment criteria regarding Psitive / negative attributes Imprtance Accuracy f final result (taking int accunt dead band between passing & failing test) Once defined, the system will autmatically assign the apprpriate weights r scres t each rule prir t the assessment cmmencing 08-M-WISETECH 12

15 Mnitring f the assessment Thrughut the assessment the instructr will be prvided with regular infrmatin with regard t hw the student is perfrming against the defined gals. The infrmatin will be displayed bth in a text and graphical frmat. In the fllwing example, three tests have been defined. Each f the tests is based n the mnitring f the same rules, but the respective imprtance f the rules when the scres are cnsidered have been changed, ie sme have been defined as being mre significant than the thers. The higher the graph, the nearer the gal f the task r scenari is being achieved. As can be seen, the cnsequence is immediately visible n the trend f the results. The purple line in particular shws very clearly hw the system will recgnise a particular result and then takes int accunt if the persn being assessed reassesses their actins by crrecting an identified mistake (in this case). In additin t the immediate text and trend utput the running scre f the assessment will be able t be requested by the instructr at any time during the assessment. In additin the verall results will be utput t a lg file and trend recrding system fr later evaluatin and archiving. 08-M-WISETECH 13

16 In additin t the graphical display prvided abve, thugh ut the exercise the instructr is prvided with cntinuus feedback abut hw the student is prgressing in cmparisn t the defined test by the use f a tree type f display. Each number represents an individual rule r test and the icn alngside changes as the relative scre is changing cnsequently prviding the instructr with an immediate indicatin f the students prgress and any prblem areas that they may be encuntering. Benefits The majr benefits f the WISE Assessment system are: Instructrs d nt need t becme invlved in trying t define the very cmplicated scring mechanisms required when trying t assess cmplicated tasks. The system takes full accunt f the student s effrts in trying t reach a successful utcme. Fr example if at the early stages f a task they cmmit an errr, this will result in a negative scre being applied t their ttal at that pint. If hwever they subsequently initiate the crrect steps t crrect the errr the system will recgnize this and adjust the scre accrdingly. Use in Assessing Industry Standards The new assessment system can be used t assist in the prcess f evaluating peratrs against the criteria defined by industry standards such as thse recently intrduced by the Sciety f Internatinal Gas Tanker & Terminal Operatrs (SIGTTO) and DNV SeaSkill relating t training fr LNG Carriers. In this cntext the WISE system can be supplied with a cmplete set f pre defined scenari cnditins fr the simulatr that may be used in the assessment prcess. In additin the WISE Assessment System will be supplied with a database f apprpriate assessment criteria fr each f the exercises t allw an instructr t quickly begin the assessment prcess and assist in their final evaluatin f a student. Whilst pre defined criteria will be supplied, using the facilities prvided an instructr will be able t adjust the assessment criteria fr each scenari allwing them t adjust it t suit the particular standards and apprving authrity fr the curse being delivered. 08-M-WISETECH 14

17 WISE Dcumentatin The fllwing dcumentatin will be supplied Lchs Sftware instructin manual Mdel Instructin Manual as apprpriate Crude Oil / Prducts Tanker VLCC LNG (Membrane) Carrier LNG (Spherical) Carrier Carg Operatins Manual Crude Oil / Prducts Tanker VLCC LNG (Membrane) Carrier LNG (Spherical) Carrier 08-M-WISETECH 15