Preparing for a CFI Outcomes Measurement Study Dr. Martin Kirk, Director of UBC ORS CAURA West Nov 2010 Saskatoon
WHY OMS? A valuable tool for the CFI, institutions and their partners The Outcome Measurement Study (OMS) is an important evaluation tool that helps the CFI remain accountable, make decisions and communicate the impacts of its investments to its stakeholders
1) PREPARATION
OMS Instructions for the Institution http://www.innovation.ca/docs/acco untability/oms/2010/oms_instructi ons_institution_2010.pdf
What is OMS? OMS Entirely Different from any audit most of us have ever undergone. Measuring the value of the research rather than how closely we have followed the funding terms/conditions Qualitative and Quantitative measures of research impact
OMS Looking for Realization of 5 outcomes 1. Strategic Planning 2. capacity 3. Highly Qualified Personnel 4. productivity 5. Innovation
Selecting the Theme CFI presents a research theme that defines the scope of the OMS study Obviously in an area where an institution has had significant CFI funding and has strong research competence. Negotiate with CFI to ensure theme/subthemes/projects make sense for your institution
Institutional Deliverables Data Report (submit draft-cfi reviewfeedback-final draft) List of project leaders (PLs) and PUs Principal Users (of the infrastructure) CVs of the PLs and PUs Site visit presentations in the outcome categories to a CFI Expert panel
Preparing for OMS: Advanced Planning Start 6-9 months in advance of the CFI site visit or as soon as OMS date confirmed by CFI Confirm a theme, subthemes, list of projects with CFI Establish an academic champion and team for OMS. Develop plan with timelines for development of the OMS report/collection of data etc Communicate OMS expectations to CFI project PIs Speak to peers and secure their OMS reports Read the expert panel assessment form
Communications Vast Collaborative Project Need platform like SharePoint Regular OMS team meetings Monthly teleconference with CFI/regular sharing of drafts Need skilled writer (research) to knit together compelling stories
2) COLLECTING DATA and Creating the Report
DATA Sponsored Revenue for PLs and PUs Utilization of Infrastructure Updated CVs HQP number/destinations Where did your graduates go from CFI projects OMS scope? Bibliometrics H-indices, citation impacts Level of social Networking/collaboration pre and post CFI Sharing of Infrastructure with external groups?
Data UBC had trouble with Tracking of instrument/infrastructure users, especially external Tracking of where our HQP went? Social Networking data Lots of manual data collation Scopus/COS: Data needs cleansing Analyses of data
Accessible Data?? If you are not able to deliver certain required data: Negotiate with CFI on acceptable proxy data.early in the process!
Qualitative Data: Mostly from PIs Engaging er Input: Critical Face-to-face meetings Surveys Collecting research impact stories (working with research communications) not just data! Presence at the site visit
Summary Enlist credible person to champion the OMS with authority and support of VP Ensure you have negotiated appropriate research theme, sub-themes and projects with CFI Develop solid plan for creation of OMS report with process, timelines and roles (PIs and staff) clearly understood and communicated Engage project researchers on day 1 communicate expectations of them including involvement in the site visit
Summary Con t Ensure you have up-to-date CVs of project PIs and PUs, tracking of infrastructure utilization and HQP destinations (CFI projects) Ensure you have access to data and advanced data analytics tools Incite and/or SciVal/Collexis and have data cleansing capacity Ensure you have capacity to deliver polished research stories and coherent data and data analyses.
Questions?