PBR MODEL GOVERNANCE CHECKLIST: Some Considerations for Practicing Actuaries 2016 American Academy of Actuaries. All rights reserved. May not be reproduced without express permission. PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Agenda Introductions Why Model Governance? Purpose of the Checklist Categories of Model Governance Wrap-up/Q&A 2 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Introductions Peter Weber, MAAA, ASA Brian Fomby, MAAA, FSA Mike Failor, MAAA, ASA
Why Model Governance? It is required by the Valuation Manual Good actuarial practice An important tool to mitigate model risk Part of exposure draft of new proposed ASOP on modeling
Why Model Governance? SVL, Section 12, B A company shall establish procedures for corporate governance and oversight of the actuarial valuation function consistent with the Valuation Manual (VM). This includes a certification to the company board of directors of the effectiveness of internal controls to assure that all material risks inherent in the liabilities and associated assets are included in the valuation. 5 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Model Governance VM-G Requirements The board is responsible for establishing a process, including the effectiveness of internal controls. Senior management responsibilities include determining a process exists that: Ensures models and procedures produce appropriate results. Validates data for determination of model input assumptions. Avoids bias towards underestimation of reserves. Ensures model input is appropriate given experience. Improves model performance where material errors and weaknesses exist. Provides regular cycle of model validation that monitors model performance, reviews model relationships, and tests model outputs against outcomes. 6 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Model Governance VM-G Requirements Qualified Actuaries responsibilities include: Reviewing and approving assumptions, methods, and models. Reviewing and approving internal standards for actuarial valuation processes, internal controls, and documentation used for reserves. Providing a summary report to the board and to senior management on the valuation processes used to determine and test principle-based reserves (PBR) to assist their understanding of results and any significant or unusual issues and/or findings. Working with internal and external auditors and with senior management to resolve significant issues regarding PBR. 7 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
VM-31 Model Requirements Include: Description of the modeling system(s) used. Description of approach and rationale to group assets and policies into model segments. Description of the approach used to validate model calculations within each model segment for both deterministic and stochastic models including: How the model results compare with actual historical experience. What, if any, risks are not included in the model. The extent to which correlation of different risks is reflected in the margins. Any material limitations of the model. 8 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Purpose of the Checklist
What the Checklist Is Intended as a potential resource for practicing actuaries and others involved in actuarial model governance In response to the need for good model governance addressed in PBR regulation Potentially useful beyond PBR valuation modeling and wherever actuarial modeling is performed A collection of question-based considerations for the purpose of organizational introspection across the model governance landscape Aimed to foster awareness of model governance issues Non-exhaustive A work in progress 10 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
What the Checklist Is NOT NOT intended as an instrument for rating or assessing an organization s level of model governance NOT prescriptive in any way, and not constituting a list of model governance requirements NOT a list of questions intended to be directed towards any specific individual, actuary, or group (as the questions are intended only for organizational introspection) NOT the answer to model governance, but a potential tool that may aid in its development 11 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Categories of Model Governance
How Questions Are Grouped General Guiding questions vs. Technical Focus questions General Guiding questions may be more suited for those that manage or govern a modeling process. Technical Focus questions may be more appropriate for those directly involved in the day-to-day modeling activities. Policy/Process vs. Validation questions Policy/Process level questions are intended to address existing model governance policies and procedures. Validation type questions are typically more investigative in nature and often require a more focused hands-on review. 13 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Model Governance Functional Categories Governance Standards Modeling Process Assumptions Setting Input Data/Tables/Mapping Access Controls System/Model Changes Versioning/Model Selection Consolidation of Results Reporting Analysis/Validation 14 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Governance Standards Formal, documented governance policy: Modeling environment (development, testing, production, etc.) Organizational design (including roles & responsibilities) Change management procedures Standards for documentation, testing, design, workflow, etc. Need buy-in from all stakeholders! 15 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Governance Standards (Sample Questions) Policy/Process level: Has a model governance framework been formally developed, documented, and adopted throughout the organization? Have model documentation standards been established and adopted throughout the organization? Validation level: Are model governance protocols strictly enforced during quarterly and annual reporting? (Bypassing protocols may indicate a lack of management buy-in, and/or the need to better address fast-track changes.) 16 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Modeling Process Development: Define requirements and enhancements Implement and test model change/construction Production: Execute models to produce model results Analyze and interpret model results 17 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Modeling Process (Sample Questions) Policy/Process level: Is there a formal model/system change request procedure? Are model outputs reproducible? (i.e., have the inputs and model versions been archived and has an operable version of the modeling software been retained?) Validation level: Is there a single source or repository that documents the data flow/system processing from end-to-end? 18 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Assumptions Setting Assumptions need formal governance: Process (including scheduling, testing, & documenting) Roles & responsibilities The level of expertise needed Approval of assumptions 19 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Assumptions Setting (Sample Questions) Policy/Process level: Has an assumption governance framework been formally developed, documented, and adopted throughout the organization? Does the framework include a centralized approach for common assumption setting and formal approval? Are experience studies performed on a regular schedule, and assumption updates performed at a consistent time each year? Validation level: Are assumptions documented and signed-off in accordance with a specified process? Are the impacts of proposed assumptions correctly estimated? 20 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Input Data/Tables/Mapping Asset & liability inputs: Assumptions Data (e.g., premium rates) Inforce files Need to govern: Data entry Changes to the data Storage of the data Access to the data 21 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Input Data/Tables/Mapping (Sample Questions) Policy/Process level: Are model input feeds obtained automatically from a centralized data warehouse? Has the organization automated and standardized a set of test analytics for validation of model input? Validation level: Was the correct seriatim input data used? Are data sources well defined and documented? 22 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Access Controls Access to modeling components should be controlled Input data Models/Systems Output/Consolidation Processes/Reports Environments Production Test/Development Types of access Read Write Run Create 23 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Access Controls (Sample Questions) Policy/Process level: Are access controls in place for models and modeling systems? Is there a documented process for granting access to models, systems, assumptions, reports, etc.? Are access levels reviewed on a regular basis? 24 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
System/Model Changes System access and change control processes are in need of the most improvement* Assumption validation and model output testing alone can miss many system coding errors Code changes require proper code management Beneficial to work closely with IT professionals * A Survey of Actuarial Modeling Controls in the Context of a Model-Based Valuation Framework (Copyright 2012 The Society of Actuaries, Schaumburg, Illinois. Reproduced with permission.) 25 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
System/Model Changes (Sample Questions) Policy/Process level: Are model and system changes achieved through a formal change management process? (Is there documented evidence?) Are there dedicated model/system/data stewards? Validation level: Are test packs (or test beds) kept up to date with new models and product features? Are system/model errors adequately documented? 26 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Versioning / Model Selection Model selection: Matching the model task Purpose Theory Accommodating product features Versioning All related modeling, aggregation and reporting components Older versions archived 27 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Versioning/Model Selection (Sample Questions) Policy/Process level: Have experienced modeling experts been involved in algorithm selection and implementation of modeling concepts? Are reports automatically labeled to indicate data and version sources? Validation level: Are all the modeling components versioned? (i.e., the code, reports, test cases, etc.?) Was the correct model and version used? Was the "purpose" of the model documented? 28 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Consolidation of Results Aggregation and reporting processes Data management underpinning reports Aggregation of multiple modeling results Increased Focus on validation Numerous one-offs Eleventh hour adjustments 29 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Consolidation of Results (Sample Questions) Policy/Process level: Is the reporting process documented? And, is there specific documentation of the mapping describing how reporting levels are linked to each other? Validation level: Have all products been accounted for in the consolidated results? Have consolidated results been adjusted to account for unmodeled business? Are late adjustments to consolidated results implemented across all related fields (i.e., reflecting unmodeled premiums, but forgetting to account for unmodeled reserves)? 30 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Reporting Crucial Component of Model Governance Questions are broad in nature Not restricted to a single reporting/accounting framework Model governance carried through report development Many process type questions Clarity Transparency 31 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Reporting (Sample Questions) Policy/Process level: Are automated data feeds used to populate reports, as opposed to manual report preparation? Do the reports indicate how robust or sensitive key figures are, along with impacts due to estimation error of input parameters? Validation level: Did the process owner and business owner sign off on the report? Are results communicated using institutionally accepted metrics that are commonly understood by the reporting audience? 32 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Analysis/Validation Broadly defined in this context to include the governance of validation processes across a spectrum of modeling activities Assumption setting Model input Systems/models Modeling results Reporting Often calls for expertise in systems validation and testing 33 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Analysis/Validation (Sample Questions) Policy/Process level: Has the organization planned and budgeted for the time and cost of analysis of results? Are assumptions (including both deterministic and stochastic) given an independent review? Are test coverage reports available? Validation level: Has an analysis of change from a validated model to a modified model been performed and documented? 34 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Resource Documents A Survey of Actuarial Modeling Controls in the Context of a Model-Based Valuation Framework (Copyright 2012 The Society of Actuaries, Schaumburg, Illinois. Reproduced with permission) System Access and Change Controls (Copyright 2012 The Society of Actuaries, Schaumburg, Illinois. Reproduced with permission.) Model Validation for Insurance Enterprise Risk and Capital Models ( 2014 Casualty Actuarial Society, Canadian Institute of Actuaries, Society of Actuaries, all rights reserved) (Copyright 2012 The Society of Actuaries, Schaumburg, Illinois. Reproduced with permission.) Report from the Actuarial Processes and Controls Best Practices Working Party (Institute and Faculty of Actuaries, May 2009)* *This report was prepared by and/or on behalf of the IFoA. The IFoA does not accept any responsibility and/or liability whatsoever for the content or use of this document. Institute and Faculty of Actuaries 35 PBR Boot Camp: Basic Training and Beyond for PBR Implementation June 6-8, 2016
Wrap-up / Q&A Principle-Based Reserve Model Governance Checklist Work Group Michael Failor, MAAA,ASA, Chairperson Mary Bahna-Nolan, MAAA, FSA, CERA Brian Fomby, MAAA, FSA Peter Weber, MAAA, ASA Michael Boerner, MAAA, ASA Cande Olsen, MAAA, FSA Ali Zaker-Shahrak, MAAA, FSA