Data Analytics - An Outline
Incrasingly more companies and financial institutions base or augment their decision making on data and analytics. For audit to keep pace with expectations, more analytics and advanced technology has to be integrated into the review process. The sampling process traditionally used in reviewing leaves the risk to overlook important review areas such as key transactions, models or business activities. Hence, using advanced analytics for reviewing can have a significant impact on quality and outcome. The review of analytical models renders an area where the use of advanced analytics suggests significant benefits.
→ READ MORE ... Guide on Risk-Based approach: Financial Action Task Force (FATF)
Machine Learning models methods are meanwhile commonly used within the context of credit and fraud customer screening, marketing, and multiple other areas.
Interpretability of both model input and model prediction is important for obtaining the acceptance from business and user community. Importance of interpretabiliy applies especially for black-box models used in Machine Learnings uch as boosted trees and similar black-box type of models. Interpretation should therefore be considered a key step before deployment of model takes place which is why review should take place. The questions users and business may ask include to ask whether a particular feature is more important than other features. Fortunately, verifying interpretability does not have to be model dependent. In fact, the model agnostic nature of methods used for interpretability checks is highlighting their usefulness for use in review and validation across multiple application areas and model types.
Data Bias can have a significant negative impact on models. Degradation in model performance due to data bias is a key issues in other areas of model development too. One reason for data bias is that prediction models are typically built and calibrated based on data sampled over a certain time period, and therefore the sampled data are conditioned based on internal and external events occuring over this eriod. Another reason for data bias can be due to the choice of certain business strategy.
Compliance is an example for an area where attempts to prevent money laundering and to establish surveillance over transactions use machine-learning (ML) oriented approaches. Within this context, ML methods are meanwhile commonly used. Caveats can occur however due large numbers of false positives requiring operators to have to process all false positives of an outcome run. Clearly, a business objective in this regards is to minimize false positives and thus reduce the operational costs involved.
From the examples just sketched it becomes clear that evaluating ML model is a laborous and complex task with risks of foregone opportunities for performance enhancements due to errors in the model development and review processes.
→ Auriscon can help to enhance data analytics used in review and validation
Validity of data and data quality are therefore important aspects any review should opine on. Aspects of data quality such a missing data or outdated data should receive sufficient attention to summarize data limations. Testing of data quality can also consider the data quality monitoring with respect to relevant data quality dimensions viz. completeness (no missings), accuracy (no outliers), timeliness, controlled data sources, usage monitoring. Review of suitable and controlled Data Sources links to data integrity requirements. More often that not a central data repository is used and review checks may cover controls implemented to ensure data consistency across several IT systems. Governance aspects pertaining to definition of Critical Data Elements should be considered depending on the scope and the scrutiny of the review.
Regulatory expectations on data integrity and governance establish demand for detailed metadata repositories, data lineage and automated data quality checks to be implemented with sufficient granularity and currentness. Industry standards are set out holistically in the BCBC 239 report with internal reporting, regulatory reporting and management decision systems being in scope.
Furthermore, institutions are expected to have data quality management and standards in place with data quality indicators and tolerance level attached.
Consequently, technical leaned audit assignments should incorporate data quality and data governance aspects into the reviewing process.Typical challenges in implementing standars on data quality management are heterogeneous system landscapes where multiple source systems from which data are obtained are used across businesses and functions. Auriscon can help in identifying the inconsistencies and effectivity weaknesses that affecct the data quality measurement and reporting tools.
→ READ MORE ... ECB guide on effective risk data aggregation and risk reporting
- Thematic Drivers in Technical Auditing
Model Design
Methodology may have limitations or model design may be incomprehensive leading to bias in model outputs.
Data Quality
Data quality may have weaknesses leading to incomplete, inconsistent or inaccurate data with negative impact on accuracy of model outputs.
Documentation Quality
Poor documentation quality with lack of detail on important model and methodology assumption hinders effective validation and may lead to wrong model use.
Governance Effectiveness
Ineffective governance on thematic aspects such as data, models, IT, risk may lead to breaches with regulatory requirements.
Reviewing Model Design:
Typically the review part in this section concerns the model assumptions and limitations alongside a description of the model. Theoretical and mathematical exposition should reference literature and classify best practice whenever possible.
In addtion, the calibration process and details should be explained and assessed against potential gaps and weaknesses in relation to model uses.
The audit testing of adequacy of Model Design would be suitably expanded to cover comparisons to industry practice and model benchmarking.
Robustness of Model Design should stand the test of time and the test of stress. Risk drivers should be captured to the extent to permit model reactivity in times of economic stress.
Reviewing Model Performance:
Reviewing the Performance of the Model is best executed against clear criteria.
Common approach is to set up these criteria before model use and to establish a documented performance monitoring with tresholds on tolerated and alerted breaches of model performance.
Performance monitoring would capture model use and design issue, as well as issues pertaining to proxy data and model calibration.
The approach to escalation of breaches in model performance should be defined as part of Model Governance. A clear path for escalation reporting should be communicated and documented.