(57r) Calculating Risk Earns a “Needs to Improve Score”

Calculating Risk earns a “Needs to Improve Score”

From a facility developer perspective all to often Quantitative Risk Analysis (QRA) performance is beginning to hide behind closed doors. QRA performance has developed some bad habits of late including vague input basis, hidden calculations and analyst inability to understand outcomes and their sensitivities.

Here QRA includes grouping any calculation based analysis, including CFD and CFD like effort, which allows/requires the analyst/engineer to develop and apply assumptions and figures into a computer based calculation. The basis with which these assumptions and figures are developed, supported, incorporated, and communicated has deteriorated to a point where analysts in some settings have a difficult time understanding what is really going on within their analysis. When "engineering judgement” as an answer as to why an assumption or figure was incorporated becomes common place it brings immediate questions into play about the validity and reliability of the inputs and the analysis as a whole. Yet, many developers and operators seem to have allowed this to occur. Is it because the client base is not knowledgeable enough of the models being run to ask the tough questions, or is it the belief in the analyst is so over whelming questions are not asked. Or, is it the developer feels they hired the “ expert organization” and because of their hiring process questions don’t need to be asked, or is it the analyst’s education and experience level that leads to the easy answer, "engineering judgement”. This condition is compounded by developers allowing the calculations themselves to be enclosed within a “black box” - where analyst tell developers, “the calculation" is a formula protected by corporate Intellectual Property rules and can not be released. Yet when questioned explain it further the analyst responds as if the questions insult their education or their reputation while being steadfast in protecting the calculation. So, even if the inputs are clearly defined and communicated and the calculations are transparent and understood, the analyst and the report the developer receives is often nothing more than a set of snapshot from the model. Here the analyst fails really analyze the outputs to gain an understanding why anomalies exist and what variable/inputs would create improved or deteriorating conditions. Developers need to understand the analysis, its inputs, calculations, results and what design changes impact might be encountered. 

Because of this need, the basis of the paper is to establish or reaffirm the expectations of how engineering analysis should be performed beginning with the Rule Sets and Terms of Reference, and ending with the reported Results and Findings. When it comes to quantitative analysis, the industry should expect a more scientific methods approach to their engineering analysis - incorporating validity, reliability and replicability elements.