Probabilistic Risk Assessment (PRA) 's growing popularity with regulators & industries - A behind the scenes view
Are they using it right?
Suppose that your financial advisor calls you on the phone and says,
“Your investments are doing well. But, we are recommending that you adjust your portfolio according to our new recommendation. We have run a sophisticated and detailed risk analysis, and we calculate that, on average, clients who follow our recommended portfolio design will loose at least 40% of their entire investment wealth within 10 years.”
What??? Are you kidding? That’s ridiculous. No rational person would design anything on the basis of, “ … it performs at least this badly” risk analytics.
Yet, that is one way to look at what is being advocated by with Probabilistic Risk Assessment (PRA). What gives, here?
The problem with PRA the disconnect with the underlying math and physics. To be perfectly honest, it starts and ends with the “P” in PRA. Even though you don’t really need a lot of fancy math to recognize that PRA has a “performs at least this badly” bias. So, how did regulators like the NRC, BSEE, and PHMSA get so far off the rails when it comes to PRA? And, what does the pubic need to know about their safety when PRA studies guide regulators? To answer these questions, we will first identify where PRA came from and then explain what PRA tries to compute and why it fails.
PRA, as it is practiced today, begins with the “Rasmussen Report.” In 1975, the NRC published NUREG 75/014, the “Reactor Safety Study - An Assessment of Accident Risk in U.S. Commercial Nuclear Power Plants”, a 1400+ page study lead by MIT Professor Norman Rasmussen who was tasked with “… trying to reach some meaningful conclusion about the risk of nuclear accidents” in commercial electric power generating facilities.
Rasmussen claimed that he could rigorously quantify the risk of a nuclear accident using modern mathematical models calibrated with available operations data. Rasmussen understood that his predictions could not rely on historical commercial nuclear accident data as none was available (the Three Mile Island meltdown would not occur until a few months after publication of NUREG 75/014).
Rasmussen’s central idea was that a stream of arriving trigger events (operations anomalies that could lead to accidents and for which there was plenty of calibration data) could be bifurcated by the reliability of nuclear plant protection, thus, allowing him to compute the rate at which accidents would occur.
PRA is a risk analysis methodology devised to predict the ‘frequency’ of industrial accidents. The word frequency can be confusing. In the PRA context it means “the number of accidents per unit time” … or accident rate. The rate of industrial accidents is a constant (e.g., 0.002 accidents per year which is the same as 1 accident every 500 years).
Of course, accident rate is not especially informative when it comes to understanding the risk of a nuclear reactor meltdown. In practice, ‘rate’ predicts a long-run average number of accidents over time; it tells you nothing about how soon the next nuclear accident is likely to occur. For example, suppose PRA predicts an accident rate for the commercial nuclear industry of 0.002 accidents per year over the long-haul.
This may seem like an acceptable public risk, but if you happen to be living next door to a 50 year old nuclear power plant seeking to extend its operating license, an accident rate of 0.002 is uninformative for your immediate purposes. You are far more concerned with the immediate threat of an old nuclear reactor that what things look like in the long haul.
So, why didn’t Rasmussen simply compute the likelihood of an imminent meltdown instead of focusing on accident rate? The answer is two fold: (a) He didn’t know how, and (b) even he had known how, there is not enough data to support accurately calibrating the likelihood of an imminent industrial accident.
Aside from the fact that accident rate is a limited risk metric, PRA is encumbered by an optimistic bias.
That is, under all practical circumstances PRA will predict an accident rate that is less than available information should inform. PRA’s under-prediction of accident rate is a consequence of two very practical physics-based facts:
The stream of arriving trigger events is not in stochastic steady-state. Atmospheric physics, alone, dictates climate change induced trigger events like floods, tornadoes, and hurricanes are not in stochastically stationary (i.e., the likelihoods events are not changing over time).
Worse yet, if there are exist yet unknown technology-based trigger events (a very common circumstance with new technologies), then the trigger event stream cannot be stochastically stationary. When the stream of trigger events is not in a stochastic steady-state, PRA’s prediction of accident rate is guaranteed to be low.
Trigger events’ arrival times almost always depends on their history. For example, any trigger event induced by a technology wear-out failure guarantees that the arrival time of the trigger event depends directly on the technology’s operating history. Any time that trigger event arrival times depend on past history, PRA’s prediction of accident rate is guaranteed to be low.
In light of items 1 and 2 above, it doesn’t take too much effort to figure out that PRA gets into trouble because it ignores lots of information. Note that in item 1, stationarity means that event likelihoods are not changing over time, in which case you don’t need to keep up with information in the form of historical changes in order to see what’s going on. While in item 2, independence of history says that history information is not useful for predicting the future.
But, climate change, unknown-unknowns, and technology wear-out are reality, and this reality makes an already marginally valued risk metric like long haul accident rate even worse because it dismisses information about how things have been changing.
So, why haven’t regulators reconsidered accident rate (what PRA tries to accurately compute) in favor of a better risk metric? Well … because, there is still a belief that PRA has use-cases and validity. The gap in what the mathematics behind PRA offer versus what is being assumed it delivers continues to demand a reckoning.
Is it fair to bash the late Norman Rasmussen for not recognizing the flaws of PRA. Well … maybe, may not?.In the 50 years since the Rasmussen Report purveyors of PRA have not furthered or challenged the math used on a daily basis. There you have it. A risk analysis that reports a constant that allows regulators to have an “at least this bad benchmark“ a the long haul industrial accident rate.
Finally, one might perceptively ask, “What should replace of PRA?” The answer is really straightforward: “Prescriptive protection design requirements.” Prescriptive requirements (sometimes called deterministic requirements) have been successfully used by design engineers and regulators for centuries.
As the terminology ‘deterministic’ suggests, it would get safety regulators and the hazardous industries they regulate out world of probability mathematics. The irony is best described by the great reliability engineer Ralph Evans who once told me by way of analogy, “Babies and razorblades are both really good things to have, but not together.”