AI developers face a difficult problem: How do you train a system to recognize and avoid a catastrophic outcome when that outcome rarely occurs?

Enterprise risk management leaders face much the same challenge. The ERM Initiative’s thought paper, How AI Development Techniques Present Useful Analogies for Improving ERM Processes, explores how techniques used in artificial intelligence and reinforcement learning provide useful analogies for designing more effective enterprise risk management (ERM) processes

Reinforcement learning offers ERM leaders a useful way to think about rare-event risk, proxy metrics, incentives and whether risk-management processes are actually producing better decisions.

Key Takeaways on Using AI Development Tools to Improve ERM Practices

This thought paper highlights three important parallels between reinforcement learning (RL) and enterprise risk management (ERM):

1. The Sparse Reward Problem → “No News Is Good News” Bias

In reinforcement learning, an AI agent can struggle to learn when success or failure occurs infrequently. ERM operates in a similar environment. Catastrophic events such as major cyber breaches, liquidity crises, or serious safety failures are fortunately rare, leaving executives with limited experience from which to determine whether existing risk-management actions are actually effective.

The danger is assuming that because a disaster hasn’t occurred, the organization’s risk management must be working.

2. Continuous Dense Proxies → Information Overload

AI developers compensate for sparse outcomes by giving autonomous agents frequent feedback through multiple proxy measures that provide feedback about errors and successes related to specific steps in a decision path leading towards a desired outcome.

Organizations do something similar through key risk indicators (KRIs), key performance indicators (KPIs), dashboards, risk reports, and other metrics. Those measures can provide valuable signals, but they remain proxies for the underlying risk.  They merely capture signals of events that might be related to the risk event of concern, but they aren’t exact measures of that event.

The danger arises when executives become increasingly sophisticated at monitoring the proxy without knowing whether it actually predicts the risk they are trying to manage.

3. Reward Hacking → Managing to the Metric

An AI agent can discover ways of maximizing its programmed reward without accomplishing the ultimate objective. This behavior is often called reward hacking.

Business leaders can face similar incentives. When performance evaluations and compensation become tied to particular KPIs or KRIs, managers may optimize the metric rather than the underlying objective. This reflects the risk associated with Goodhart’s Law: when a measure becomes a target, it can lose its usefulness as a measure.

What Can ERM Leaders Do About It?

The paper identifies six practical considerations for strengthening ERM process design:

1. Evaluate the quality of the process, not merely the outcome. Assess the rigor of management’s risk analysis and decision-making rather than assuming a good outcome proves that risk management was effective.
2. Create synthetic feedback through scenario exercises. Use techniques such as war-gaming and adversarial simulations to expose executives to possible consequences of catastrophic tail risks that have not yet occurred.
3. Align incentives with the time horizon of risk. Consider whether executive compensation and vesting schedules extend long enough to capture risks that may emerge years after a strategic decision.
4. Establish hard risk boundaries. Treat critical risk limits as genuine constraints rather than allowing strong financial performance to compensate for risk-limit violations.
5. Look for KPIs and KRIs vulnerable to gaming. Examine whether managers can improve a metric without actually reducing the underlying risk.
6. Use hindsight to distinguish skill from luck. Revisit important decisions and ask whether successful outcomes resulted from effective risk management or simply because a material downside risk happened not to occur.

The central question for ERM leaders: Are our risk processes actually helping us manage the uncertainty that matters, or have we simply become good at managing the metrics?

Real-World Examples Explored in the Paper

The thought paper illustrates these concepts with examples involving Value at Risk models and mortgage-backed securities associated with the 2008 financial crisis, Wells Fargo’s cross-selling metrics that led to huge reputational and regulatory harm for the bank, and Boeing 737 MAX disaster, showing how proxy measures and incentives can potentially draw attention toward achieving an intermediate metric instead of the ultimate risk-management objective.

RL CONCEPT	AI CONTEXT	ANALOGY FOR ERM	SYSTEM RISK GOVERNANCE VULNERABILITY
Sparse Reward Problem	An AI agent rarely receives feedback from its errors or successes, or only after the entire task is completed, making it difficult for the agent to really learn the task.	Executives have to assess potential risk events that are inherently rare, so they don’t have sufficient information about what has been helpful in avoiding the risk in the past.	“No News is Good News” Bias: Absence of disaster is falsely presumed to be due to an organization’s “superior risk management.”
Continuous Dense Proxies	The AI agent is given lots of rewards and penalties for accurately completing individual steps, which helps the agent learn about successes and missteps at each step towards the ultimate goal.	Management monitors numerous daily, monthly, quarterly key performance indicators and key risk indicators that serve as proxies, but not exact measures, of a specific risk event of concern to management.	Information Overload: Software systems have glitzy charts and graphs, with management focused on continuous slicing and dicing of data while overlooking the fact that the proxy does not correlate with the risk to be avoided.
Reward Hacking	AI agent takes shortcuts or leverages loopholes to maximize rewards (e.g., points) for accuracy on the tiny steps with little focus on achieving the final outcome.	Executive compensation and performance plans incentivize management to focus on ensuring certain risk metrics (e.g., KRIs) stay within acceptable thresholds while overlooking a looming risk event.	Metric Gaming: Local managers optimize indicator scores by managing to the metric rather than taking actions to truly mitigate the risk concern.

Who Should Read This Thought Paper?

This paper will be particularly useful for:

• Chief risk officers and ERM leaders
• Boards and board risk committees
• CFOs and senior financial executives
• Internal audit leaders
• Strategy executives
• Risk and governance professionals
• Researchers interested in AI, governance and enterprise risk management

About the Author

Don Pagach is Professor of Accounting and Director of Research for the ERM Initiative in the Poole College of Management at NC State University. His research examines enterprise risk management practices, how capital markets value ERM, and the integration of ERM with sustainability. He also contributes to the ERM Initiative’s annual Executive Perspectives on Top Risks research conducted in partnership with Protiviti.