Taking The Measure of Risk: Benchmarking Risk Management

Benchmarking or Benchsetting?

For any business function, one key sign of having ‘arrived’ is the pressure to ‘benchmark’. Benchmarking has been a popular past-time for organisations over the past decade. Entire books have been written on the subject but, in a nutshell, benchmarking is a process of comparing two or more business processes in order to understand how to improve them. The processes can be compared within an organisation (e.g., a multi-national contrasting finance processes in subsidiaries in different countries) or among different organisations (e.g., a ‘club’ of companies sharing information on their finance processes).

As a business process, risk management is well suited to benchmarking, however, risk management benchmarking does have a few idiosyncrasies. For instance:

• risk management is a new process: benchsetting (i.e., standard setting as opposed to comparison) may be as much of the benchmarking process as detailed comparisons;
• risk management is a holistic process: unlike some operational processes, good risk management affects the entire organisation and has diffuse benefits so it can be difficult to limit the scope of the benchmarking project;
• risk metrics are in their infancy: benchmarking studies must do some groundbreaking work to deliver quantitative results rather than qualitative comparisons.

Benchmarking, in any area, is not as straightforward as it looks. At first glance, it seems simple to garner comparisons. Nevertheless, the problems can be enormous – Are we comparing like with like? Can we define risk management in two different organisations? Can we trust the people who are providing the comparisons? Have people just given us easily available numbers or have they worked hard on them? How do we reconcile different levels of information within different organisations? How do we fairly compare organisations of different sizes? How can we ensure that the information has been validated: interviews, published accounts? Etc.

Despite the difficulties, good benchmarking has been the starting point of many successful change programmes. Equally, poor benchmarking has scuppered change programmes or has overlooked opportunities for improvement. Practitioners of risk management should welcome benchmarking as a sign that their function has become more crucial to their organisation.

Progress to Date

There is a tremendous amount of literature and advice on benchmarking. When benchmarking across organisations, an independent third party often provides confidence in data gathering and comparisons as well as some anonymity for sensitive data. Many trade associations, institutes, publications and consultancies (Z/Yen included) provide a valuable role helping their clients to determine what to benchmark, when to benchmark, how to plan the process, how to manage the process and how to interpret the results wisely and sensibly. Many semi-permanent benchmarking clubs or information exchanges exist already for a wide variety of activities, e.g., treasury transaction costs, manufacturing set-up times, facilities management standards, financial processing cycles, property investment returns, information technology performance or banking charges.

To date, most risk management process studies have been qualitative. A number of studies have been conducted. One example is a 1997 comparison of risk management among six multi-nationals – Fluor, Gillette, British Aerospace, Schlumberger, Microsoft and Northrop Grumman. These industry leaders in risk management, despite the obvious differences in their business areas, attempted to compare their principles and procedures, briefing and communication, risk transfer and financing. Although the study was entirely qualitative, the final result was an illuminating document. Some organisations were risk-exposure driven, some functionally risk-focused, some site driven. The rate of business change seemed to affect the type of risk management approach chosen, yet certain communication processes seemed common. All six organisations gained new perspectives in risk management. Risk management benchmarking studies have been conducted in shipping, property, health, oil and other sectors.

Despite the recent introduction of risk management benchmarking, some common themes are already emerging from the initial studies. Clive Moffatt of Moffatt Associates notes: “Competitive pressures and compliance (e.g., Turnbull) are forcing major companies to integrate their approaches to managing business risks. Our research has revealed that the key obstacles to progress are (a) internal conflicts over the scope and ownership of risk management and (b) the lack of a common communications platform.”

Hitting a Moving Target

Benchmarking can be an expensive or lengthy process. The first stage, as in any major project, is a clear definition of the objectives and scope accompanied by a statement of the anticipated benefits. The process of setting objectives is often iterative: we find out what is possible, perhaps a competitor wishes to share information, and then re-examine the objectives, scope and benefits. In benchmarking programmes we have seen, the clearer and simpler the objectives, the better the end result. The objectives, scope and benefit need to gain support within the organisation before we move on to planning the project.

The next stage in benchmarking is a thorough consideration of what constitutes comparable organisations. A number of questions show the flavour of this stage: are we trying to learn best practice or see how comparable organisations tackle our sort of risk problems? Are we going to learn more from people like us, or from people outside our sector working with different problems? Will our own people find X or Y organisation more credible? Are our people going to believe a study which does not include Z organisation? Can we get outside sponsorship? How will we encourage people to share sensitive information? What benefits will other participants get from the benchmarking?

Because benchmarking intrinsically involves third parties who provide the comparisons, this dependence needs to be addressed early on. Can we do all or most of the benchmarking from the outside or do we need direct contact with the comparative organisations? Whom do we know in these organisations? Can trade associations help us? Organisations must be sampled at this early stage to assess their genuine interest and commitment. In principle, many organisations are open to benchmarking, however the timing and the effort may not coincide with their current plans. Benchmarking will need to be sold to them.

There is a large amount of detailed work involved in designing the comparative assessment, data gathering, analysis and conclusions of benchmarking. In design the benchmarking analysis, it is helpful to have a structure. Risk management, in common with other business processes, is a living system. Therefore, risk management benchmarking involves designing a set of comparisons which can be applied to a system. By way of comparison, it is as if we asked, “what is the best database?” Although we can examine inputs and outputs from databases in raw numerical terms, such as the amount of data handled in a period of time, this is unlikely to generate a helpful answer about the best database. The correct question is far more difficult to answer, “which database is best fit for the purpose we need?” We therefore need to understand the purposes and contexts of risk management systems in order to compare them usefully.

As a living system, we can structure our risk management benchmarking around standard system components. One model, specifying seven standard elements, is widely used by Z/Yen. The seven elements are drawn from cybernetics – inputs, processes, outputs, feed-back, feed-forward, monitoring and governance. By considering each of the seven elements, we can begin to design our approach to benchmarking. For each of the elements, we might like to consider some of the following areas for comparison, in no particular order:

• inputs: expenditure on risk management, management time spent, related insurance expenditures, compliance time and effort, external advisors and costs;
• processes: risk identification, risk assessment, risk management implementation, risk audits, risk transfer and financing, modelling, documentation, education;
• outputs: numbers of risk assessments, training days, scale of communications and any other numbers dealing with efficiency;
• feed-back: looking at measures of effectiveness in measurement and reporting structures, outcome measures, risk reduction or mitigation measurements, reductions in the cost of risk, event and impact comparisons, testing wider awareness in the organisation;
• feed-forward: targets and objectives, motivational structures, risk-based planning, event horizon scanning;
• monitoring: target and objective setting, structures assessing payback, communications and briefings;
• governance: risk strategy setting process, organisational inclusiveness in decisions, seniority of governance, independent reporting route(s) to the board, policy inhibitors, policy trends.

Using Results

Having specified the areas for examination, we can then design interview structures, questionnaires or data analysis exercises. In benchmarking, as in any major project, it is easy to list a long series of ‘essential’ factors for success, ‘crucial’ pointers for taking actions, etc. The design of a full scale project is too involved task for a short article. Jumping to the end result, benchmarking will typically take the form of comparative tables – we don’t do this, they do; this is our % of risk assessed sites, this is their %; here is our near miss rate per £ of turnover, here is theirs. This is fine, as far as it goes, but the essential purpose is to obtain benefits through understanding how we compare with other organisations and then taking appropriate action to improve by using what we learn.

Results need to be used. Workshops and other mechanisms need to communicate the results and confront managers with performance. Managers need to develop theories about what can use improvement. Managers need to be challenged to change their behaviour in order to reach the goals which benchmarking implies can and should be achieved. In the end, analysis can always be more detailed; comparisons will always be slightly unfair; results will probably be somewhat incomplete. The benefits, however, are not always in the end result. Frequently, the very process of benchmarking prompts much needed thought on why we conduct risk management and what we expect from it.

Organisations obtain benefits of many different sorts from benchmarking:

• new levels of performance: learning about other organisations expectation levels. One organisation with which we worked set about contrasting their risk management with internal audit, sparking a healthy internal competition in efficient practice;
• new targets: seeing what others expect and how they measure it. In one instance, an organisation which thought of risk management as an overhead, turned its measures into those of a profit centre;
• new ways of working: learning tools and techniques from others. One organisation started using cost/benefit analysis to determine the areas of greatest potential benefit from risk management techniques. The emphasis of risk management was shifted from large risky projects which had full-time project management attention to more mundane operational work activities. Another started to require demanding incident reporting times;
• new roles: changing the essential core of the organisation’s risk objectives. Lately, many organisations have begun to find ways to give risk management more ‘teeth’. Some of the techniques include internal insurance markets, genuine profit centre deductions, using risk management as an internal certification agency or infusing risk management into other audited systems, e.g., ISO9000.

In summary, benchmarking is an opportunity for risk management functions to learn. Through learning they can become more effective and more useful. While we still have a long way to go before there are common, comparable risk metrics, the increasing amount of benchmarking is forcing us to learn what we expect from risk management and how we might measure the results.

[A version of this article originally appeared as “Taking the Measure of Risk: Benchmarking Risk Management”, Handbook of Risk Management, Issues 35, Croner Publishing (10 December 1999) pages 5-8.]