Defining Success & Measuring Regulatory Performance


During this two-part seminar we will be discussing the meaning of “success” in a regulatory environment and the associated challenges of performance measurement and management. Professional regulators and law-enforcement officials certainly deal with customers, and so the normal range of customer-service metrics has some relevance for them. But their principal business is to deliver protections to society against various classes of harm, and towards that end they deliver and, where necessary, enforce obligations, influence behaviours, and manage compliance. The nature of “success” to be pursued and assessed is therefore quite different in the “protective” role of government than in service-provision roles.
Many public agencies select three or four metrics they think are particularly important, and designate them as “Key Performance Indicators,” drawing to them an inordinate amount of internal and external attention. Sometimes this focus can have perverse effects and bias operations in a way that is not quite as valuable as executives intended and may even distort operations in a way that is contrary to the public interest.
For regulators, much attention is typically paid to functional outputs (such as inspection or audit rates), to the rates at which agents discover or detect non-compliance, and to process metrics such as timeliness and accuracy of decision making. Connecting these measures to each other, and to valuable social outcomes, remains a complex puzzle.
This seminar provides an opportunity for attendees to assess their own organisation’s traditions in performance measurement and reporting, and—where necessary—to identify opportunities to broaden the range of indicators and analyses in a way that will enable their agency to present a more rounded and meaningful picture of their contributions to society.
Key Learning Outcomes

What it takes to “prove causality”. The meaning of “evidence-based” policy and practice, and how these ideas apply to regulatory & enforcement agencies.

The notion of an “information rich environment,”and making the distinction between management information & performance metrics.

The multiple purposes of performance measurement

Where “customer-service” metricsfit within regulatory operations, and where they do not

Classic ambiguities in assessing regulatory performance(e.g. When would finding more violations be good news, or bad? How should we interpret increased volumes of consumer complaints?)

What difference it makes to the performance account when regulators turn their attention to “risk-based” strategies

What it takes to “prove causality”. The meaning of “evidence-based” policy and practice, and how these ideas apply to regulatory & enforcement agencies

The role of program evaluation. Where it fits naturally within regulatory operations. Other forms of analysis and pattern recognition relevant to performance measurement

The distinctive challenges of performance measurement when tackling invisible risks, catastrophic risks, and risks involving conscious opponents (adversaries).

The implications of different regulatory structureson performance monitoring and evaluation

Professor of the Practice of Public Management, John F Kennedy School of Government, Harvard University
Malcolm K Sparrow is a leading international expert in regulatory and enforcement strategy, security and risk control. Malcolm is the Professor of the Practice of Public Management at Harvard’s John F Kennedy School of Government and he is Faculty Chair of the school’s executive program “Strategic Management of Regulatory and Enforcement Agencies.”
Professor Sparrow served 10 years with the British Police Service, rising to the rank of Detective Chief Inspector. He has conducted internal affairs investigations, commanded a tactical firearms unit, and has extensive experience with criminal investigation. His research interests include regulatory and enforcement strategy, fraud control, corruption control, and operational risk management. He is also a patent-holding inventor and deadly serious at tennis.
Agenda
All times are shown AEDT
Welcome remarks & virtual learning tips
Current practice & common puzzles
- Pressure for change in performance reporting
- The difference between management information and performance metrics
- Eight different purposes for measuring performance
Measuring risks-controlled, harms-reduced, compliance achieved, behaviours changed
- Populating the regulatory “cockpit”
- Experiencing the cockpit simulator
- Matching metrics to classes of work (functional, process-based, risk-based, and crisis-response)
What difference will “risk-based” strategies make to performance measurement & reporting
- Increased emphasis on problem-centric work
- The nature of the risk-control performance report (structure and components)
- Graduating from risk-reduction to continued risk-suppression: how the story changes
- The importance of vigilance, nimbleness, and skill
Open Q & A
Day 1 closes
Welcome remarks & virtual learning tips
Special performance measurement puzzles presented by seminar participants
- Chosen in advance via submissions
Making the connection between external conditions & internal (agency) actions
- What it takes to “prove causality”
- What is “evidence-based” policy and practice, and how does the idea apply to regulatory and enforcement agencies?
- The role of “program evaluation.” Where it fits naturally within regulatory operations
- Other forms of analysis and pattern-recognition relevant to performance measurement
Alternate regulatory structures & implications for performance measurement
- The distinctive challenges of performance measurement when tackling “wicked” categories of risk:
- Invisible risks
- Catastrophic risks
- Risks involving conscious opponents (adversaries)
- The role of random sampling in inspection and audit
Alternate regulatory structures & implications for performance measurement
- When regulators delegate, or share, parts of the risk-management task with the industries they regulate, who ends up being responsible for what?
- When things go wrong, which party has failed, and in what way?
- How does the choice of regulatory structure alter the nature of the performance report and affect appropriate forms of accountability?
Close of Day 2
Pricing
Early bird savings before 15 September
+GST
Early bird pricing
Save $500
Early bird savings before 6 October
+GST
Early bird pricing
Save $300
Early bird savings before 27 October
+GST
Early bird pricing
Save $100
Standard prices after early bird
+GST
Early bird savings before 15 September
+GST
Early bird pricing
Save $500
Early bird savings before 6 October
+GST
Early bird pricing
Save $300
Early bird savings before 27 October
Early bird pricing
Save $100
Standard prices after early bird
+GST
Bring your colleagues and save with a group discount
Groups of 5-8 save 10%
Groups of 9-11 save 15%
Groups of 12+ save 20%
A very engaging and thought-provoking session. Malcolm is truly an expert and it was so useful hearing his practical suggestions to the audience’s questions.”
Assistant Director, Strategy
Australian Financial Security Authority (AFSA)
A very engaging and thought-provoking session. Malcolm is truly an expert and it was so useful hearing his practical suggestions to the audience’s questions.”
Assistant Director, Strategy
Australian Financial Security Authority (AFSA)
A fabulous seminar, essential to all regulators involved in the performance measurement of the agency they work for.”
Business Intelligence Reporting Analyst
Oversight Strategy Branch | Regulatory Oversight Division, Civil Aviation Safety Authority