Beyond benchmarking: What are the key metrics that agencies and companies should use to measure performance?
4-5th October 2004, Lansdowne Resort, Virginia, USA
Why participate in benchmarking exercises?
This question was asked and answered at the CMR International Institute Workshop in October 2004 when regulatory experts from government agencies and pharmaceutical companies met to discuss metrics for monitoring regulatory performance. In a series of presentations regulators and industry speakers discussed the motivation for undertaking studies that are often time- and resource-consuming. Whilst motivations may differ between companies and agencies it was apparent that both have a common goal of ensuring that new medicines are made available to the patient in the most efficient and cost effective manner.
It was also evident from the presentations and discussions, however, that it is not enough to measure performance in terms of timelines and the speed of the review alone. The quality of the process, from the construction of the dossier to the ultimate regulatory decision must also be monitored and added to the equation.
‘Score card’ Proposal
This focus on quality was followed up in the Syndicate Sessions and led to proposals that the CMR International Institute for Regulatory Science should initiate a project to elaborate a system for collecting feedback after a regulatory review using:
-
- Scorecards on the Industry: Completed by the agency on the quality of the dossier, the robustness of the data and way in which the company interacted during the review process
- Scorecards on the Agency: Completed by the company on the openness, fairness and consistency of their interactions and communication with the agency before and during the review, including scientific advice, questions and negotiation of the final label.
It was agreed that this proposal and other recommendations from the Syndicates should be explored further at the upcoming Institute workshop, in December 2004, on ‘Building Quality into regulatory dossiers and the review process’
CMR Institute Benchmarking Study
Participants at the Workshop were in a unique position to receive, at first hand, the initial reports of a major six-year study ‘Benchmarking the regulatory review process’ that had recently been completed by the Institute. Five regulatory agencies, FDA, EMEA, Health Canada, the Australian TGA and Swissmedic had worked together with the Institute to provide data on new drug applications submitted in the years 1997 to 2002 and tracked through to July 2003. The methodology and study results were presented, in the opening Session, by Dr Neil McAuslane, CMR International Institute, and were the focus of the subsequent Syndicate discussions.
Regulatory Response
The six regulatory agencies in the study were all represented among the speakers at the meeting and provided insights into their objectives in participating. They also discussed their ongoing priorities which included:
- the implementation of Good review management principles (GRMP) in the US (Dr Sandy Kweder, FDA)
- responding to stakeholder expectations for a timely, transparent, predictable and consistent review process (Dr Robert Peterson, Health Canada)
- continuing the benchmarking study on quality management systems undertaken in preparation for EU expansion (Dr Bo Aronsson, EMEA)
- encouraging earlier inclusion of Australia in the global submission process for new drugs (Dr Leonie Hunt, TGA)
- learning from the practices of larger agencies –positive and negative – as highlighted in the study (Professor Samuel Vožeh, Swissmedic).
Regulatory Performance of Industry
The final Session included discussions on measuring and benchmarking the regulatory performance of industry. The presentation by Dr John Jenkins, FDA, underlined the need to focus on quality and suggested it was time to turn attention from agency to industry procedures. “We could have a perfect regulatory process”, he said, “but this will not result in an approval if the application you submit is not up to standard”.