NHS Performance Indicators : February 2002
| Index | Contacts | HA PIs | Trust PIs | NHS Performance Indicators : February 2002 |
|
|---|
REPORT ON THE 2001 CONSULTATION ON NHS PERFORMANCE INDICATORS
INTRODUCTION
The performance improvement agenda is a key part of the NHS Plan to modernise health and social services. This agenda is underpinned by an improved system for collecting and presenting performance information within the context of the Performance Assessment Framework.
It is important to measure and publish performance information:
Central to this is the ongoing development and improvement of a series of headline and benchmarking performance indicators to measure quality and progress throughout the range of NHS services., and to highlight issues for closer examination.
From 2002, our intention is to publish the NHS Performance Indicators and Performance Ratings together, every Summer. The benchmarking indicators will be made available to the NHS for performance management purposes. The development programme will continue to include an annual consultation process.
This first consultation process started in May 2001 with the publication of the document: NHS Performance Indicators: A Consultation. This was circulated extensively throughout the NHS and other interested organisations. Issue of the document was followed by a nationwide series of seminars to raise awareness and knowledge about the consultation, to discuss associated issues and to encourage an NHS-wide response.
The deadline for receipt of responses was 8th August 2001. The responses have now been analysed. Initial results have already been used to inform the new Headline and benchmarking Performance Indicator sets. The results have also been used to inform the next consultation document, due to be published in spring 2002.
We have embarked on what will be a long process. Due to the nature of indicator development we expect it to be a number of years before we reach the stage where the indicator set matches our aspirations. Therefore, it may appear that the next indicator sets show little sign of improvement. Nevertheless, we are committed to developing a set which is supported by both the NHS and the public.
An essential, but often overlooked element of a consultation process is sharing the responses and feeding back the messages received from the process. Such feedback enables participants to see how their views are used, to see their views shared with a wider audience and to see what other participants have said. This document seeks to do just this..
RESPONSE
This exercise has sought to involve as wide an audience as possible within the heath industry. It received back:
The responses contained differing levels of detail ranging from comments on specific indicators to matters of policy. With such a wide audience some comments contradicted those made by others.
DOCUMENT STRUCTURE
The size of the response, the broad nature of the response and the varying depth of response made analysis of the results very difficult. In order to maximise the feedback we have structured this report around the nature of the information received.
Section 1 of the report follows the format of the questionnaire, giving a brief summary of the responses to each question, and highlighting the main points made.
Section 2 summarises any remaining general comments made that were not encapsulated by Section 1
Section 3 comprises a sub-section for each of the 25 policy areas, each containing a response that has been carefully considered by the policy team concerned.
OVERALL
There were a number of common themes arising from the consultation process:
|
The next consultation will be more focussed on how to improve the Headline set, but will also cover areas on which we feel we need to take more views from the NHS and the public. This will include areas which were omitted in the previous publication, and were highlighted in the response, for example, ambulance services. We intend to make the Headline set more closely aligned to NSF areas and other priority initiatives. We also intend to broaden the scope of the benchmarking set to improve the coverage of areas which the NHS finds useful. Work is ongoing to determine what effect "Shifting the Balance of Power" will have on the indicator sets. |
|
SECTION 1: ANALYSIS OF QUESTIONNAIRE RESPONSES |
Question 1. Do you use the PAF for local planning and development? (YES / NO)
65% of respondents confirmed they used the PAF for local planning and development. This two thirds response firmly suggests that THE PAF information cannot simply be regarded as a centrally imposed reporting requirement for central purposes only.
Question 2. If so, what benefits have there been from using the PAF?
The most significant benefits of using the PAF for local planning and development were identified as being:
Other benefits included:
The majority saw local benefits in using the PAF. Some organisations, for example, reported they were using the PAF in the planning of service development priorities and the creation of systems to bring things together. One organisation reported that it was looking to combine local health care data with local authority data so as to inform planning and the better targetting of resources
It was noted that there has been a distinct change of tone by shifting emphasis from the use of indicators as proscriptive tools to regulate and manage the service to being facilitative and educational tools to improve care.
a number saw the PAF as a central imposition that might actually prejudice further local use and local ‘buy-in’. It should however be noted that such comment often arose in conjunction with comments about the number of performance indicators.
... enthusiastic about the development of a full indicator set to compare and contrast performance but feel this can be achieved by a smaller set. Such a large set distracts resources. It is important that the statutory data requirement does not discourage the setting and managing of local priorities through central dictation or blocking of local resource
One view is concerned about the sheer size of the list when you look at all the indicators together, the data collection burden, and the feeling that too much is being dictated to local communities
Clinical involvement in all aspects of PIs is crucial. It is often stated that it is difficult to involve clinicians at local level in national indicators….
The use of the PAF in the development both of planning and of good data capture would result in the collection of contextualised data that would facilitate the further development of good practice.
In terms of local planning there were a number of concerns about the indicator set:
The current indicator set does not fully reflect the entire range of priorities within the local health/social care planning systems. There are, for instance no indicators on welfare to work, learning disability, child and adolescent mental health services, Action on Health and Poverty and community based services
It was noted that there is local responsibility for the indicators, but in practice, performance is subject to input from non-NHS organisations eg social services.
The inclusion of age and sex standardisation was welcomed. However it was pointed out that the use of the PAF was difficult in the area of health improvement due to a wide range of influences that are difficult to measure, not least a mobile population. There was thus a request that population / demographics data be used in the calculations so that information could be replicated at more local levels to ensure fairness, consistency and comparability:
as HFEA do for infertility so that apples are compared with apples.
The need for a consistent approach to undertaking long term analysis was widely recognised:
However users of performance measures have tended to better exploit indicators when the publication date is consistent, when there is a degree of involvement with users and when definitions are published in advance of data. There is much to be said for adhering to consistent definitions for a couple of years to enable trending and also for service change to be made over a period of time. On the other hand users of indicators do not wish for static measures but equally it is important that the service does not feel that definitions change every year
It was commented that the full extent of local knowledge may not be captured through a national consultation process and that this area needs permanent development either around the system for performance management or, failing that, around developing models / frameworks / balanced scorecards that can be of use.
Question 3. How could wider use of the PAF be promoted/supported?
The most popular suggestions made were:
Other significant suggestions included:
The PAF is intended to result in improved performance of services. Key to this is ensuring that the indicator set is relevant to existing targets and initiatives
Serious consideration needs to be given to the number of PIs being implemented and where their place is amongst all of the various NHS Plan targets, NSF milestones, HImp Targets, SAFF monitoring etc. In light of this perhaps guidance/advice should be issued by the centre as to how they feel the PIs can best be used
Integration with Traffic Lighting (now Performance Ratings) was also remarked on
The headline indicator set must be consolidated with Traffic Lights as soon as possible. We understand that these two sets will have differing compositions in 2001 but this must not happen in 2002
Appropriate relevance would encourage greater clinician support
We applaud and support the move towards greater accountability and to base that accountability on robust and relevant evidence. More particularly we support a culture of improvement where information can be shared within NHS organisations to enable a culture of continuous improvement to be developed. ... there is a strong argument for health care management not adopting practices until they are known to be effective. We therefore suggest where indicators are used that as a minimum confidence intervals should be used in analysing performance. Further and importantly large scale changes including the use of contentious performance targets must not be deployed until it is known that they improve the publics health
A great majority of respondents commented there were far too many indicators, making the process overbearing and thus meaningless. There were a great many suggestions on how to avoid this in the future
Keep it simple. ... Indicators should make sense and be a measure of quality and performance. … There should only be 6-7 headline indicators to identify hot-spots so as to inform decision making, resulting in improved performance. … If we are meant to deliver NHS Plan then PIs should reflect NHS Plan targets
A very small and manageable set of headline indicators might be used for performance management purposes and the much larger set can be used to build on the national performance assessment framework and be used locally for improvement
A common point of concern was data quality
Data quality has to be good enough to engage the stakeholders … The current data available in most areas is nowhere near this point
It was widely commented that production of indicators should be more frequent and that a number are based on out of date data eg the PAF indicators published in July 2000 relate to performance in 1998 / 99 - indeed some indicators cover the period 1996 / 98. There is thus a risk of defending a position that may be at least 12 months old.
|
The Department is working to bring the different parts of the performance management structure more into line. It is our intention to bring the PIs more into line with NSFs and other priorities. From July 2002, we will be publishing PIs and Performance Ratings together. Arrangements for performance management of the NHS will continue be streamlined so as to reflect the limited set of required actions set out in the Planning and Priorities Framework for (2002/03) We are working to improve the timeliness of data, and tighten up the process from data collection to publication. The indicator development process, including the annual consultation will both shape the indicator sets, and inform both the Department and the NHS of better ways of working, and better ways of improving performance. |
Question 4. What information do you currently use to performance manage your services?
The main forms of information identified were:
Other information forms included:
While the majority felt that performance indicators were a good thing, they would become more useful if their strengths and weaknesses were fully understood. This depended upon data collection exercises. However until data collection and quality issues were addressed the results should be treated with caution
Data collection in the NHS is given too low a level of priority. There should be a commitment to provide resources to improve the quality of data collection throughout. Part of the conclusion of the annual consultation process on PIs should be the publication of targets for improvement in data collection for the following year
Differences in re-coding activity against standards within and between trusts mean results have to be interpreted with care and intelligence. These have often resulted in bad publicity. Significant resources have had to be invested to monitor published results only to find adverse trends are often the result of misinterpretation of national data standards. Only when data standards are revised, training provided and submissions validated for quality and completeness will the benefits of benchmarking be realised
Question 5. What factors should influence the number and choices of indicators in the headline set?
The most popular suggestions were:
Other suggestions included:
It was commonly agreed that there are too many indicators. As such they represent a serious drain on limited NHS resources and could even mask some of the important issues. There were many suggestions about a reasonable number although as stated earlier the issues of data quality were felt to be closely related
There are many indicators that should not be included. Many need to be re-examined. Include, for example, only 2-3 per specialty but cover all specialties. There need to be at least 2 outcome indicators for each specialty … there are many missing. But current problems of poor quality data should be resolved first
It would be more relevant to have indicators that were meaningful to service delivery or clinical effectiveness, with perhaps 50, rather that 500, giving staff time to improve patient care rather than collecting data.
At any given level (e.g. from NHS Executive to PCT) there should be a limited number (probably no more than 12) high level indicators.
462 indicators in 25 areas is too much by a factor of 10 to manage effectively or draw proper conclusions, compounded by fact that the majority of the indicators cover areas where data is not currently collected - indicators should focus on areas where data is already available
and most interestingly
Performance management has become a self-perpetuating juggernaut and an example of Parkinson's Law. Together with other central directives (eg the NHS Plan and NSFs) it stunts initiative. The more people are bombarded by comments, the less they use their ingenuity. This may be harsh but the way central control of the NHS is going is reminiscent of Stalinism - admittedly a more benign form
There was a strong desire to avoid reinventing the wheel by duplicating information that was already available. This could be best achieved through the development of an overall framework, drawing in data already collected and used locally. The comparative data that was fed back would then also be meaningful locally.
With the emphasis now on primary care, it is increasingly important to link indicators to existing or emerging data capture. The capture of accurate, consistent, peer to peer data is seen as an issue that the use of the PAF will help to address.
It was agreed that the indicator set should reflect NHS Plan and modernisation targets. This would entail:
It was suggested that
any indicator that does not relate to NSFs, cancer plan etc, ie Himp priorities should be dropped and that as a matter of principle, all PIs should emerge from an NSF or equivalent process which will impose a certain rigour on the selection of PIs.
A common view was that there should be a systematic series of processes to review existing data flows and the identification of new dataflows; indeed that new ones should be discouraged until the existing ones had been fully utilised
As it can take up to 3 years to put mechanisms and procedures in place to capture data there needs to be a systematic approach to review the existing data flows and in prioritising the introduction of a selected number of new PIs requiring new data collection and then fast tracking the process of getting the new data collection approved. Such processes should review the effort to collect versus usefulness of the data and even undertake some form of cost benefit. It would be useful if the final document resulting from the consultation expresses a commitment to this
The centre need to recognise the difficulty involved in data collection in such circumstances, particularly where it requires professionals to collect data from individual clients. It would be very helpful to be clear about the need for new indicators before data collection. There is a sense that we need to be very clear about the appropriateness of old indicators and about whether or not they can be used before moving on to determining and collecting new indicators.
The American HEDIS approach was cited as a possible approach – this focuses on smaller indicator set, but also publishes others as being under development
It was agreed that indicators should be dropped where it is not possible to validate the source data
All agreed that the vast amount of data involved represented a major drain on NHS resources
Question 6. In what ways can we ensure that indicators are constructed so as to be fair and comparable?
There were two significant suggestions:
Other suggestions included:
The first of the two main suggestions concerned the issues of casemix / demography and socio-economic factors. The sections dealing with the 25 policy areas pick up on this theme constantly, for example the Dentistry heading and specifically when discussing General Dental Surgeries (GDS) and Community Dental Surgeries (CDS). The key issue is being able to compare ‘apples with apples’
Performance Indicators are positive so long as they are interpreted with other variables such as demography, social circumstances, urban issues - otherwise we will be comparing apples with pears
Need to be weighted for population base e.g. as HFEA do for infertility so that apples are compared with apples
It is widely recognised that this is not easy to achieve
I think it is reasonable that Performance Indicators include measures of efficiency, but to be meaningful must include casemix on a national scale … recognise that the whole area is fraught
In developing systems to allow for easier access to compatible and trend information, the influencing factors need to be clearly identified eg deprivation versus allocative efficiency versus levels of primary care development, social services support etc.
One respondent recommended the Audit Commissions approach
… Audit Commissions PAF family of comparators for Social Services indicators may help in identifying a common approach. They have an integrated services approach … development of indicators evaluating benefits of integrated working is to be welcomed. They see integration of Social Services and NHS indicators as being important
The second of the two main suggestions concerned the often cited need for consistency and clarity of data and indicator definition. The comment applies at a number of levels – national, health communities, PCTs etc.
Much collected data may be similar, but not the same, or is requested in different formats. It would be preferable if data sets and formats could be consistent across all these bodies and indeed across the 4 countries within the UK
Interface indicators - Slight concern around the basis for the collection of PIs by both Social Services and the respective health economy; Social Services have different client category definitions, collect information on a resident-basis rather than registered PCT basis which obviously will lead to inconsistencies in overall performance measurement and benchmarking e.g. delayed discharged for patients aged 75 and over - the information source for the indicator is the SAFF return; however the definition for the SAFF return states that the collection should be on a HA commissioner basis not solely providers within HA boundaries; is this consistent with the return that Social Services will provide which will make up their part of the interface indicator?
It is important that where possible, the data collected for performance indicators has the same definition as other data collection requirements, e.g. Sitreps
Data definitions and standards should be UK wide to facilitate the electronic sharing of data for patient care
To have real confidence in Performance Indicators, data collection and interpretation must be uniform across the NHS. Without precise definitions of the indicators it is difficult to determine which should not be included. Similarly without baseline definitions there is the danger of different starting points ie we cannot compare like with like
Until this is fully addressed the process will suffer distrust and will not gain buy-in from clinicians. For example, in surgical mortality rates: the current definition of surgery is inadequate and includes a number of procedures which are clearly not surgical; also, mortality following surgery should be in total not just in hospital
Problem with different starting points - targets should set against relative performance rather than absolute targets - implies agreement / standard definitions - urgent problem
Need to ensure that like is compared with like and in that sense we support adoption of standard measures and ways of measuring via PIs.
Clinical involvement in all aspects of PIs is crucial. It is often stated that it is difficult to involve clinicians at local level in national indicators. If the Department continues to use raw data without casemix adjustment then it will continue to be a difficult case. The development and definition of indicators process was perceived as a weakness within the consultation document
Question 7. Should we construct indicators from self-assessment/peer-assessment exercises? (YES / NO)
Support for this was lukewarm and unconvincing, with only 55% of respondants in favour of constructing indicators from self-assessment / peer assessment exercises
Question 8. Are there any indicators which are included that shouldn’t be?
Information about redundant indicators was fed into the September trilateral meeting process. A summary of their deliberations can be found in Section 2 of this document.
Question 9. How should indicator sets for Health Authorities, NHS Trusts and Primary Care Trusts develop in order to reflect the different roles of these organisations within the local health community?
By far and away, the most common suggestion was the ‘need to reflect different foci of these organisations and their relative ability to influence / determine what happens’.
Other suggestions included:
‘Shifting the Balance of Power’ is a major current issue. Most existing indicators will be unhelpful at the new HA level and initially unavailable / unreliable at PCT level. Therefore, initially the service will require aggregate indicators for the various groupings of PCTs.
Subsequently while a set of indicators is required for each component organisation, a holistic or aggregated view is also required in the form of a further set of indicators to monitor where and how they join up.
In both scenarios however aggregation requires sound definitions of indicators in order to prevent one components data / performance skewing the performance of the health community as a whole.
|
Work is ongoing to develop indicator sets for PCTs, at the same time as improving the acute Trust set. These will take time to fully develop, and there will be a process of improvement, including the annual consultation round. |
Question 17. How can we improve data quality?
The most popular suggestion was for the provision of ‘Additional Resources – capital and training’
Other suggestions included:
It was generally acknowledged that data quality is a major issue that undermines the effectiveness of the Performance Indicator development process through poor data, inaccurate results and lack of staff commitment. Many felt that this must be addressed sooner rather than later
Until the current issues of information overload and data quality have been resolved, the NHS should not be creating more indicators
It was openly acknowledged that such an exercise would require a huge amount of extra work for clinical and non-clinical staff alike.
Data definitions were seen as an important issue. until this is addressed, clinician buy-in would not be encouraged.
… need to improve definitional issues around clinical indicators to have any credibility with clinicians e.g. surgical mortality rates: current definition of surgery is inadequate and includes a number of procedures which are clearly not surgical; mortality following surgery should be in total not just in hospital
The data quality issue is closely enmeshed with the issues of the large number of indicators and poor / inappropriate definition.
There are too many indicators in this consultation. The industry required to generate all of these in a timely accurate way will inevitably detract from the business of providing accurate information to ensure good quality performance
There is thus a need for an indicator set that is more manageable, robustly constructed, clearly defined, understandable, relevant to what organisations are trying to achieve and capable of being validated locally.
If performance targets centred on quality or even called quality markers or indicators then this would encourage more clinician enthusiasm
There are too many Performance Indicators, rendering the process overbearing and thus meaningless
There needs to be a rigorous set of criteria applied to each indicator including "collectability and costs".
It was suggested that the introduction of proposed indicators should be phased to blend with the planned introduction of new systems to collect the data. Ideally the indicators should only be published once the data has been proved to be of reasonable quality and sufficient history is available to show trends.
One respondent queried how many Trusts have named a board member responsible for data quality, while some suggested that there should be PIs for data quality. An interesting suggestion was that source organisations should be made more accountable for the quality of their data, through the extension and mandation of data accreditation to performance information.
The sheer volume of indicators reflects many of the demands from the National Service Frameworks. Whilst collection of the data may be valid, the exercise demands considerable effort in compiling the data. The BMA has previously highlighted its concerns regarding the time and manpower involved in gathering this data without commensurate resources being made available
Question 18. How can the NHS data accreditation process be improved to support the accreditation of data quality of performance indicators?
The main suggestion was for ‘Additional resources to carry out the process correctly’
Other suggestions included:
The responses to the earlier questions identified a desire for greater resource to address data quality, definitions and the volume of indicators. Many responses commended data accreditation as a means of encouraging data quality.
To encourage a data quality environment it was suggested that data quality performance indicators should be put in place, that data accreditation mechanisms be extended and that there be greater accountability
some data quality indicators would be useful, to give PI users an indication of the quality of data that was used to calculate the indicator.
Need to address data quality issues through extension and mandation of data accreditation for performance information and by making source organisations more accountablefor the quality of their data
It was commented that data collection in the NHS is given too low a level of priority and that there should be a commitment to provide resources to improve the quality of data collection throughout. Part of the conclusion of the annual consultation process on PIs should be the publication of targets for improvement in data collection for the following year
A role for the NHSIA was identified
There are 469 indicators in the consultation. The trust can supply data for 311. They would be required to report on 211. However it is unlikely a trust can pay adequate attention to some 200 indicators. A subset relevant to a single service area would be more manageable. Most of the data items feeding the 469 indicators are not in the NHS data dictionary, others inadequately defined, not clearly understood or inconsistently recorded in different places … many data definitions hark back to the 1980s … NHSIA must review these
Question 19. How we can improve arrangements for validating data?
The most popular suggestion by far was for ‘Internal audit including checking at source prior to submission
Other suggestions included:
Making the data more relevant, pertinent and leading to tangible benefit and improvement to the service will encourage staff buy-in.
This questionnaire omits any mention of how poor NHS information currently is and how important the support of clinicians is to providing it. Trained coders are required at present the NHS relies on poorly paid largely untrained coding clerks
How the performance indicators are perceived by staff is important
If indicators are to be used to feed information back to the service which helps with benchmarking and identifying good practice so that services can improve then good. If just a means of traffic lighting then it will be hard to sell to the clinicians
It was suggested that if performance targets are centred on quality or even called quality markers or indicators then this would encourage more clinician enthusiasm
To motivate clinicians to input high quality data they need to see an output that is relevant and of interest to them. Ideally the clinician should be able to access his / her own data online and be able to compare and review performance against colleagues, as well as regional and national averages
Ultimately it was pointed out that the best way to achieve clinician commitment and thus improved quality data entry was through involvement of clinicians at the outset in the setting of priorities and performance indicators
Accuracy of data is a major problem. We are still not collecting adequate diagnostic outpatient data. Clinicians and managers may have different expectations of what to do with data. It is vital that clinicians are involved in setting priorities so they are based on clinical data and inappropriate targets do not drive the service
Clinical involvement in all aspects of PIs is crucial. It is often stated that it is difficult to involve clinicians at local level in national indicators. If the Department continues to use raw data without casemix adjustment then it will continue to be a difficult case. The development and definition of indicators process was perceived as a weakness within the consultation document
Question 20. How we can improve consistency of data definitions?
The most popular suggestion by far was to ‘Improve guidance - clear and timely definitions’
Other suggestions included:
The subject of clear and timely issue of definitional guidance has an associated resource issue
We have very real concerns about defining performance indicators and expecting these to be met without adequate resources.. We welcome the aim of this initiative but would urge the DoH to recognise its implications
Many responses can be summarised as ‘forewarned is forearmed’ in so far as indicators can be better exploited when the publication date is consistent, when there is a degree of involvement with users and when definitions are published in advance of data. As an example:
The Patient Survey is due to be carried out later this year. This is designed to collect data in relation to the 'patient experience' Performance Indicators. We are concerned to learn that the Guidance to be issued to the NHS Acute Trusts on this will not be available for general consultation - disappointing given that the NHS Plan is intended to reshape the service from a patients point of view
Given the number of PIs being implemented against the various NHS Plan targets, NSF milestones, HLMP target, SAFF monitoring, holistic health service provision etcetera, serious consideration needs to be given over the timely, and clear issue of guidance from the centre as to how they feel PIs can be best used
Clearly, common rules need to be developed and applied. Various offerings are available from Annex 6 of the Consultation Guide (which many respondents endorsed) through to HM Treasury / Audit Commission principles. Many respondents have offered guidance including
The key messages are:
Need for indicators that are robustly constructed, clearly defined, understandable, relevant to what organisations are trying to achieve and capable of being validated locally.
Need for more and better indicators around primary and community care. …
Need for a smaller, more focussed, high quality indicator set and a more user-friendly presentational format.
Need to improve definitional issues around clinical indicators to have any credibility with clinicians e.g. …
Need to address data quality issues through extension and mandation of data accreditation for performance information and by making source organisations more accountablefor the quality of their data.
Need for greater promotion of the PAF nationally as a means of improving local performance: alignment of PAF with modernisation agenda and NHS Plan targets; more consistent use nationally of the PAF for performance monitoring e.g. link to SaFF monitoring.
Question 21. Are there any DH datasets that we could use but currently don't?
The main datasets identified were:
In addition to the above, respondents suggested the use of complaints logs and the use of focus groups to run alongside the planned patient surveys and finally
Surveys and questionnaires are a good substitutes for routine collection across the NHS on a continuing basis and should be considered very strongly before starting new data collection systems
One non-NHS source identified
We believe that the single most valuable extension to the current proposed data collection and management system would be to link the HES with the ONS statistics on death
Question 22. Should we seek to use data from other organisations, eg. private and commercial companies, Royal Colleges, Audit Commission, to construct indicators?
81% felt that we should seek to use data from other organisations to construct indicators. Of these 37% identified the Royal Colleges as the best source, 24% the Audit Commission, 22% Private and Commercial sources and 12% Social Services
One commentator said
There should be one set of indicators for NHS and this should reflect the work of CHI, NICE, Audit Commission etc – this is not currently joined up
Another (non-NHS) commented that whilst the NHS may be the primary data source it may not be the best to analyse and understand the results
The NHS is in a strong position to create an unparalleled source of quality data about its healthcare system - it is not in a strong position to analyse and uncover all the lessons to be learnt from this data. This threatens to undermine the whole process. A very open process with the public, academia and non-NHS organisations should be considered. Failure to do so limits the debate about measurement of quality in healthcare
While the majority of responses took their lead from the question by ‘voting’ for the Royal Colleges, Audit Commission and Private & Commercial sources, it is interesting that many also identified Social Services reflecting perhaps, the movement towards monitoring a holistic view of health services provision
We are supportive of the process but point out that benchmarks must take into account social indices and deprivation … the Audit Commissions PAF family of comparators for Social Services indicators may help in identifying a common approach. They have an integrated services approach … development of indicators evaluating benefits of integrated working is to be welcomed. They see the integration of Social Services and NHS indicators as being important
|
The Department is in discussion with the royal colleges onr what would be appropriate |
Question 23. What better data can / should we collect?
The main data type identified was ‘Outcomes Data’
Other types of data identified included:
It was recognised that the latest consultation exercise reflected the greater interest in moving away from process to outcome based data. This was seen to reflect the development of specific service reviews via the NSFs and the consequences of specific reports like Bristol. However based on the comments received there is clearly still further to go
Some real clinically accepted outcome measurement based on qualitative measurement rather than quantitative numbers have to evolve. This will start to tell things like which surgeons are operating most effectively
Other observers were more cynical in believing this thrust to be more one of convenience
The PIs appear to be a mix of clinical outcome and efficiency indicators. The latter is a judgement. Quicker hospital throughput does not necessarily equate with better care. Perhaps you could separate this project into indicators for the benefit of better informed patients and indicators of value for tax payers money
One observer summarised the collective response thus
Indicators should be directed at (1) clinical outcomes either directly or through proxy measures, (2) quality of care and (3) quality of patient experience. Much of the data currently collected (eg PAS) does not necessarily help. We have the strong feeling that the document includes all currently collected data without thought as to whether it actually helps
Question 24. What is the impact of different data collections?
The 3 main impacts identified were:
Other impacts identified included:
Many of the responses cited earlier have addressed this issue. Duplication is viewed as being wasteful of relatively scarce and precious NHS resources. Duplication often results in skewed data requiring additional resource to investigate and rectify and only serves to discredit the whole initiative
Differences in recoding activity against standards within and between trusts mean results have to be interpreted with care and intelligence. These have often resulted in bad publicity. Significant resources have had to be invested to monitor published results only to find adverse trends often the result of misinterpretation of national data standards. Only when data standards are revised, training provided and submissions validated for quality and completeness will the benefits of benchmarking be realised
Question 25. What data collections are no longer required for performance management purposes? Identify for each why they are no longer required.
Information about redundant collections was fed into the September trilateral meeting process. A summary of their deliberations can be found in Section 3.
Question 26. How can we make data dissemination more useful for the public and the NHS?
The main suggestion was to ‘Make more easily available to the public (eg use the internet, display in waiting rooms)’
Other suggestions included:
The most favoured approach suggested was the utilisation of internet technology, for example an easier to use front end section to the NHS web site. However it should be remembered that not all members of the public have access to the internet. There should also be a requirement that the PIs are published in a user friendly way through the local media, in libraries, local authority premises and voluntary offices.
There was near-universal, serious concern at how the data would be used and reported on
Performance data can be used to improve organisations. However we are concerned about its linkage in the NHS to the proposed traffic light system and with it presumably the use of league tables
There are real resource implications to using HLPIs in any meaningful way (people, time and money). Indicators need to be clinically useful. How they are used is arguably more important than what is measured and they are reported on. The name and shame approach which seems to be the current predominant approach will not engender local ownership of the HLPIs.
One respondent suggested that a proper planned and co-ordinated release of information would be necessary after observing
Also constructive engagement in performance is essential. In 2000 it bordered on a cloak and dagger exercise with passwords being issued at midnight and host organisations being given 24 hours to produce a press release which in many instances was to manage indicators that had a low or poor performance ranking
|
Work is underway to improve the public presentation of the indicators, and include more information on the nhs.uk website. |
Question 27. How can we present the data so that it is easier to understand and use?
The most popular idea suggested was to ‘Use graphs and diagrams to simplify presentation format etc’
Other ideas included:
The use of lists and league tables was unpopular..
Much of this question has already been addressed by responses to the previous question. However given the comments made earlier about data quality and inconsistencies, until these issues are addressed any information dissemination exercise must be conducted with great care and caution.
Question 28. How could national benchmarking toolkits be developed to support greater use of performance indicators
The main suggestion was to ‘Consult with stakeholders, and CHKS’
Other suggestions included:
It was suggested that the best way would be to involve those most likely to be involved in collection and those most likely to use them. However it is also quite possible that these are the people who have least time to make available.
Many responses recommended that the NHS either work more closely with professional groups like the Royal Colleges or more preferably learn from such private organisations as CHKS. Many NHS organisations are putting a great deal of money into such companies as comparable services do not exist centrally.
The new national benchmarking service is a step in the right direction, but this needs to be tackled with a lot more energy as there is definitely a gap in the market
Many agreed that a system needs to be established where outlier performance can be reviewed, to ascertain whether it is genuine or is the result of poor quality data. Toolkits to do this would be invaluable
Others commented that they were encouraged by the potential of the Performance Analysis Toolkit, especially in the provision of comparative information at specialty level. They would be equally happy to receive the raw data rather than pre-designed graphs and charts, especially if this would increase the speed of delivery
.., and would also welcome the inclusion of denominator and contextual data to facilitate the identification and build of peer groups
A further comment on the Performance Analysis Toolkit
However it could be developed to incorporate more indicators. Ultimately we would look to see one database which covered all health and social care PIs alongside other indicators which are currently pulled together in the KIGS package
|
The Department is continuing to work on improvements to benchmarking toolkits, including improving the quality and range of indicators included in the benchmarking indicator set, and developing more functionality within the software toolkits. It is hoped this will enable NHS organisations to engage in more sophisticated performance analysis and benchmarking. |
|
SECTION 2: OTHER COMMENTS MADE |
This section picks up on other comments received that have not been addressed in detail in earlier sections of this document
The most significant area of comment concerned the fact that there appeared to be a number of areas not addressed by the Consultation Document. To some this came as a surprise
‘This document is startling not so much for its content, but for what is omitted . The aim is to monitor the performance of a highly complex organisation known as the National Health Service and presumably whilst such monitoring cannot be totally comprehensive it should be representative of client groups served by the NHS. It is therefore of concern that there is little mention of …’
The areas detailed include the following:
Following on from this was a comment that while the indicator set is mainly NHS and outcome based, it does not reflect the impact of government health inequality initiatives such as smoking cessation and health education in schools, which have an impact on the nation’s health
The last point concerns confusion about whether the additional indicators in the consultation document are new ones alongside the existing set or intended to replace others.
|
. SECTION THREE: POLICY AREA SPECIFIC COMMENTS |
CHD
Responses for consultation provided a valuable range of suggestions on refining the current indicators. Many of these are promising and will be developed as the Central Coronary Audit Database comes on stream, enabling routine and reliable collection of data on interventions and outcomes. There was an encouragingly large number of suggestions for indicators on heart failure, rehabilitation and on patient experience. The Department agrees that these are important areas for performance measurement in the future. The Department of Health is currently developing policy and standards for heart failure and rehabilitation and this work will include the development of performance indicators. Similar work is needed to develop meaningful indicators on patient experience.
Cancer
Over 300 consultation responses were received and have been taken fully into account in agreeing the third set of PIs. The comments will also help structure and define new performance indicators to be consulted upon over the winter. Performance indicators not included within headline indicators can be monitored at a local level.
Comments received recognised the limitations of using registry data to measure survival and death rates from cancer but overall were broadly in favour of their retention for some cancers.
Consultation strongly supported the inclusion of breast cancer and cervical screening Pis. Because there is strong evidence of cervical screening developing in those women who have never been screened, the percentage of women aged 20-64 who have never had a cervical screen will also be monitored. Cancer waiting time performance indicators will help monitor performance against cancer waiting times targets set out in the NHS Cancer Plan. Comments noted about the need for clear definitions.
There were clinical and definitional concerns about the inclusion of cost per drug regimen for breast cancer and type of course of radiotherapy and proportion of patients having their care reviewed by a specialist. Additional draft performance indicators will be drawn up for consultation over the winter.
Mental Health
The current paucity of robust mental health data leaves us little option but to continue using the existing three "high level" indicators. Additional "benchmarking" indicators are now included from which "headline" indicators may emerge as data quality improves e.g. the indicators on self-harm, and suicides by people under care of mental health services. Also, definitions covering expenditure on mental health have been improved recently, which may lead to better indicators around "value for money" being developed in future.
Diabetes
The Diabetes NSF will be published in two parts. The Diabetes NSF: Standards will be published this year and will include a consultation a set of performance indicators across the six PAF domains. The Diabetes NSF: Delivery Strategy, which will be published in summer 2002, will include the final set of performance indicators. The comments received on the PI Consultation will inform the development of the final performance indicators.
Older People
A strong message received as part of the consultation was that some of the performance indicators being currently used could be providing perverse incentives that could lead to older people not receiving fair access to services. The Department is carefully considering this point as part of the process of defining the next headline set of PAF indicators, supporting benchmarking indicators, and future performance indicator development. The consultation process has been enormously helpful and all comments received will be looked at in detail with a view to developing future performance indicators in the best interests of older people and their carers.
Health Inequalities and Public Health
Responses to the consultation on inequalities indicators were generally in favour of the indicators suggested. Only a small minority of respondents wanted to ditch specific indicators. There was support for the national inequality targets (life expectancy and infant mortality) being included in the PIs. Some respondents wanted lower level data eg down to ward level - this may be addressed by the neighbourhood statistics initiative with NRU and ONS. A number of specific detailed queries eg age ranges or definitions could be followed up in separate correspondence if necessary. The health inequalities consultation exercise covered much of the same ground, and common issues will also be addressed as part of the response to this consultation.
Smoking
The responses to the draft PIs on smoking – 7001 to 7007 – was encouraging, in that overall they reflected our assessment of the importance of indicators in this area and of the constraints and difficulties associated with some of them. A number of respondents made helpful suggestions about how these might be addressed. Some rightly pointed out that some areas are not purely the province of the NHS, and that central Government has an important role to play.
Many comments centred around data collection problems for some of the draft indicators, notably those concerned with information about smoking and/or quitting among particular groups – manual socio-economic groups, pregnant women, children. We agree with these concerns and propose that further work is needed on these indicators, with the exception of that on children smoking, which we reluctantly propose to drop on account of the lack of reliable data at local level.
The proposed indicator on the prescribing of Nicotine Replacement Therapy (NRT) attracted considerable comment. It is clearly right to cover bupropion also, as many respondents pointed out. Many thought that taken on its own, this indicator would not provide a complete picture of activity; partly because there are alternative methods to supply NRT, and partly on account of the desirability of linking prescribing information with the provision of advice and support to clients. We recognise the validity of these points but feel that the indicator nevertheless has potential value for local use, given the relevance of easy access to smoking cessation aids to the achievement of key targets.
There was a general consensus among those who made comments in this area of the importance of the indicator of smokers successfully quitting through the NHS services, supported by the indicator of numbers who set a quit date. We agree that these indicators may need to be expressed as a rate according to population in order to be more meaningful. It would not be feasible at present to propose an indicator for longer-term success, as some respondents wanted, as such information is not currently collected in all services; this might be a desirable objective for the future.
Teenage Pregnancy
Around 15 to 20 responses were received on each of the indicators in this area. In each case, where a view was expressed, a clear majority of respondents were in favour of including the indicator within the published set.
In relation to the proposed indicators on conception rates for under 18s and under 16s, all respondents who expressed a view felt that these should be included in the published set. All of those who expressed a view felt that the under 18 rate should be a headline indicator and all but one said the same for the under 16 rate. A small number of respondents felt that it would be helpful to distinguish between planned and unplanned pregnancies. In practice, however, this information would be very difficult to collect and the fact remains that overall socio-economic, educational and health outcomes are significantly poorer for teenage mothers as a whole.
Most respondents felt that the remaining indicators fell into the benchmark category, and we agree with this. We have also made detailed changes to the wording of all of these indicators in response to comments made in order to incorporate clearer definitions and to reflect the practicability of collecting data. We have removed one indicator concerning registrations at community clinics by ethnic group, because of the difficulty of collecting information at present. Two indicators (8007 and 8008) will be joint health and social care "interface indicators".
NHS Estates Management
The outcome of this years consultation exercise was generally a success and worthwhile. It has resulted in headline indicators which reflect more accurately the service priorities for the NHS Estate Management function. Key messages emanating from the consultation exercise were:
There is now a clear recognition that the views of the citizens should be sought and considered in order to identify the performance standards to be achieved in respect of the patients experience and their healthcare environment. It is acknowledged that the headline indicators and the benchmark indicators must represent both the citizens’ perspective and the requirement that NHS service providers ensure the quality and functional suitability of their healthcare property and services are compliant with NHS standards.
Childhood
Most comments on the proposed children’s indicators were made in relation to 10009 – 10012 (admission rates for three diagnoses of respiratory disease and for gastroenteritis).
The point was well made that having separate indicators for bronchiolitis, bronchopneumonia and pneumonia was likely to lead to problems with data quality, and the rationale for keeping them separate was unclear. We do intend that these indicators should be combined into a composite indicator for lower respiratory infections, something that was not clear in the consultation document.
We had some concerns that the indicators for emergency admissions for lower respiratory disease and for gastroenteritis were not good indicators of the quality of health services, given that incidence is known to be related to socio-economic factors, access to primary care, and referral practices. These indicators have been proposed because of the links to socio-economic status and access to health services. Trusts acting alone would be unlikely to make a significant impact on these indicators, but local health services (including primary care), working together with other agencies should be addressing the needs of groups that have difficulty accessing health services and that have poorer health outcomes.
Maternity
We are developing a National Service Framework for Children and Maternity Services and as part of the process we will be reviewing our indicator and data collection needs. We will also be considering how best we can provide women with the information they need to make more informed choices about the maternity care they want.
Dentistry
It was felt that indicator 12003, Average Cost per Appointment with a Community Dental Officer, should be dropped. There was little professional support for this indicator, community dentistry is in the process of evolving and what constitutes an appointment varies.
Indicator 12002 relates to the PM’s dental access pledge. Hence the need to include it in some form. However, regional variations make it very difficult to compare across different areas. For this reason it was thought best to include this only as a benchmarking indicator at this stage.
The use of the percentage improvement in decayed missing or filled teeth in five year was supported, but a number of commentators pointed out that, with an increasingly large proportion of children with no DMFT, there needed to be a benchmarking (supporting) indicator to show the distribution of decay in the affected children.
Sexual Health/Communicable Diseases
The consultation did not show support for indicator 15001 and this will not be taken forward. Consultation on the recently launched sexual health and HIV strategy is underway until 21st December and decisions on performance indicators for this area will be made in 2002 when we have had an opportunity to consider consultation responses.
The antenatal HIV testing and hepatitis B vaccine policies are outside of the strategy and can be taken forward separately.
Pharmacy
The intention had been to highlight areas within the Pharmacy Programme, Pharmacy in the Future, where specific performance indicators might be produced to monitor progress and seek input to their development. It is recognised that this was not clearly stated within the consultation. Nevertheless, some useful feedback was provided, which will inform the development of performance indicators, in particular for medication review of older people, hospital dispensing for discharge, clinical governance in community pharmacy and out-of-hours access to medicines.
Workforce
The 2001 consultation responses demonstrate broad support for a wide range of workforce performance indicators. These ranged from the strategic - accreditation to the Improving Working Lives Standard – to more specific, individual indicators of performance such as sickness absence rates for NHS staff. Several respondents raised definitional issues in relation to specific workforce indicators, along with the need to ensure that indicators were linked appropriately and data collection streamlined. These issues will be addressed through the Human Resources Performance Framework.
NICE Appraisals
The key message from the consultation was that all of the topics were sufficiently and equally important to be kept as either headline or benchmarking indicators. However, none of the indicators were sufficiently well developed to be included in the third set. The Department is now working to build on the work already undertaken and will bring back a more refined set of indicators for consultation as part of the potential fourth set.
Controls Assurance
There are still issues to be resolved around the self-assessment against the standards/indicators by organisations. Whilst production of league tables of performance against the standard is some incentive for organisations to unjustifiably increase their overall scores, the funding of functions based on the organisations assessed need should bring balance to the process. With the maturity of the process, including greater staff involvement and the improvements to the validation of results by an external audit body (incl, internal audit, clinical audit, CHI, NHSLA and others) we expect to see more realistic ‘scoring’ over time.
The standards themselves comprise of several criteria which they themselves could be used as individual Indicators – this would, however, increase the size of the Controls Assurance Indicators from the current 21 PIs to circa 340. As the Controls Assurance Standards endeavour to bring together various legislation and Department of Health guidance, there would also be scope for duplication with other Performance Indicators.
It is proposed the overall (weighted) score for each Controls Assurance Standard be retained within the Indicator set with the whole criteria/subsets used as Benchmarking Indicators. If the Standards are to continue to be included within future Performance Indicator sets, there is a need for further development of the Controls Assurance Standards in their use as indicators.
Information Management & Technology
Many respondents stressed the need for tight and consistent definitions to make both headline and benchmarking PIs strictly comparable across the Service. The Consultation has highlighted the need for both rapid and continuing work to ensure that any new or revised information requirements are developed in accordance with NHS data and dataset standards. There was broad support for measuring the level of access to, and usage of, improved electronic communications that underpin both the availability of benchmarking material and the provision of services for patients.
Prescribing
A working group, the Prescribing Indicators National Group (PING) has been established to consider the development of prescribing indicators. Responses to Consultation will shape agenda of PING.
In particular, the HRT indicators received a number of comments about their suitability, and will therefore be referred to this expert Group to assess their value.
In addition, all numerators will be changed from DDDs to ADQs.
Indicators on combination and slow release products will be split (into 6 indicators) and made available within the prescribing toolkit, since they are meaningful at practice level.
The Consultation responses to indicators 21017 – 21031 were very encouraging; but there is no method of collection at present from GP systems. These will also be referred to PING
Efficiency
The responses to the consultation process broadly supported the use of most of the indicators for use in the benchmarking set. There was less support for measures to be included in the headline set of indicators. Indicators should not be included in the headline set if the data was not sufficiently robust, or if there was a potential perverse incentive. For example, an ever lower length of stay may harm quality, and may not be a good indicator of efficiency. The indicators with the most support for inclusion in the headline set were: proportion of prescribing that is generic, financial balance, and theatre utilisation.
Very few of the responders thought that any of the indicators should be ‘ditched’. There was, however, widespread feeling that more work should be done to improve the quality of the existing set of indicators. In particular, many of the indicators would be significantly improved if they were able to adjust for casemix complexity. Ideally, the data would also become more sophisticated in other ways, such as being able to measure the ‘return to theatre rate’ in a meaningful way.
Hospital Acquired Infection and Antimicrobial Resistance
A range of views on the validity of developing performance indicators on hospital acquired infection and antimicrobial resistance were received. Respondents felt that the proposed indicators were applicable to acute Trusts. It was agreed that current information systems would not allow for capture of the data and that collecting this data would have resource implications. The importance of standardisation was emphasised.
Data is currently being collected on MRSA bacteraemias, and this could be included in the July 2002 Performance Indicator set. Indicator 23005 – Trusts should have an antimicrobial policy in place which is audited annually was thought to be adequately addressed through the Controls Assurance infection control standard. The remaining indicators in the main were thought to be credible as either headline or benchmarking indicators, although further development is required.
Immunisation
Very few comments were received on the immunisation PIs.
Primary Care
The key message for primary care from the consultation exercise is that Performance Indicators needs to be free of perverse incentives and constructed from robust, accurate data. The Department still does not collect on a national basis the sort of data needed to produce many of the proposed indicators. The data we have is sourced from the GP pay system and the GMS census and as such reflects activity and volume rather than measurements of quality. PCTs will be better placed to collect data on quality and delivery as part of their performance management and commissioning role [eg peformance management of NSFs] and it is likely that many of the proposed indicators, if not used in the PAF, will be available locally for benchmarking activity.
Accident & Emergency Services
The key messages from the Performance Indicators Consultation on Accident and Emergency (A&E) are waiting times in A&E and safety/security issues for patients and staff. Other key messages are the protocols for the management of people with heart conditions and rates of death (with a heart attack) in hospital within 30 days of emergency admission for patients’ aged 35-74.
Ambulance Services
The Department’s Winter and Emergency Services Team is currently co-ordinating work to develop a range of performance indicators across the six fields in the performance Assessment Framework (PAF) for NHS Trusts.
There is clearly a great deal of interest in this area. NHS ambulance Trusts, commissioners, and Department of Health officials are already involved in, and contributing to this work. The aim is to have identified and defined (in outline) some NHS ambulance Trust indicators to be included in the next performance indicator consultation in early 2002.