Index | Contacts | HA PIs | Trust PIs
NHS Performance Indicators : February 2002

Annex D - Summary Report on NHS Performance Indicators: A Consultation, May 2001

1.ii It is important therefore, to measure and publish performance information:

(a) to enable continual service improvement supported and informed by the best possible management information
(b) to demonstrate accountability to the public and government for the spending of public resources.

1.iii In order to take views on the indicator development programme from the NHS and patients, the Government is committed to an annual consultation process. The next consultation is due to start later this year and will inform the indicator sets for 2003 and beyond.

1.iv The first consultation process started in May 2001 with the publication of a consultation document: NHS Performance Indicators: A Consultation. This was circulated extensively throughout the NHS and other interested organisations. Issue of the document was followed by a promotion process entailing a significant nation-wide series of seminars to raise awareness and knowledge about the consultation, discuss associated issues and encourage an informed response.

1.v The deadline for receipt of responses was 8th August 2001. The responses have now been analysed. Initial results have already been used to inform the new NHS Performance Indicator sets now being published. The results will also be used to inform the next consultation document, due to be published later this year.



2. RESPONSE

2.i The exercise sought to involve as wide an audience as possible. We received back:

(a) 200 completed general questionnaires
(b) 450 general responses
(c) Approximately 7,000 specific comments on the individual indicators

2.ii The responses contained differing levels of detail ranging from comments on specific indicators to matters of policy. With such a wide audience some comments contradicted those made by others.

 
3. Common Themes

3.i The overall concept of an ongoing consultation exercise was welcomed. At a time of significant and continual change it presented the opportunity to re-examine the current framework and fine-tune it so that it more closely mapped onto the current needs of the NHS.

3.ii The Performance Indicator development programme was generally endorsed by the respondents, however it is apparent from the range of comments that there is still work to be done to explain the purpose of the Performance Indicator development process.


SPECIFIC RESPONSES

4. Using the Performance Assessment Framework and Performance Indicators

4.i A significant number of respondents confirmed that they used the PAF for local planning and development. It was argued that the PAF should be better publicised, particularly at PCT level. It was suggested that the indicators should be more timely, published more frequently, and that we should avoid publishing out of date information.

4.ii A wide range of other sources of information were also used, including:

· activity levels
· waiting list/times information
· local indicators
· Service and Financial Frameworks
· National Service Framework targets
· reference costs
· patient satisfaction data

4.iii The PAF was seen as providing a common framework to support discussion of best practice amongst peers within the health community. It is useful for planning, and promotes the development and monitoring of local indicators. The PAF provides a consistent approach, and ensures reliable coverage of key issues within internal performance systems.

4.iv Over time there has been a distinct shift of emphasis from using NHS Performance Indicators as a proscriptive tool to regulate and manage the service, into something which facilitates performance improvement.

4.v There was a strong message that the NHS Performance Indicators should be linked more closely to performance targets (eg NHS Plan, Service and Financial Frameworks, and Local Modernisation Reviews) and core business of the NHS.

4.vi The data collected for performance indicators should, wherever possible have the same definitions as other data collection requirements, be easy to collect and be drawn from existing operational systems.

4.vii It was suggested that there should also be a core set of indicators that can be used to develop into a time series. Some responses suggested that there should be testing and consultation before making changes, or introducing new indicators and that this should involve those who provide and use the data, and the professional bodies.

4.viii The inclusion of age and sex standardisation was welcomed. Use of the indicators is particularly problematical in the area of health improvement, due to a multi-variable range of influences, not least a mobile population. There was thus a request for population/demographics data used in the calculations so that information could be replicated at more local levels to ensure fairness, consistency and comparability.

The Department's intention is to develop a set of indicators that more closely reflects NSFs and other priorities. However, this will take time, partly because of the length of time it takes to develop new indicators. In the meantime, we intend to improve the indicator sets through the annual consultation process. The next set of Performance Indicators and Performance "star" Ratings will be published together in summer 2002, as part of the same system, by the Department of Health in conjunction with the Commission for Health Improvement. From 2003 onwards the Commission will take over sole responsibility for independent publication of these performance results.

5. Data Quality

5.i A significant number of comments acknowledged that data quality can be a major issue that undermines indicator development and use. Many felt that this should be addressed sooner rather than later.

5.ii It was strongly argued that resources should be directed into training and capital (standardisation of software etc), that clinicians and managers should be encouraged to have ownership of their data, and that a cultural change was needed to recognise the importance of clinical coders and the work they do. The tri-partite letter on improving data quality (June 2001) reminded NHS Chief Executives and Medical Directors of the need to ensure that their data is accurate. A copy of the letter can be found at: www.doh.gov.uk/improvingdataquality. Emphasising the importance of clinical coders was recognised in the recommendations arising from the Kennedy Report on the Bristol Inquiry. During 2002, there will be a review of the professional structures for clinical coders. From 2003, there will be a programme of work to improve the professional standing and education of clinical coders.

5.iii There was also support for improvements to data definitions, to make them clearer, nationally consistent, and more robust. A new system for approving NHS Information standards has been introduced as part of the implementation of the NHS Information Strategy, Information for Health. All information standards which need to be approved at national level will be expected to be approved by the NHS Information Standards Board and its sub-boards for clinical information, management information and technical standards. More information on the NHS Information Standards Boards can be found at www.isb.nhs.uk.

5.iv To encourage a data quality environment it was suggested that a data quality indicator should be included in the NHS Performance Indicator set, that data accreditation mechanisms should be extended and that there should be greater accountability. The indicator set now being published includes a HES Data Quality Indicator.

5.v The NHS Information Authority has been commissioned to review and update the NHS Data Accreditation process to support the PAF requirements, and to develop a scheme which can be extended to non-acute trusts. More detail about the NHS Data Accreditation scheme can be obtained from www.nhsia.nhs.uk/dataquality/pages/accreditation/q_reports.asp.

5.vi A Data Quality Reporting System (DQRS) exists for data submitted to the NHS-wide Clearing Service (NWCS). Currently this is available to the Department (including Regional Offices), CHI and the Audit Commission. During 2001 an abridged version has been developed for dissemination through NHSNet (available to all NHS organisations). The Audit Commission checked on the use of the NHS Data Accreditation process as part of their independent light-touch data review in 2001.

 
6. Dissemination

6.i The consultation exercise recognised that performance information should be made more accessible to the public using the internet (with an easier to use front end), libraries, waiting rooms, local authority premises and voluntary offices.

6.ii The presentation should make more use of graphs and diagrams, and we should provide background information which avoids jargon.

6.iii We should make a distinction between information for members of the public and information for healthcare organisations, and provide the right amount of detail for each.

6.iv For the NHS the toolkits should include drill down to more detailed levels as appropriate and should provide fully searchable databases. We should consult stakeholders about what kind of toolkits would be most useful for them. We should spend time working with those who collect and use the information, and learn from private organisations providing similar services.

6.v Many agreed that a system should be established where outlier performance can be reviewed, to ascertain whether performance is a genuine outlier or is the result of poor quality data.

The Department of Health will shortly be producing a supporting set of benchmarking indicators, and is aiming to improve the ability of NHS organisations to analyse that information, through the implementation of improved software, with more functionality

7. DEPARTMENT OF HEALTH ACTION

7.i Future indicator sets will make greater use of material from patient surveys, starting with acute inpatients and expanding to cover PCTs, A&E and other parts of the health service. We also expect to see better reflection of staff issues and views in future indicator sets.

7.ii The Patient Survey programme is intended to provide a mechanism for making the NHS more patient-focused and to provide a quantifiable way of achieving this.

7.iii Two surveys have been undertaken, the GP Survey in 1998 and the Coronary Heart Disease Survey in 1999. We believe that results from these surveys should be reassessed as part of a coherent package of quality indicators addressing Primary Care and Coronary Heart Disease. In future years surveys like these will be repeated and will enable progress on the patient dimension to be measured. As results from future NSF-based surveys become available, such as the national survey of cancer patients, these will also be linked with quality indicators.

7.iv Results from the new trust-based surveys will be available for use in Performance Ratings in summer 2002 and could form a first step in measuring the patient experience of local health services as part of the overall performance assessment. This will demonstrate to both the NHS and the public that improving patient experience is seen to be an essential part of service delivery.

 
8. NEW DATA COLLECTIONS

8.i The consultation exercise recognised that for new indicators to be developed there would sometimes be a need for new data collections, but that this should be considered alongside reductions in data collections that did not have a clear purpose or use.

8.ii Consultation respondents were concerned about the capacity of the NHS to respond to too many new data collections and sought reassurance that the Department of Health would seek to minimise this burden. The Department's intention is to use existing data collections wherever possible and to only authorise new collections where there is a clear value to be added for the service. The Department's Review of Central Returns Committee will take into consideration the potential burden of additional data collections on the NHS if requested to authorise new collections.

 
9. OTHER ISSUES

9.i Other key developments which support the indicator programme to provide comparative information on quality of care to clinicians, managers, patients, the public, DH and organisations such as CHI are:

• the linkage of HES and ONS mortality data;
• the potential to enhance existing datasets;
• clinical quality programme.

9.ii Linkage of HES and ONS data: The Department has linked Hospital Episode Statistics (HES) and Office for National Statistics (ONS) mortality data files to provide indicators on true 30-day mortality following treatment in hospital irrespective of whether the patient died in hospital or died following discharge. Such indicators are considered to be more clinically relevant and a major step-change from our current set. For example this work will provide us with indicators to compare trust performance on mortality following treatment for hip fracture within a defined time period irrespective of length of stay.

9.iii Enhancing existing datasets: The Department is exploring whether additional data items can be collected via the HES dataset which would permit the calculation of risk-adjusted measures of outcome.

9.iv Secondly, the Department has commissioned work from the National Centre for Health Outcomes Development (NCHOD) to identify whether HES can be used for surveillance purposes. Pilot work on this project is underway covering Trauma & Orthopaedics. Specialty specific indicators are being developed in a joint exercise with clinicians.

9.v Clinical quality programme: The introduction of clinical governance in the NHS requires a change in culture so that clinical teams continuously review the quality of their services. Part of that cultural change is a new approach to information - whether accurate recording of the details of patient care in paper or electronic records, or the use of information drawn from patient records to supply clinical audit data. The Information for Health programme recognises the significance of this change and has included an education, training and development programme to support clinicians in the use of IM&T as systems come on stream.