Index | Contacts | HA PIs | Trust PIs

NHS Performance Indicators : February 2002

 Department of Health logo

Annex A – How to Interpret the Graphs

 

Current Performance

Graphs representing current performance have one of two main presentational styles.

In the first style, as for HA indicator 1(i) Life Expectancy (Male) or Trust indicator B(i) Day case rate, the data values are displayed in a bar chart. The organisations are listed down the left hand side of the graph ordered by indicator value, with the values presented in the form of a bar. The scale is shown along the bottom of the graph, and the indicator value can be read directly from this scale.

The unit of the scale can vary. In some instances it is a percentage or a rate per head of population, in other instances it is a number of occurrences.

The graphs are colour-coded to show good through to poor performance, relative to other organisations in the graph. The better performers on the indicator are represented by dark blue bars, with various shades through to pale blue to represent poorer performers.

In the second style, as for HA indicator 3(iii) Returning home following treatment for stroke or Trust indicator A(i) Returning home following treatment for stroke, the data values are displayed with their associated 95% confidence intervals. The organisation name is again listed down the left hand side of the graph, but this time the indicator values are presented in the form of a dot with the 95% confidence interval around the dot represented as a line. Again, the scale is shown along the bottom, and the indicator value can be read directly from this scale.

Confidence intervals give an idea of the level of uncertainty caused by chance occurrences in the indicator values. A 95% confidence interval gives the range in which the true indicator value would be expected to fall 95 times out of 100. In interpreting these types of graph, it should be noted that if the confidence interval of an organisation does not overlap with the associated national confidence interval, it is highly likely that the organisation’s performance is genuinely different from the national rate.

The dot representing the indicator value is colour coded to show whether or not the HA or Trust appears to be different from the national average. If the organisation has a significantly poorer performance than the national average, the dot is coloured red. If their performance is significantly better than the national average, the dot is coloured green. Otherwise the dot is coloured orange, indicating that there is no strong evidence that the HA or Trust is different from the national average.

It is possible for a Health Authority or Trust to have a different shade of dot from organisations that immediately surround them on the graph. This difference is due to the width of the confidence intervals, which are related to the size and characteristics of the individual organisations.

For example, in Graph A(vi) of the Trust indicator set, the dot representing Calderdale Trust is shaded orange. This shading indicates that the Trust value is not statistically significantly different to the national average, because its confidence interval overlaps the national confidence interval. However, the confidence intervals for Trusts above and below Calderdale Trust do not overlap the national confidence interval. The dots for these Trusts are therefore shaded green to indicate that their mortality rates are statistically significantly lower than the national average.

There is a similar occurrence in the Health Authority indicator set on Graph 6(x). In this case, Herefordshire’s confidence interval overlaps the national confidence interval so its dot is coloured orange, whereas the confidence intervals of surrounding Health Authorities do not overlap the national confidence interval and their dots are therefore coloured green.

 

Performance Improvement

Graphs representing improvement values for each indicator are presented in bar charts, and show the change in performance within the HA or Trust itself.

For indicators not presented with confidence intervals in the graphs showing current performance, the improvement graphs should be interpreted in exactly the same way as described for indicator 1(i) or B(i). The improvement is calculated as the proportional change from the previous to the current values for each organisation. As for the current performance graphs, the improvement graphs are colour-coded to show good through to poor performance, relative to other organisations in the graph. A good performance is represented by pale blue shading, with various shades through to dark blue to represent poor performance.

For indicators presented with confidence intervals in the graphs showing current performance, the improvement is also calculated as the proportional change from the previous time period to the current time period. The graphs are again colour coded. Green indicates that there has been an improvement in performance and red a deterioration. In addition, each organisation is allocated to one of six bands, based on the confidence intervals for the percentage change. Thus, the band labelled "A" indicates that the improvement in performance is statistically significant, and it is highly likely that this improvement did not arise by chance. Similarly, "F" indicates a significant deterioration, ie it is highly likely that the deterioration is not due to chance. For the other bands, although there may have been an improvement or deterioration, any interpretation must be treated with some caution.

The six bands should be interpreted as follows:

A

Significant improvement: Improvement in performance is statistically significant at a 5% level (ie there is less than a 1 in 20 chance that the change in values arose by chance).

B

Moderate improvement: Improvement in performance is statistically significant at a 10% level (ie there is less than a 1 in 10 chance that the change in values arose by chance).

C

Some improvement: It is possible that there is a real improvement in performance, but the evidence available is not statistically significant.

D

Some deterioration: It is possible that there is a real deterioration in performance, but the evidence available is not statistically significant..

E

Moderate deterioration: Deterioration in performance is statistically significant at a 10% level (ie there is less than a 1 in 10 chance that the change in values arose by chance).

F

Significant deterioration: Deterioration in performance is statistically significant at a 5% level (ie there is less than a 1 in 20 chance that the change in values arose by chance).

A more detailed description of the improvement calculations and how the banding thresholds were determined are available on the Department of Health website (www.doh.gov.uk/nhsperformanceindicators/2002).

 

General Information

Various calculations have been undertaken to facilitate 'like' with 'like' comparisons, these are summarised below.

Local populations vary in their age structures. For instance, some towns have more older people resident than other areas. This variation could result in higher rates of emergency readmission to hospitals in these towns, even if there were no differences in the quality of care. To avoid comparisons being distorted, the indicators have been adjusted for differences in the age and gender structure. The detailed description of the method used to calculate age and gender standardised rates is available on the website (under Technical Issues).

There are differences in the social and economic mix of local populations, which will be reflected in the numbers and types of patients admitted to hospital. Social and economic factors may influence both health and the results of health care, irrespective of the quality of care. However, where there is evidence that a service is effective or that a certain standard should be met, then it should be applied equally to all patients irrespective of their socio-economic status. Social and economic influences may explain but do not necessarily justify variation. Adjustment of the indicators for socio-economic variation may mask the very information on inequalities that the NHS should know about and address. Therefore no attempt has been made to adjust the indicators for social and economic characteristics.

There are different types of hospitals ranging from small community hospitals to large teaching hospitals. Consequently their patients differ in terms of the severity of their conditions (case-mix). Such variations will affect treatment outcomes irrespective of the quality of care. Therefore comparisons between hospitals of the same type are more appropriate than comparisons between different types. In order to facilitate 'like' with 'like' comparisons, NHS Trusts have been grouped into similar types, and graphically displayed by these groups. Annex B lists all acute NHS Trusts and type of hospital cluster to which they are allocated.

It is likely that there will be less variation in case-mix between resident populations of Health Authorities than between smaller hospital patient populations, therefore Health Authorities are not grouped into clusters.

Many of the indicators are based on Hospital Episode Statistics (HES) data, supplied to the Department of Health by the NHS. This data set is not always complete and missing information impairs the reliability of comparisons. Therefore data quality analyses were undertaken, which are described in more detail on the website. On the basis of these analyses, every Health Authority and every NHS hospital Trust was given a data quality mark for each year. The graphs in this publication only include organisations which met certain standard criteria in terms of data completeness and validity of coding. Those with "poor" data quality have been excluded from the graphs. Trusts may also be excluded from the graphs for other reasons, including having low levels of activity for the indicator which would give unrepresentative results.

In the process of compiling this document, a small number of Trusts found local data recording problems. The results should therefore be treated with some caution. However, generally the quality of data recording and data completeness has improved considerably since we started to publish Performance Indicators.

More detailed explanations of the graphs and the underlying methodological issues are available on the Department of Health website (www.doh.gov.uk/nhsperformanceindicators/2002).