Benchmarking (BM) is the process of comparing services. The aim of BM is to evaluate efficacy and efficiency, in the pursuit of excellence in standard practice.1 Its application in healthcare services has been limited to date, and only a few experiences in public health services and some hospital specialties have been published.2,3

In Spain, the first BM study in thoracic surgery departments was conducted in 2004, examining data from 2002 and 2003, with the participation of 9 units.4 In this first edition, proposals were made for improving the data recorded in discharge reports, avoiding unnecessary hospital admissions, and standardizing measures aimed at improving the quality of lung resections. The second BM study was conducted in 2008, with the participation of 13 units.5 A third procedure was undertaken in 2013, in which 17 units participated, the results of which are presented in this article.

BM results from the 17 Spanish TSUs were analyzed. Of a total of 1125011 hospital episodes, 8250 treated in the participating TSU were selected. After exclusion of episodes of MDC14 and MDC15 and patients <18 years of age, 7807 episodes remained.

Overall Results

The TSUs were responsible for between 0.6% and 1.6% of all discharges from their respective hospitals. The numbers ranged widely, from 168 to 794 (BM norm 432; EN 366). The mean weight lay between 2.39 and 5.02 (BM norm 3.2; EN 2.9) (Fig. 1). The relative weight with respect to the BM ranged between 0.7 and 1.6. The case-mix in each TSU followed the same trend. The distribution of DRGs 075. 077. 095 and 538 is shown in Fig. 2, which includes the calculation of the N ratio (DRG 538/(DRG 075+DRG 538).

Fig. 1.

Mean weight and complexity index of participating thoracic surgery units, compared with benchmarking and external standards.

(0.21MB).

Fig. 2.

Case-mix variability, showing the most common diagnosis-related groups.

(0.33MB).

ALOS varied widely among the TSUs, ranging from 4.1 to 10.7 (BM 7.5; EN 7.1). The RAALOS ranged between 0.5 and 1. The preoperative ALOS was between 0.2 and 1.7 (BM 0.9; EN 1). LOS index was between 0.2 and 4.5, also confirming high variability.

Mortality ranged between 0.2 and 2.5% (BM 1.1%; EN 1%). RAMI ranged from 0.1 to 1.2. The complication rate was 0.6–11.1% (BM 6.3%; EN 6.5%). RACI was between 0.2 and 1.6. Postoperative empyemas occurred at a rate of 0–1.5% (BM; EN 0.7%); bronchopleural fistula, 0–1.1% (BM 0.4%; EN 0.3%); nosocomial pneumonia, 0–3.2% (BM 1.3%; EN 1.1%), and operative wound dehiscence, 0–0.9% (BM 0.3%; EN 0.2%). Readmissions within 30 days of discharge ranged between 0% and 6.4% (BM 4.5%; EN 4.4%). RARI was 0–1.9.

Activity assessment is shown in Fig. 3. The number of beds per TSU ranges from 5 to 13 (BM 9), and the number of surgeons from 2 to 7 (median 4, BM 4). The number of beds per surgeon ranges from 1 to 6; most TSUs have between 1.5 and 3 (BM norm 3). The BM norm for units of hospital output per surgeon was 363, but most TSUs are below this figure. Occupation rates ranged between 33% and 151% (EN 86%). The ratio of first outpatient visits ranged between 0.6 and 5.4 (BM 2.6).

Fig. 3.

Activity of participating thoracic surgery units.

(0.23MB).

Analysis of Surgical Procedures

The percentage of surgical activity among the total number of admissions ranged from 71% to 91% (BM 84%). The surgical ALOS ranged from 4.3 to 14 (BM 8.7; EN 9.1), and RAALOS from 0.5 to 1. The rate of surgical mortality ranged from 1.3% to 6.1% (BM 3.02; EN 2.7%) and RAMI from 0.5 to 1.8. The complication rate varied between 1% and 14% (BM and EN, 9.5%), and the RACI ranged from 0.3 to 1.6. The number of interventions varied widely (Fig. 3). Activity per surgeon also showed wide variability (34–158; BM 42).

Pulmonary Lobectomies

Half of the units (50%) performed between 50 and 70 lobectomies (Fig. 4) (BM 107; EN 79), with an ALOS of between 5.9 and 10.8 (BM 8.6; EN 8.8), and a RAALOS of between 0.6 and 1.2 (Fig. 5). Preoperative ALOS was 0 to 1.3 (BM 0.9; EN 1). The LOS index ranged from 0 to 2.8; most units scored more than 2. Mortality rates ranged from 0% to 5.2% (BM 2.2%; EN 1.9%), while RAMI was between 0 and 1.6. The rate of complications ranged from 0% to 20% (BM 9.3%; EN 12.1%), with a RACI of 0–2. Most TSUs scored between 1.3 and 1.5. The readmissions rate was between 0% and 8.6% (BM; EN 4.3%), with a RARI of 0–3.5.

Fig. 4.

Number of lobectomies performed in thoracic surgery units participating in the benchmarking.

(0.09MB).

Fig. 5.

Average length of stay and risk-adjusted length of stay index in thoracic surgery units, compared with benchmarking and external standards.

(0.23MB).

Other Interventions

Between 3 and 20 pneumonectomies/year were performed (BM 9; EN 8). Most TSUs performed 8–9/year, with an ALOS of between 6.6 and 32.5 (BM 11.3 and EN 12.8), RAALOS 0.7–2.1. Complications and mortality ranged from 0%–35% to 0%–25%, respectively.

A large number of video-assisted lobectomies were performed in some TSUs (with up to 68 in 1 unit, and more than 30 in another 4). Other TSUs performed fewer video-assisted resections (between 1 and 25/year). ALOS ranged between 3.7 and 12.6 (BM and EN 7.4); ALOS was 1.2 days shorter than for conventional lobectomies.

Atypical resections varied widely. In 4 TSUs, 10 or fewer atypical resections were performed, and another 7 TSUs performed between 20 and 33 of these procedures. Other TSUs performed interventions of this type on more than 40 occasions. ALOS ranged between 4.3 and 11.2 (BM 7.5; EN 7.9).

Pneumothorax also occurred at varying rates in the different TSUs, ranging from 7 to 79 (BM 38; EN 33), with an ALOS of 2.6–8 (BM 5.8; EN 6.2).

BM has increased in recent years.6–8 Despite the reported advantages of analyses of this type, no well-designed studies have demonstrated the clinical benefits of undertaking benchmarking.5 Indeed, the results of this edition of our BM study reveal that problems detected in previous analyses persist, as does the wide variability among TSUs. Specific corrective measures obviously need to be implemented and quality standards need to be created. This issue should probably be addressed by ad hoc committees already operating within scientific societies. Other authors have also suggested that the medical societies could initiate BM procedures, as was the case in this study, and could even create prospective databases.9,10

The most significant limitation of a BM study is that it is carried out without a full analysis of the processes. This flies in the face of some observations on the validity of using administrative databases in the evaluation of healthcare quality,11 although some groups defend this approach.12–14 Another problem is the origin of the data. The basic minimum data set has some limitations that were pointed out in our earlier publications.4,5 The creation of prospective databases with internal quality control, supported by the European Society of Thoracic Surgeons,15 may be of great importance for benchmarking in the future.

One important finding, also a feature of previous benchmarking studies, was the wide variability in the results obtained from the different TSUs. The number of discharges, the population profile, the distribution of discharges by DRG, and the rate of surgical activity all vary widely, and this affects the mean weight and complexity index of the participating TSUs. This has also been reflected in the percentage of DRG 075 in each of the TSUs, which ranged between 25% and 50%, and in the number of lobectomies performed (25–200). Complexity calculations and risk adjustments are therefore essential when comparing data from different units.16 Measuring severity, however, depends heavily on the quality of departmental discharge reports, which must detail all patient comorbidities and complications. The importance of this has been discussed at length in previous BM evaluations.4,5 The analysis of complications also revealed wide differences among the participating sites, possibly because these data were missing from discharge reports. Improvement and standardization of discharge reports is one of the main areas of improvement in some TSUs.

Some specific complications associated with thoracic surgery occurred at rates similar to internationally accepted standards, and mortality was in the same range.17 These aspects have varied little since previous BM studies, although some sites should perform a detailed analysis of their data and their impact on patient safety.

Emergency admissions generally impact negatively on clinical management and services. A significant number of admissions of this type occurred in the BM participants, ranging from 17% to 46%. This is another area for improvement in the future, and the introduction of patient admission protocols would probably be useful.

ALOS is a widely used hospital indicator,18 and varied greatly among our TSU participants, as can be seen in the RAALOS, a more objective data point. It ranged from 5.9 to 10.8 in lung resections, with a mean BM value of 8.6 and a LOS index of 0.6–1.2. Preoperative ALOS also varied significantly, from 0.3 to 1.6.

Lobectomy outcomes were good, in general, and RAALOS values were low. Preoperative ALOS, however, could be improved in the future. Mortality after lobectomy was low (2%), but morbidity varied widely, possibly due to differences in the quality of the discharge reports. Some departments reported a high rate of related emergency readmissions, also reflected in the widely ranging RARI, suggesting that the discharge procedures for this type of surgery need to be reviewed.

The use of video-assisted lobectomies varied widely among the participating groups. A few units carried out more than 30 procedures of this type every year, but others performed hardly any. Another point of interest is the moderate impact of the procedure on ALOS: 1.2 days less than for conventional lobectomy.

Clinical pathways and fast track procedures need to be implemented in pulmonary resection surgery.19 Benchmarking, along with other more conventional methods of auditing healthcare performance10 and patients’ perception of quality,20 is useful for the overall analysis of clinical activity. The analysis of activity, not included in our previous BM studies, provided widely variable data, as can been seen from the range of units of hospital output – 200–800. Activity needs to be analyzed in greater depth in future BM studies, and some areas, such as outpatient visits, merit particular attention.

We found considerable variability in quality, activity, and efficiency parameters among our participating TSUs.

We identified obvious areas for improvement: admission processes must be standardized to avoid urgent admissions and to shorten preoperative ALOS, hospital discharges must be streamlined, and discharge reports improved by including all procedures and complications.

Some thoracic surgery units have parameters which deviate excessively from the norm, and these sites need to review their processes in depth. Coding of diagnoses and comorbidities is another area which could be improved in some units.

The authors state that they have no conflict of interests.