Benchmarking entails continuous comparison of efficacy and quality among products and activities, with the primary objective of achieving excellence.
ObjectiveTo analyze the results of benchmarking performed in 2013 on clinical practices undertaken in 2012 in 17 Spanish thoracic surgery units.
MethodsStudy data were obtained from the basic minimum data set for hospitalization, registered in 2012. Data from hospital discharge reports were submitted by the participating groups, but staff from the corresponding departments did not intervene in data collection. Study cases all involved hospital discharges recorded in the participating sites. Episodes included were respiratory surgery (Major Diagnostic Category 04, Surgery), and those of the thoracic surgery unit. Cases were labeled using codes from the International Classification of Diseases, 9th revision, Clinical Modification. The refined diagnosis-related groups classification was used to evaluate differences in severity and complexity of cases.
ResultsGeneral parameters (number of cases, mean stay, complications, readmissions, mortality, and activity) varied widely among the participating groups. Specific interventions (lobectomy, pneumonectomy, atypical resections, and treatment of pneumothorax) also varied widely.
ConclusionsAs in previous editions, practices among participating groups varied considerably. Some areas for improvement emerge: admission processes need to be standardized to avoid urgent admissions and to improve pre-operative care; hospital discharges should be streamlined and discharge reports improved by including all procedures and complications. Some units have parameters which deviate excessively from the norm, and these sites need to review their processes in depth. Coding of diagnoses and comorbidities is another area where improvement is needed.
Benchmarking hace referencia a la comparación continuada de la eficiencia y la calidad entre productos y actividades con el objetivo fundamental de alcanzar la excelencia.
ObjetivoAnalizar los resultados del benchmarking realizado en 2013 con la actividad asistencial de Cirugía Torácica en el año 2012 en 17 servicios de Cirugía Torácica españoles participantes.
MétodosLa fuente de información para el estudio ha sido el conjunto mínimo básico de datos de hospitalización correspondiente al año 2012. Los datos han sido proporcionados por los centros participantes, a partir de los informes de alta hospitalaria, sin intervención de los responsables de los correspondientes servicios asistenciales. Los casos objeto del estudio han sido todas las altas de hospitalización registradas en los centros participantes. Los episodios incluidos han sido los de enfermedad quirúrgica respiratoria (CDM4-Q) y los del servicio de Cirugía Torácica. La identificación de estos casos se realizó usando los códigos de la novena edición de la Clasificación Internacional de Enfermedades, Modificación Clínica. Para valorar las diferencias en gravedad y complejidad de los casos se ha utilizado la clasificación de los grupos relacionados por el diagnóstico refinados.
ResultadosLos diversos parámetros generales estudiados (casuística, estancia media, complicaciones, readmisiones, mortalidad y actividad) han tenido una gran variabilidad entre los participantes. El análisis concreto de intervenciones (lobectomía, neumonectomía, resecciones atípicas y neumotórax), también han oscilado considerablemente.
ConclusionesSe observa, al igual que en ediciones previas, una considerable variabilidad entre los grupos participantes. Existen áreas de mejora evidentes: estandarización de los procesos de admisión, evitando ingresos urgentes y mejorando la estancia preoperatoria; agilización de las altas hospitalarias y mejora de los informes de alta, reflejando toda la actividad y las complicaciones habidas. Algunas unidades de Cirugía Torácica deben hacer una revisión profunda de sus procesos porque pueden tener algunos parámetros con una desviación excesiva de la norma. También deben mejorarse los procesos de codificación de diagnósticos y comorbilidades.
Benchmarking (BM) is the process of comparing services. The aim of BM is to evaluate efficacy and efficiency, in the pursuit of excellence in standard practice.1 Its application in healthcare services has been limited to date, and only a few experiences in public health services and some hospital specialties have been published.2,3
In Spain, the first BM study in thoracic surgery departments was conducted in 2004, examining data from 2002 and 2003, with the participation of 9 units.4 In this first edition, proposals were made for improving the data recorded in discharge reports, avoiding unnecessary hospital admissions, and standardizing measures aimed at improving the quality of lung resections. The second BM study was conducted in 2008, with the participation of 13 units.5 A third procedure was undertaken in 2013, in which 17 units participated, the results of which are presented in this article.
MethodsParticipating CentersA total of 17 thoracic surgery units (TSU), all in university hospitals, participated in this study (Annex 1). In this edition, the thirteen units previously involved in the second BM study were joined by a further 4 TSUs.
Data SourceInformation was obtained from the minimum basic data set (MBDS) for hospitalization in 2012, retrieved from discharge reports issued during that period. The data was processed, anonymously and independently, by IASIST S.A., a company specializing in the conduct of studies of this type. The databases of 33 teaching hospitals from the Spanish National Health Service were used as an external reference pattern, known as the external norm (EN).
Case SelectionFor the purpose of comparison, 13 of the participating centers were used to determine an internal norm for the BM (4 centers were excluded, as their MBDS were incomplete). Cases of major pulmonary resection for lung cancer were selected (lobectomies, pneumonectomies, atypical segmental resections, and video-assisted lobectomies). These cases were identified using the codes of the 9th edition of the 2008 International Classification of Diseases, Clinical Modification (ICD-9-CM) retrieved from the records of the surgical procedures. The following cases were selected: lobectomy (codes ICD-9-CM: 32.3 and 32.4), pneumonectomy (codes ICD-9-CM: 32.5 and 32.6), video-assisted lung resections (codes ICD-9-CM: 32.20, 32.25 and 32.28), atypical segmental resections (code ICD-9-CM 32.29), pneumothorax (code ICD-9-CM 512.0 and 512.8). To ensure that all thoracic surgery activity was recorded, surgical cases coded CM 04 (respiratory system) were also included.
Performance IndicatorsComplexity of the Case-MixThe following indicators were used:
- •
Mean weight. Calculated from the diagnosis-related groups (DRG), version AP 21: all patients seen in all hospitals were classified.
- •
Relative weight. This is the ratio between the mean weight of the BM study cases and the mean weight of the external norm. This is a measure of the complexity of the case-mix compared to the external norm.
- •
Indicators of performance outcomes, such as average length of stay (ALOS) and readmissions, were adjusted by case-mix, using refined DRG, with a subclassification of DRG in categories of severity based on secondary diagnoses recorded for each patient.
The following indicators were included in the evaluation of outcomes:
- •
Average length of stay (ALOS). Preoperative, postoperative and overall ALOS were analyzed with respect to lobectomies, pneumonectomies, video-assisted resections, atypical lung resections, and pneumothorax. ALOS adjusted for severity of the case-mix in each TSU [risk-adjusted ALOS, or RAALOS] was also analyzed. Severity was adjusted using the IASIST procedure, which is not specific to thoracic surgery. The length of stay (LOS) index was calculated, i.e., the ratio of the observed length of stay and the expected length of stay (that which would be applicable if the patient's stay had been according to the norm).
- •
Mortality. In-hospital mortality was evaluated and adjusted for risk. The risk-adjusted mortality index (RAMI) was the ratio of observed and expected mortality in the study case series. Expected mortality was calculated using a logistic regression model, which calculates the probability of mortality for each patient from a database of more than 3 million hospital discharges.
- •
Complications. A group of 25 general complications and 7 sentinel complications were evaluated. The following complications were analyzed: pulmonary (atelectasis, pneumonia, respiratory failure), pleural (pneumothorax, empyema, hemothorax), cardiovascular, post-operative bleeding, and wound infections. Specific complications included bronchopleural fistula, and wound dehiscence. The indicator used was the risk-adjusted complication index (RACI), the ratio between the number of observed and expected complications. The expected complications were calculated in the same way as mortality, using a logistic regression model to calculate the probability of complications from a database of more than 3 million hospital discharges.
- •
Readmissions. Urgent readmissions related to the original admission were analyzed. Readmissions were also adjusted for the complexity of each TSU (risk-adjusted readmission index [RARI]). The RARI index was used to show the ratio between observed and expected readmission, according to the readmissions indicator of the norm.
All Major Disease Category 04 (MDC4) surgical activity, successive and first outpatient visits, and hospitalization were taken into account. The following aspects were studied:
- •
Surgical activity by department.
- •
Number of hospital beds and bed/surgeon ratio.
- •
Number of surgeons/department.
- •
Hospital activity by department (units of hospital output per surgeon).
- •
Resolution capacity of outpatient visits (ratio of successive/first visits).
- •
Occupation rates.
BM results from the 17 Spanish TSUs were analyzed. Of a total of 1125011 hospital episodes, 8250 treated in the participating TSU were selected. After exclusion of episodes of MDC14 and MDC15 and patients <18 years of age, 7807 episodes remained.
Overall ResultsThe TSUs were responsible for between 0.6% and 1.6% of all discharges from their respective hospitals. The numbers ranged widely, from 168 to 794 (BM norm 432; EN 366). The mean weight lay between 2.39 and 5.02 (BM norm 3.2; EN 2.9) (Fig. 1). The relative weight with respect to the BM ranged between 0.7 and 1.6. The case-mix in each TSU followed the same trend. The distribution of DRGs 075. 077. 095 and 538 is shown in Fig. 2, which includes the calculation of the N ratio (DRG 538/(DRG 075+DRG 538).
ALOS varied widely among the TSUs, ranging from 4.1 to 10.7 (BM 7.5; EN 7.1). The RAALOS ranged between 0.5 and 1. The preoperative ALOS was between 0.2 and 1.7 (BM 0.9; EN 1). LOS index was between 0.2 and 4.5, also confirming high variability.
Mortality ranged between 0.2 and 2.5% (BM 1.1%; EN 1%). RAMI ranged from 0.1 to 1.2. The complication rate was 0.6–11.1% (BM 6.3%; EN 6.5%). RACI was between 0.2 and 1.6. Postoperative empyemas occurred at a rate of 0–1.5% (BM; EN 0.7%); bronchopleural fistula, 0–1.1% (BM 0.4%; EN 0.3%); nosocomial pneumonia, 0–3.2% (BM 1.3%; EN 1.1%), and operative wound dehiscence, 0–0.9% (BM 0.3%; EN 0.2%). Readmissions within 30 days of discharge ranged between 0% and 6.4% (BM 4.5%; EN 4.4%). RARI was 0–1.9.
Activity assessment is shown in Fig. 3. The number of beds per TSU ranges from 5 to 13 (BM 9), and the number of surgeons from 2 to 7 (median 4, BM 4). The number of beds per surgeon ranges from 1 to 6; most TSUs have between 1.5 and 3 (BM norm 3). The BM norm for units of hospital output per surgeon was 363, but most TSUs are below this figure. Occupation rates ranged between 33% and 151% (EN 86%). The ratio of first outpatient visits ranged between 0.6 and 5.4 (BM 2.6).
Analysis of Surgical Procedures
The percentage of surgical activity among the total number of admissions ranged from 71% to 91% (BM 84%). The surgical ALOS ranged from 4.3 to 14 (BM 8.7; EN 9.1), and RAALOS from 0.5 to 1. The rate of surgical mortality ranged from 1.3% to 6.1% (BM 3.02; EN 2.7%) and RAMI from 0.5 to 1.8. The complication rate varied between 1% and 14% (BM and EN, 9.5%), and the RACI ranged from 0.3 to 1.6. The number of interventions varied widely (Fig. 3). Activity per surgeon also showed wide variability (34–158; BM 42).
Pulmonary LobectomiesHalf of the units (50%) performed between 50 and 70 lobectomies (Fig. 4) (BM 107; EN 79), with an ALOS of between 5.9 and 10.8 (BM 8.6; EN 8.8), and a RAALOS of between 0.6 and 1.2 (Fig. 5). Preoperative ALOS was 0 to 1.3 (BM 0.9; EN 1). The LOS index ranged from 0 to 2.8; most units scored more than 2. Mortality rates ranged from 0% to 5.2% (BM 2.2%; EN 1.9%), while RAMI was between 0 and 1.6. The rate of complications ranged from 0% to 20% (BM 9.3%; EN 12.1%), with a RACI of 0–2. Most TSUs scored between 1.3 and 1.5. The readmissions rate was between 0% and 8.6% (BM; EN 4.3%), with a RARI of 0–3.5.
Between 3 and 20 pneumonectomies/year were performed (BM 9; EN 8). Most TSUs performed 8–9/year, with an ALOS of between 6.6 and 32.5 (BM 11.3 and EN 12.8), RAALOS 0.7–2.1. Complications and mortality ranged from 0%–35% to 0%–25%, respectively.
A large number of video-assisted lobectomies were performed in some TSUs (with up to 68 in 1 unit, and more than 30 in another 4). Other TSUs performed fewer video-assisted resections (between 1 and 25/year). ALOS ranged between 3.7 and 12.6 (BM and EN 7.4); ALOS was 1.2 days shorter than for conventional lobectomies.
Atypical resections varied widely. In 4 TSUs, 10 or fewer atypical resections were performed, and another 7 TSUs performed between 20 and 33 of these procedures. Other TSUs performed interventions of this type on more than 40 occasions. ALOS ranged between 4.3 and 11.2 (BM 7.5; EN 7.9).
Pneumothorax also occurred at varying rates in the different TSUs, ranging from 7 to 79 (BM 38; EN 33), with an ALOS of 2.6–8 (BM 5.8; EN 6.2).
DiscussionBM has increased in recent years.6–8 Despite the reported advantages of analyses of this type, no well-designed studies have demonstrated the clinical benefits of undertaking benchmarking.5 Indeed, the results of this edition of our BM study reveal that problems detected in previous analyses persist, as does the wide variability among TSUs. Specific corrective measures obviously need to be implemented and quality standards need to be created. This issue should probably be addressed by ad hoc committees already operating within scientific societies. Other authors have also suggested that the medical societies could initiate BM procedures, as was the case in this study, and could even create prospective databases.9,10
The most significant limitation of a BM study is that it is carried out without a full analysis of the processes. This flies in the face of some observations on the validity of using administrative databases in the evaluation of healthcare quality,11 although some groups defend this approach.12–14 Another problem is the origin of the data. The basic minimum data set has some limitations that were pointed out in our earlier publications.4,5 The creation of prospective databases with internal quality control, supported by the European Society of Thoracic Surgeons,15 may be of great importance for benchmarking in the future.
One important finding, also a feature of previous benchmarking studies, was the wide variability in the results obtained from the different TSUs. The number of discharges, the population profile, the distribution of discharges by DRG, and the rate of surgical activity all vary widely, and this affects the mean weight and complexity index of the participating TSUs. This has also been reflected in the percentage of DRG 075 in each of the TSUs, which ranged between 25% and 50%, and in the number of lobectomies performed (25–200). Complexity calculations and risk adjustments are therefore essential when comparing data from different units.16 Measuring severity, however, depends heavily on the quality of departmental discharge reports, which must detail all patient comorbidities and complications. The importance of this has been discussed at length in previous BM evaluations.4,5 The analysis of complications also revealed wide differences among the participating sites, possibly because these data were missing from discharge reports. Improvement and standardization of discharge reports is one of the main areas of improvement in some TSUs.
Some specific complications associated with thoracic surgery occurred at rates similar to internationally accepted standards, and mortality was in the same range.17 These aspects have varied little since previous BM studies, although some sites should perform a detailed analysis of their data and their impact on patient safety.
Emergency admissions generally impact negatively on clinical management and services. A significant number of admissions of this type occurred in the BM participants, ranging from 17% to 46%. This is another area for improvement in the future, and the introduction of patient admission protocols would probably be useful.
ALOS is a widely used hospital indicator,18 and varied greatly among our TSU participants, as can be seen in the RAALOS, a more objective data point. It ranged from 5.9 to 10.8 in lung resections, with a mean BM value of 8.6 and a LOS index of 0.6–1.2. Preoperative ALOS also varied significantly, from 0.3 to 1.6.
Lobectomy outcomes were good, in general, and RAALOS values were low. Preoperative ALOS, however, could be improved in the future. Mortality after lobectomy was low (2%), but morbidity varied widely, possibly due to differences in the quality of the discharge reports. Some departments reported a high rate of related emergency readmissions, also reflected in the widely ranging RARI, suggesting that the discharge procedures for this type of surgery need to be reviewed.
The use of video-assisted lobectomies varied widely among the participating groups. A few units carried out more than 30 procedures of this type every year, but others performed hardly any. Another point of interest is the moderate impact of the procedure on ALOS: 1.2 days less than for conventional lobectomy.
Clinical pathways and fast track procedures need to be implemented in pulmonary resection surgery.19 Benchmarking, along with other more conventional methods of auditing healthcare performance10 and patients’ perception of quality,20 is useful for the overall analysis of clinical activity. The analysis of activity, not included in our previous BM studies, provided widely variable data, as can been seen from the range of units of hospital output – 200–800. Activity needs to be analyzed in greater depth in future BM studies, and some areas, such as outpatient visits, merit particular attention.
ConclusionsWe found considerable variability in quality, activity, and efficiency parameters among our participating TSUs.
We identified obvious areas for improvement: admission processes must be standardized to avoid urgent admissions and to shorten preoperative ALOS, hospital discharges must be streamlined, and discharge reports improved by including all procedures and complications.
Some thoracic surgery units have parameters which deviate excessively from the norm, and these sites need to review their processes in depth. Coding of diagnoses and comorbidities is another area which could be improved in some units.
Conflict of InterestsThe authors state that they have no conflict of interests.
We thank Dr Mercè Casas, Dr María Soler, and Mrs Marta Albacar from IASIST, for their technical support.
Mercedes de la Torre. Hospital Universitario Juan Canalejo de la Coruña.
Laureano Molins López-Rodó. Hospital Universitari Clinic de Barcelona.
Juan José Fibla Alfara. Hospital Universitari Sagrat Cor de Barcelona.
Florentino Hernando Trancho. Hospital Clínico San Carlos de Madrid.
Joaquín Pac Ferrer. Hospital Universitario de Cruces. Baracaldo.
José Miguel Izquierdo Elena. Hospital Universitario de Donosti.
Jorge Freixinet Gilart. Hospital Universitario de Gran Canaria Dr. Negrín. Las Palmas de Gran Canaria.
Benno Baschwitz. Hospital Universitario de Alicante.
Pedro López de Castro. Hospital Universitari Germans Trías i Pujol de Badalona.
Juan José Rivas de Andrés. Hospital Universitario Miguel Servet de Zaragoza.
Ángel Carvajal. Hospital Universitari Son Espases de Palma de Mallorca.
Mercedes Canela. Hospital Universitario Vall d¿Hebrón de Barcelona.
Emilio Canalís Arrayás. Hospital Universitari Juan XXIII de Tarragona.
Ángel Salvatierra Velázquez. Hospital Universitario Reina Sofía de Córdoba.
Juan Torres Lanzas. Hospital Universitario Virgen de la Arraixaca de Murcia.
Gonzalo Varela Simó. Complejo Asistencial Universitario de Salamanca.
Nicolás Moreno Mata. Hospital Universitario Virgen del Rocío de Sevilla.
Please cite this article as: Freixinet Gilart J, Varela Simó G, Rodríguez Suárez P, Embún Flor R, de Andrés JJR, de la Torre Bravos M, et al. Benchmarking en cirugía torácica. Tercera edición. Arch Bronconeumol. 2016;52:204–210.
