Validating an EHR Sepsis Prediction Model: How Well Does it Perform?
BACKGROUND AND PURPOSE:
The Epic Sepsis Model (ESM) has been implemented for sepsis prediction at hundreds of US hospitals, despite a lack of testing of its predictive ability
Wong et al. (JAMA Intern Med, 2021) externally validated the ESM in the prediction of sepsis and assessed its potential clinical value compared with usual care
Retrospective cohort study
Admitted to a single academic health system
ESM score, calculated every 15 minutes | Used an ESM threshold of 6 (range of 5 to 8 suggested by developer)
Sepsis was defined by a composite of
CDC surveillance criteria
2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another (International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnostic codes)
Model discrimination was assessed using the area under the receiver operating characteristic (ROC) curve at
Hospitalization-level (when first considered at risk during hospitalization) | 4 hrs | 8 hrs | 12 hrs | 24 hrs
Added benefit of ESM was compared to clinical practice | Based on timely administration of antibiotics
Alert fatigue was evaluated by comparing the clinical value of different alerting strategies
ESM identified patients with sepsis who did not receive timely administration of antibiotics: 7% (183/2552)
Sepsis patients missed by the ESM: 1709 (67% of sepsis cases)
Alert fatigue: ESM generated alerts for 18% of all hospitalized patients
Among hospitalized patients, the Epic Sepsis Model demonstrated low sensitivity, was a poor predictor of sepsis, and also generated a lot of false positives vs usual clinical care
The authors state
The increase and growth in deployment of proprietary models has led to an underbelly of confidential, non–peer-reviewed model performance documents that may not accurately reflect real-world model performance
Owing to the ease of integration within the EHR and loose federal regulations, hundreds of US hospitals have begun using these algorithms
Medical professional organizations constructing national guidelines should be cognizant of the broad use of these algorithms and make formal recommendations about their use
OBG Project CME requires a modern web browser (Internet Explorer 10+, Mozilla Firefox, Apple Safari, Google Chrome, Microsoft Edge). Certain educational activities may require additional software to view multimedia, presentation, or printable versions of their content. These activities will be marked as such and will provide links to the required software. That software may be: Adobe Flash, Apple QuickTime, Adobe Acrobat, Microsoft PowerPoint, Windows Media Player, or Real Networks Real One Player.
Disclosure of Unlabeled Use
This educational activity may contain discussion of published and/or investigational uses of agents that are not indicated by the FDA. The planners of this activity do not recommend the use of any agent outside of the labeled indications.
The opinions expressed in the educational activity are those of the faculty and do not necessarily represent the views of the planners. Please refer to the official prescribing information for each product for discussion of approved indications, contraindications, and warnings.
Participants have an implied responsibility to use the newly acquired information to enhance patient outcomes and their own professional development. The information
presented in this activity is not meant to serve as a guideline for patient management. Any procedures, medications, or other courses of diagnosis or treatment discussed or suggested in this activity should not be used by clinicians without evaluation of their patient’s conditions and possible contraindications and/or dangers in use, review of any applicable manufacturer’s product information, and comparison with recommendations of other authorities.
Jointly provided by
NOT ENOUGH CME HOURS
It appears you don't have enough CME Hours to take this Post-Test. Feel free to buy additional CME hours or upgrade your current CME subscription plan