Measurement
Measurement is fundamental to being able to demonstrate improvement. You need to be able to define what it is that you will strive to improve, and how you will know if you have achieved it. "If you can't measure it, you can't improve it.” In business management, this quote is often attributed to Peter Drucker.
These are useful to review and monitor to ensure that the AMS program has the adequate structural supports to be successful. Most services will include this in their accreditation review.
It is possible that an AMS program is functioning despite these structural measures; however, they should be regularly reviewed, even in well-established programs, and the gaps demonstrated and reported to maintain advocacy. For example, it is possible that an established program may find that its committee is not meeting regularly enough or is not reporting adequately to surrounding governance structures. It can be useful to clearly define targets so that the program can be measured against these and identify where it needs to improve. For example:
The latter can help to illustrate when staff resources are falling short; e.g., if the organisation changes in size, the resource allocation may need to be adjusted.
This keeps the AMS team accountable for its activity. Setting targets and monitoring activity against these targets can identify performance issues. For example:
These constitute the main aims of the AMS program and they describe what the team is trying to achieve. The outcomes can broadly be grouped as those related to the drugs, those related to the bugs and those related to the patient.
In many cases, hospital AMS programs can become fixated on measures related to the antimicrobial agents because these are often the easiest to monitor. While it is important to try to reduce unnecessary antimicrobial consumption where possible, it is also important to show that any reduced consumption has not been to the detriment of patients and not led to worse patient outcomes (i.e., there has not been any unintended harm to the patient).
Furthermore, while monitoring of local pathogen resistance profiles is important, we caution against the use of this as a primary measure of the success of the AMS program as these can change for a number of reasons that may be outside the control of the AMS program (e.g., community prescribing, infection control activity). Clinical outcomes are likely to be the most meaningful for clinicians and, where possible, they should be built into the AMS program.
Outcomes measures will be described later.
It is useful to develop a framework to guide your measurement activities. This will provide some clarity on what needs to be measured and how they are to be measured, and help you ensure that you have covered all the relevant measurement types.
The following table describes a range of measures that should be considered as part of the evaluation of your AMS program. Not all of these may be relevant or possible, or there may be other measures not mentioned in this table that are important.
Refer to: British Society for Antimicrobial Chemotherapy's A Practical Guide to Antimicrobial Stewardship in Hospitals & Australian Commission on Safety and Quality in Health Care's Antimicrobial Stewardship in Australian Health Care (2018).
The measurement of antimicrobial consumption is part of the evaluation process to understand and compare the use of antimicrobials over time and multiple areas. The measurement of consumption allows one to:
Where possible, it is useful to monitor the consumption of all antimicrobials kept at the hospital to obtain a holistic view of antimicrobial usage. However, if resources are very limited, preference can be given to monitoring broad-spectrum antibiotics and those at higher risk of resistance development such as those categorised in the ‘Watch’ group of antibiotics in the AWaRe List from the WHO Model Essential Medicines List.
Interpretation of drug consumption information should be done in the context of other events occurring in the local area, antibiogram data and any other local changes. For example, a rise in use in one antibiotic may be the direct result of stock shortage of another. Analysis for seasonal trends may also be helpful for interpretation of data.
There are numerous ways to measure drug consumption depending on resources, availability of data and benchmarking ability.
Data mentioned above is not always available, in which case, a best attempt should be made at some sort of measurement.
Day of therapy (DOT) is a measure of drug consumption that reflects actual usage for each drug. It refers to the number of days that a patient receives an antimicrobial agent, regardless of the dose, route or frequency of administration. When adjusted for hospital activity, it is a useful measure that facilitates meaningful benchmarking, as is done by several countries around the world.
Examples of DOT calculations:
The DOTs for a given patient on multiple antibiotics will be the sum of DOTs for each antibiotic that the patient is receiving. In examples 1 and 2 above, the days of therapy for ceftriaxone and azithromycin are added together to give 10 DOTs, even if the two antibiotics are given concurrently. This is a major limitation of using DOT as broad-spectrum monotherapy will appear more favourable compared with dual therapy that is narrower in spectrum.
DOT is often standardised to 1000 patient days (DOT/1000 patient days) or PDOT to allow comparison between hospitals or services of different sizes. Patient-days is a count of the number of days a patient is present on an inpatient unit, measured at a specific time each day. Midnight is a common census time used.
DOT is a useful measure to compare data from multiple populations. DOT is the preferred measure in a number of countries for reporting national data.
DOT is broadly applicable to paediatrics (including neonates).
However, DOT is difficult to measure accurately and requires computerised medication management systems.
When this information is not readily available, DDD methodology for measurement of drug consumption is used together with point-prevalence auditing to understand quality and quantity of antimicrobial use.
In the absence of electronic medication management systems, defined daily dose (DDD) is a useful measure of drug consumption. The ATC/DDD methodology is an internationally recognised measure of drug usage, recommended by the WHO. Its purpose is to serve as a tool for producing good-quality, usable and comparable drug utilisation statistics.
Drug utilisation data presented in DDDs gives an approximate estimate of consumption. However, it does not provide an exact picture of actual usage. DDDs provide a fixed unit of measurement independent of price, currencies, package size and strength, enabling the researcher to assess trends in drug utilisation and to perform comparisons between population groups.
DDD is a measure to allow data collected from different populations to be compared.
Drug utilisation figures should ideally be presented using a relevant denominator for the health context. In the hospital setting, this would be the number of patients or number of bed days in the setting and period monitored. Description and calculation of the most common indicators such as DDD/patients and DDD/100 bed days, DDD/1000 population/day gives an approximate measure of how many people are taking an antibiotic on any given day. Alternatives include:
Occupied bed days (OBD) refers to the number of patients in hospital overnight and is a useful denominator for benchmarking and adjusting for hospital activity.
Drug usage (in DDDs) = Items issued x Amount of drug per item (grams)
DDD (g)
For example, vancomycin usage can be measured this equation.
24 vials of vancomycin 500mg (IV) injection are given.
Vancomycin (IV) drug usage (DDDs) = (24 vials x 0.5)/2 = 6 DDDs
DDDs are a standard measure for adults and not suitable for paediatrics. Thus far, a useful comparison method for paediatric drug consumption has not yet been developed.
Prescribed daily dose (PDD) is the average daily dose prescribed for a sample of prescriptions for the same antimicrobial. In institutions where prescription data (e.g., electronic prescriptions, medication charts) is more readily available than drug usage data, this may be an option for measurement. By taking into account indication, it can be more reflective of therapeutic use compared to DDDs. However, PDDs are based on individual characteristics such as age, weight, ethnic differences, type and severity of disease, and pharmacokinetic considerations, and, therefore, it is inherently difficult to use PDD for benchmarking. Also, PDD may not reflect actual drug utilisation, where compliance to therapy may need to be considered. In such cases, point-prevalence surveys may be more accurate.
Whilst antimicrobial purchasing data is the easiest to obtain, it is an unreliable measure of antimicrobial consumption for a number of reasons:
While there are several disadvantages, it is nonetheless important to acknowledge that in some hospitals, this will be the only information that is available. In these cases where more accurate data are not available, a best attempt should be made at some sort of measurement. The AMS team should work together with other hospital stakeholders to move towards more reliable measures of consumption.
In addition to monitoring the quantity of antimicrobial use, it is important to review and measure the quality of antimicrobial prescribing in terms of safety and appropriateness, and to determine whether prescribing restrictions and guidelines are being followed. This will allow an AMS program to determine whether patients are receiving the most appropriate antimicrobial therapy whilst minimising harm due to adverse effects and antimicrobial resistance.
Measuring the quality of prescribing involves conducting audits with review of patient clinical records (medication chart, medication administration record and clinical notes) to determine:
Members of the AMS team are ideally placed to conduct quality audits; however, it is equally beneficial to involve general clinicians as the act of auditing itself leads to reflection and improved understanding of appropriate antimicrobial prescribing. This allows a hospital’s workforce to further develop the skills in AMS, enhancing knowledge and improving practice.
There are two main types of audits that are suited for this type of activity, which are most effective when used in combination: prevalence audits and quality improvement audits.
Prevalence audits are a useful method of gaining an overall picture of the quality of antimicrobial prescribing at a hospital, identifying target areas for improvement and monitoring the effect of interventions over time.
Prevalence audits are usually conducted as point-prevalence surveys (PPS), providing a ‘snapshot’ of antimicrobial prescribing at a given point in time. PPS tend to be performed on a large scale, often hospital-wide and on a single day; hence, they can be resource-intensive to conduct and are usually only performed once or twice per year. Nonetheless, they hold immense value in allowing an AMS program to detect target areas to direct their efforts towards the areas that are most in need.
There are several methodological options for conducting a PPS which may suit hospitals of differing sizes and staffing levels:
Patients are included in the audit if there is an active antimicrobial prescription. It is important to include both restricted (broad-spectrum) and non-restricted (narrow-spectrum) antimicrobials and not to just assume that non-restricted or narrow-spectrum antimicrobials are necessarily well prescribed.
The following data should be considered for collection if available:
Hospitals with ready access to experts may also choose to make an assessment of the overall prescription appropriateness.
The majority of the data can be collected by pharmacists, nurses or infection control practitioners. However, input from a medical doctor (preferably an infectious diseases physician or a clinical microbiologist if available) should be sought to assist in making clinical assessments of concordance with guidelines and appropriateness. If the necessary expertise is not available on-site, the data should be reviewed by an external assessing team.
One disadvantage of prevalence audits is that, due to the time required to collate and analyse data, there may be a lag time between when the audit is conducted and when the results are available for feedback to prescribers. This may be problematic if the prescribers have rotated to different specialty units or different sites, resulting in a situation where the prescribers who receive the results of the audit were not involved in the prescribing decisions audited. The AMS team should, therefore, highlight the importance of all prescribers taking ownership and responsibility for the appropriate prescription of antimicrobials throughout the hospital.
The Hospital NAPS is a comprehensive standardised auditing tool used across Australian hospitals to measure the quality of antimicrobial prescribing. The Hospital NAPS platform is managed by NCAS. It is an example of a tool that has been utilised by different hospital types, large and small, metropolitan and remote, well-resourced and poorly resourced. The NAPS uses standardised definitions, methodology and assessment algorithms to assist participating hospitals to collect antimicrobial prescribing information in a robust and meaningful way. This consistency of data collection enables benchmarking between participating hospitals as well as a jurisdictional or national overview of the quality of prescribing.
Information is entered through a dedicated data entry portal and the subsequent reports are immediately available and presented in a visually appealing and easy-to-understand format that clearly highlights the key areas that may require improvement. Examples include:
The
Global Point-Prevalence Survey of Antimicrobial Consumption and Resistance (GLOBAL-PPS) is a project that invites hospitals for around the world to collect data on the rate and quality of antimicrobial prescribing together with microbiology and resistance information. The data collection form is available online and may be a useful tool for AMS teams to consult when designing their own data collection forms.
Quality improvement (QI) audits are small, quick audits that are designed to be conducted frequently and allow for prompt prescriber feedback. They can have large beneficial impact with minimal resources and are often a good way of involving non-AMS clinicians in antimicrobial auditing. While they are beneficial for all hospital types, they are particularly useful for small, remote hospitals or hospitals with limited resources. The utility of QI audits have been demonstrated by the Scottish Antimicrobial Prescribing Group and the Australian NSW Clinical Excellence Commission’s (CEC) “5x5 Antimicrobial Audit”.
QI audits complement PPS as these audits can be targeted towards certain problem areas that have been identified through the large-scale PPS (e.g., a particular ward, specialty, antimicrobial or indication). They assess in real time the outcomes of any interventions made and can be utilised to facilitate regular targeted feedback.
QI audits can focus on just a few key quality indicators such as:
Some audits may also prompt a clinical intervention, such as contacting the prescriber to clarify an indication or review-date if not documented, or to clarify the reasons for non-compliant prescribing.
New South Wales Clinical Excellence Commission's The 5x5 antimicrobial audit
Antimicrobial susceptibility surveillance data can be used effectively by AMS programs as a key outcome indicator to assess the effectiveness of implemented initiatives. Monitoring trends in antimicrobial susceptibility patterns is an essential step to developing and sustaining meaningful hospital and specialty group antibiograms. This data will help inform hospital or region-specific recommendations on guidelines for empirical antimicrobial therapy for common infectious diseases and pathogens.
Microbiology laboratories should provide these data to their hospitals at least annually. Additionally, hospitals or microbiology laboratories should provide these susceptibility data (for a defined minimum pathogen selection) to a national surveillance body if available to monitor and review national resistance trends. The hospital executive must also ensure that there is made available the resources and technology necessary to allow effective and meaningful detection of resistant pathogens and susceptibility reports.
If your microbiology laboratory or facility does not have the ability or resources to produce their own antibiograms, the WHO has developed and made available a program with the ability to perform this function. WHONET is a free database software program developed by the WHO Collaborating Centre for Surveillance of Antimicrobial Resistance for the laboratory-based surveillance, analysis and reporting of antimicrobial susceptibility test results.
The principal goals of this program are to:
WHONET can be used by individual laboratories or as part of a national and international surveillance network. At present, the software, available in 17 languages, is used in over 80 countries around the world, managing data from over 1000 clinical, public health, veterinary, and food laboratories.
WHONET analytical tools facilitate the:
Software features include:
WHONET has three main components:
The WHO has developed the Global Antimicrobial Resistance Surveillance System (GLASS) to help support the implementation of the Global Action Plan on antimicrobial resistance. The overall goals of GLASS are to enable standardised, comparable and validated data on antimicrobial resistance to be collected, analysed and shared with all countries and participating partners.
A national surveillance system is also vital in planning and implementing any national antimicrobial resistance strategies. This information will also inform and assist in monitoring and evaluating the impact of efforts and interventions at the local, national or global level.
GLASS objectives:
Any country is able to participate in the GLASS program and, therefore, benefit from access to supplies training and implementation tools, and support in collecting antimicrobial resistance data at a local and national level. Country participation in GLASS must be with the agreement of the national government.
Every country should develop a national antimicrobial resistance surveillance system that is able to monitor and respond to trends in resistance profiles of selected pathogens. Upon request, the GLASS secretariat, supported by WHO regional offices and WHO Antimicrobial Resistance Surveillance and quality assessment Collaborating Centres Network, may be able to provide direct support to countries in designing an effective antimicrobial resistance surveillance system.
Some healthcare associated infections (HAIs) can be used as markers of the impact of a hospital AMS program. HAIs are those infections that are not present or incubating at the time of admission to a healthcare facility; develop within a healthcare facility: or are produced by micro-organisms acquired during admission.
CDI is the most common cause of infectious diarrhoea in healthcare facilities. Nearly any antibiotic is capable of disrupting the normal gut microflora, which can allow for C. difficile to flourish and produce toxin. Prolonged antibiotic durations and multiple antibiotics are two risk factors that increase the risk of acquiring CDI. Some healthcare organisations, therefore, use it as a measure of a possible clinical adverse effect related to antimicrobial overuse.
A CDI case is an episode of diarrhoea (unformed stool that takes the shape of the container) that meets the following criteria:
For surveillance purposes, patients less than two years of age on admission and cases where a known previous positive test has been obtained within the last 8 weeks are typically excluded.
The data must, however, be interpreted with caution as increasingly community-acquired cases of C. difficile are occurring. Transmission of C. difficile within the institution is also possible, and IPC practices are thus additionally important.
SSIs are infections of the incision, organ or surrounding space that occur after surgery. Certain operative procedures (e.g., colorectal surgery and small bowel surgery) are associated with a higher risk for SSI than other procedures (e.g., herniorrhaphy). The use of antibiotic prophylaxis can reduce the incidence of SSI related to these procedures. It is important that the correct drug is selected, and that the dose and timing, and redosing are appropriate to optimise the prevention of infection.
Surveillance systems such as the internationally recognised National Health Care Safety Network System infection surveillance system enables data to be collected about operative procedures, the administration of surgical antibiotic prophylaxis and any resulting SSI, including the date of acquisition, specific criteria met for identifying the SSI, the organism(s) identified and the organisms’ antimicrobial susceptibilities.
AMS programs may elect to monitor the in-hospital mortality and length of stay of a common condition such as community-acquired pneumonia as a measure possibly impacted by the AMS program. Active intravenous-to-oral switch campaigns can help to promote earlier hospital discharge, which may become demonstrable over time.
When hospitals initiate antimicrobial restriction programs, it can be important to ensure that these do not lead to delay in appropriate therapy for patients who may need access to these medicines. Such delay may lead to ‘unintended harm’. Some hospitals measure in-hospital mortality from Gram-negative bacteraemia as a ‘balancing measure’ in an effort to monitor for such potential harm. This information can be very reassuring for clinicians, to help show that reduced antimicrobial drug consumption is safe.
SAB is an urgent medical problem as it can cause significant morbidity and has high associated mortality. There is published data to suggest that involvement of experts in the management of patients with SAB is associated with improved clinical outcome. In some facilities, the AMS team may play a role in identifying these patients, advising on management and or referring for early involvement of infectious diseases experts in more detailed management. Conceivably, mortality and length of stay of patients with SAB might be a parameter that could be monitored as a quality marker. It should, however, be done with caution. There is great variability in the clinical presentation of patients with SAB. Some will present late, with multiple complications, and the AMS or infectious diseases teams may have limited opportunity to later the outcome. The service may elect to monitor the frequency and rapidity in which expert advice is obtained for patients with SAB as a marker of AMS impact.
Worldwide there are an increasing number of bacteria resistant to carbapenem antibiotics, including Klebsiella pneumoniae, E. coli, E. cloacae and S. marcescens. All pose a critical global threat of spreading antibiotic resistance.
The objective of CPE surveillance is to detect all cases in order to understand the extent of the problem and if necessary, to ensure infection control and outbreak management processes are immediately applied. This should, however, also include ensuring that appropriate clinical management is instituted early. An AMS team may play a role in guiding appropriate use of complex last-line antimicrobial drugs and/or facilitate access to infectious diseases experts to guide empiric therapy for high-risk patients and directed therapy once susceptibilities are known. Again, clinical outcomes from these infections will be highly variable and will relate to the site and clinical severity of the infection, so they should not be used as measures of the success of the AMS program. The program may, however, choose to monitor the number of cases for which expert advice on antimicrobial therapy is provided within a nominated period.
© Melbourne Health 2024