Discussion: how to think about CMA results

Step 1: Is it the “right” objective?

In a CMA, the only thing that matters is cost. It is assumed that both options are identical with respect to effectiveness. Because of differences in wage rates between RNs and MDs, RNs are cheaper than physicians. But other health care practitioners are cheaper than RNs, right? In an editorial, Fletcher and Farraye⁶ opine that, “technical skill is not conferred by an MD degree, nor even by subspecialty certification, but rather earned by rigorous training and experience… in principle it [the results] might apply to other nonphysicians… with comparable training and experience.”

Consider the observation, “Screening sigmoidoscopy programs are limited, in part because of the workforce necessary to perform the examinations. The current number of gastroenterologists… would have difficulty performing all the sigmoidoscopies that would be required for a national screening program.”⁶ This suggests that the problem lies with limited capacity, rather than with the need to find cheaper ways of doing things. In the face of rising Canadian rates of colorectal cancer, if the goal is to increase access to screening, then options should also consider extra effectiveness (ΔE). If the objective is to address the rising cancer rates, the outcome should be measured in terms of additional numbers screened, cases detected, or life-years gained. If one of these outcomes captures the actual challenge the decision-maker is facing, then comparing two options based solely on costs is not sufficient. The actual decision requires considering both the extra cost and extra gain. Furthermore, to what should we compare the option of using RNs? Is the relevant comparator: do nothing or increase the number of physicians?

A leader’s objective will vary depending on the decision to be made. In turn, the optimal analysis to inform the decision will also vary. For example, if the goal is to provide more access to screening for a population living in flexible sigmoidoscopy deserts, and one option is to have RNs perform the procedure, cost-effective analysis rather than CMA is needed to inform the decision. In this scenario, standard practice is not allowing the organization to meet its objective of detecting colorectal cancer. The question it faces is whether investment in a new option yields additional gain at an additional cost that is acceptable to the organization. The extra cost per extra case of colorectal cancer detected is calculated with the cost-effectiveness ratio ΔC/ΔE. Extra effect (ΔE) could be measured as additional cases detected or life years gained (if detection leads to treatment that prolongs life). In the study by Cipriano and colleagues,⁴ the cost-effectiveness of testing for GA-2 was over $142 million per additional year of life. For colorectal cancer, the cost-effectiveness of an RN-performed flexible sigmoidoscopy option is likely much more economically attractive. However, the cost-effectiveness would vary depending on whether the scenario studied was one where physician-performed flexible sigmoidoscopy is not being done at all, or one where head-to-head screening (RN vs. physician) was being considered.⁷ That is, is the alternative to RN-performed flexible sigmoidoscopy no screening at all? Or, is the alternative that the RN substitutes for the MD?

In contrast, if the goal was to answer the question, “Are RNs or physicians a cheaper way to screen 100 adults aged ≥ 50 years at average risk for the colorectal cancer, who are already in our waiting room?” then the objective is cost minimization and not better access, reduced waiting times, or increasing screening. In fact, if the physicians who no longer perform flexible sigmoidoscopy (since the RNs are doing it) see other patients for other things, adopting RN-performed flexible sigmoidoscopy does not affect access. Colorectal cancer screening rates will still be below those for other common cancers, but at least the under-achievement will be cheaper. A CMA answers which option is cheaper to accomplish the same thing. But do both options accomplish the same thing?

Step 2: Do you believe the assumptions?

All CMAs are predicated on the assumption that there is no difference between two ways of doing something (so ΔE = 0). In our example, Costa et al.⁵ refer to Schoenfeld et al.⁸ to support their assertion that RN-performed flexible sigmoidoscopy is as effective as physician-performed flexible sigmoidoscopy. In their commentary on the Schoenfeld et al.⁸ study, Fletcher and Farraye⁶ observe:

It may be that there are greater differences in performance among individual endoscopists than there are between physician and nonphysician endoscopists. The variation in polyp miss rates among physicians (24–41%) and nurses (11–23%) in the Schoenfeld study was far greater than the difference in adenoma miss rates between the two kinds of endoscopists, 1%. Perhaps we should be more interested in who performs the endoscopy than in his or her training and credentials.

Schoenfeld et al.⁸ found that gastroenterologists and nurse endoscopists had very similar miss rates for adenomatous polyps (20% vs. 21%, respectively; p = 0.91). This is the source of the 1% figure that Fletcher and Farraye⁶ reference.

Although 20% and 21% seem very similar, from a statistical viewpoint, similar is not equivalent. It is as Frank Robinson once said, “Close only counts in horseshoes…” In this case, the statistics that are cited do not mean that RN and physician flexible sigmoidoscopy miss rates are the same. Rather, they indicate that the null hypothesis that they are the same cannot be rejected. Although seemingly esoteric, this conclusion is important. The implication from a health economics perspective is that, “because failure to reject the hypothesis about the equality of two therapies is not the same as finding that outcomes of two therapies are identical, cost-effectiveness analysis should still be performed if the clinical study fails to demonstrate a statistically significant difference in clinical end points.”⁹ When making a cost-effectiveness analysis model (as opposed to analyzing a cost-effectiveness dataset), this can lead to situations in which it is difficult to explain results especially when null findings actually result in evidence of cost-effectiveness.¹⁰ Nevertheless, cost-effectiveness analysis is often preferred by health economists as a way of providing a more fulsome picture of the trade-offs between two options, even if they appear similar. While CMA has been pronounced as “dead”¹¹ and “dead and buried”¹² in the scientific literature, there are practical examples of how it can be used to inform real-world funding recommendations (e.g., Tirrell et al.).¹³

Step 3: Are there other things that matter?

Even if the objective is saving money and even if the results of flexible sigmoidoscopy are equivalent if done by RNs or physicians, there may be other things that matter. Some health technology assessment processes classify these “other things” into two groups: contextual considerations and other benefits or disadvantages. Trenaman et al.¹⁴ found that judgements about the value of interventions are influenced by contextual considerations as well as other benefits or disadvantages and are anchored by cost-effectiveness. In the case of flexible sigmoidoscopy, there may be political, cultural, ethical, social justice, and other issues surrounding an endorsement of RN-performed flexible sigmoidoscopy. If this is true, then it is important to recognize that studying costs may be a sound activity but will not supply a sufficient answer on which to base a decision. In this case, CMA is an incomplete evaluation, and the cost difference is simply another variable to inform a decision addressing various impacts (i.e., cheaper may not be better).

Conclusion

In the example we reviewed, the research question answered was, “Are nurses cheaper than physicians?” The first key point is that this simplistic question may not supply a useful answer. In some scenarios, knowing the cheaper way of doing the same thing is very useful. In other cases, the cheaper way may not produce the same outcome. In situations where there may be differences (or not) in costs and outcomes, studying both costs and outcomes is the right choice.

This brings up the second key point; CMA is only useful in situations when both options being considered result in equivalent outcomes. Given that providing health care is often not akin to a game of horseshoes, “close is not good enough.” Cost minimization only gives answers about costs whereas cost-effectiveness studies both costs and outcomes. Because of its strong assumption that there is no difference in outcomes, despite being technically well-done, a CMA may not provide enough insight to inform more complex decisions.

CMA can seem like the right thing to do when there is evidence of non-inferiority for two options.¹⁵ Leaders must consider whether important facets of the decision are being overlooked when restricting one’s field of focus to finances. In our discussion of the case, we focused on three areas for evaluating the usefulness of a CMA: objectives, assumptions, and other considerations. We conclude by suggesting that leaders carefully consider whether a study’s objective correctly aligns with the organization’s objectives, and whether there is more than one relevant objective. If so, then the assumption that the cheaper way is the better way may not hold. A CMA tells you what is cheaper. Leaders often account for more than one factor in making their decisions. In fact, addressing contextual considerations as well as other benefits or disadvantages is often crucial to making good decisions. CMA does not respect strategic ambiguity.¹⁶ Results showing “cheaper” are not results showing “better,” unless the only acceptable decision criterion is cost.

References

1. Hoch JS, Dewa CS. The occult of efficiency: frank, and Stein’s, advice for physician leaders. Can J Physician Leadersh 2024;10(1):29-32. https://doi.org/10.37964/cr24780

2. Hoch JS, Dewa CS. Cost is not a four-letter word: focus on what you can change. Can J Physician Leadersh 2024;10(2):64-8. https://doi.org/10.37964/cr24783

3. Hoch JS, Dewa CS. Maximizing success when it is the product of two things that go in opposite directions: the magic of elasticity. Can J Physician Leadersh 2024;10(3):84-91. https://doi.org/10.37964/cr24784

4. Cipriano LE, Rupar CA, Zaric GS. The cost-effectiveness of expanding newborn screening for up to 21 inherited metabolic disorders using tandem mass spectrometry: results from a decision-analytic model. Value Health 2007;10(2):83-97. https://doi.org/10.1111/j.1524-4733.2006.00156.x

5. Costa SE, Coyte PC, Laporte A, Quigley L, Reynolds S. The use of registered nurses to perform flexible sigmoidoscopy procedures in Ontario: a cost minimization analysis. Healthc Policy 2012;7(3):e119-30.

6. Fletcher RH, Farraye FA. Screening flexible sigmoidoscopy: effectiveness is not enough. Gastroenterology 1999;117(2):486-8. https://doi.org/10.1053/gast.1999.0029900486

7. Richardson G, Bloor K, Williams J, Russell I, Durai D, Cheung WY, et al. Cost effectiveness of nurse delivered endoscopy: findings from randomised multi-institution nurse endoscopy trial (MINuET). BMJ 2009;338:b270. https://doi.org/10.1136/bmj.b270

8. Schoenfeld P, Lipscomb S, Crook J, Dominguez J, Butler J, Holmes L, et al. Accuracy of polyp detection by gastroenterologists and nurse endoscopists during flexible sigmoidoscopy: a randomized trial. Gastroenterology 1999;117(2):312-8. https://doi.org/10.1053/gast.1999.0029900312

9. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) — explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force. Value Health 2013;16(2):231-50. https://doi.org/10.1016/j.jval.2013.02.002

10. Raftery J, Williams HC, Clarke A, Thornton J, Norrie J, Snooks H, et al. ‘Not clinically effective but cost-effective’ — paradoxical conclusions in randomised controlled trials with ‘doubly null’ results: a cross-sectional study. BMJ Open 2020;10(1):e029596. https://doi.org/10.1136/bmjopen-2019-029596

11. Briggs AH, O’Brien BJ. The death of cost-minimization analysis? Health Econ 2001;10(2):179-84. https://doi.org/10.1002/hec.584

12. Dakin H, Wordsworth S. Cost-minimisation analysis versus cost-effectiveness analysis, revisited. Health Econ 2013;22(1):22-34. https://doi.org/10.1002/hec.1812

13. Tirrell Z, Norman A, Hoyle M, Lybrand S, Parkinson B. Bring out your dead: a review of the cost minimisation approach in health technology assessment submissions to the Australian Pharmaceutical Benefits Advisory Committee. Pharmacoeconomics 2024;42(11):1287-1300. https://doi.org/10.1007/s40273-024-01420-9

14. Trenaman L, Pearson SD, Hoch JS. How are incremental cost-effectiveness, contextual considerations, and other benefits viewed in health technology assessment recommendations in the United States? Value Health 2020;23(5):576-84. https://doi.org/10.1016/j.jval.2020.01.011

15. Cuzick J, Sasieni P. Interpreting the results of noninferiority trials — a review. Br J Cancer 2022;127(10):1755-9. https://doi.org/10.1038/s41416-022-01937-w

16. Pascale RT. Zen and the art of management. Harv Bus Rev 1978;56(2):153-62. Available: https://hbr.org/1978/03/zen-and-the-art-of-management