Why It’s So Hard to Assess False Positives, and How We Might Reduce Them

It’s been nearly a year since we “learned” about the false positives issue, and still there’s hardly any data published on this much-maligned cost of breast cancer screening by mammography.

From the U.S. Preventive Services Task Force November 2009 paper in the Annals, on the lack of information on the incidence of  FP’s:

“Published data on false-positive and false-negative mammography results, additional imaging, and biopsies that reflect current practices in the United States are limited. The probability of a false-positive screening mammography result was estimated at 0.9% to 6.5% in a meta-analysis of studies of sensitivity and specificity of mammography published 10 years ago <ref. 38>. The cumulative risk for false-positive mammography results has been reported as 21% to 49% after 10 mammography examinations for women in general <39–41>, and up to 56% for women aged 40 to 49 years (41)

False positives happen in mammography when a woman has an abnormal result that looks like it might be breast cancer, but upon a further workup – which might involve another, more expert radiologist taking a look, or a sonogram, or an MRI, or even a breast biopsy, but in the end the problem turns out not to be breast cancer.

Why is it so hard to ascertain how often false positives occur?

1. The rate of false positive mammograms is, most likely, declining (i.e. the number is in flux).

Reduced errors would result from two factors: improved methods, such as by adaptation of digital mammography, and by more careful application of extant technology due to progressive compliance with the FDA’s Mammography Quality Standards Act and Program.

2. There’s no precise definition of what constitutes a false positive in mammography, and what isn’t. As I’ve suggested previously, finding a pre-malignant condition like LCIS or an early-stage malignancy like DCIS should not be “counted” as a false positive. If over-treatment occurs, that reflects an error in clinical decision-making rather than in mammography, per se.

3. The false positive rate varies among radiologists and medical centers. That’s because a radiologist who spends her days doing nothing but reading mammograms and breast sonograms will, overall, have a lower FP rate than a general radiologist who also handles hip fracture films and MRIs of the brain.

What are financial costs of false positives? I’m not aware of any new data on this.

How can we reduce the costs of false positives in mammography? My suggestions:

1. Radiologists should be well-trained and carefully-credentialed. As in surgery and other fields in medicine, the physician’s skills and experience affects the probable outcome.

2. Avoid doing breast biopsies in an operating room whenever possible. A fine needle aspirate or core needle biopsy, performed under local anesthesia, is almost always sufficient for diagnosis and less costly.

3. Thinking for the future: Maybe, one good application of Telemedicine would be in sharing digital mammography images, so that no matter where a woman lives, her test could be checked by a radiologist working in a central cancer center and who specializes in breast imaging.

4. Do the procedure every other year for women of average risk for breast cancer (rather than annually). Quite a few theoretical calculations of mammography costs “stack” the purported costs by assuming the procedure is done every year, but there’s no data to support such frequency.

In sum, there’s every reason to think the rate of false positives in screening mammography is falling and that costs from errors will diminish in the future.

Many, if not all, of the costs attributed to false positives will be reduced by advances in technology, better training of radiologists, and education of physicians (oncologists, surgeons, primary care physicians) who recommend the procedure and make decisions based on the results.

Related Posts:

It’s Not About the Money

A recent NEJM article, Lessons from the Mammography Wars, opens with a quote from the 18th Century philosopher-economist Adam Smith:

People of the same trade seldom meet together . . . [without] the conversation end[ing] in a conspiracy against the public.

The Wealth of Nations (1776)

The opinion piece, published on Sept. 8, considers the controversy that surrounded last year’s U.S. Preventive Service Task Force recommendations for screening mammography in the context of broader issues. It contains some serious accusations:

“Advocates of breast-cancer screening, particularly breast radiologists, immediately took action, denouncing the panel’s statements as government rationing, suggesting that the panel members had ignored the medical evidence…

Ultimately, what’s incriminated is the self-interest of doctors:

“…Although it is true that individual medical providers care deeply about their patients, the guild of health care professionals — including their specialty societies — has a primary responsibility to promote its members’ interests. Now, self-interest is not in itself a bad thing; indeed, it is a force for productivity and efficiency in a well-functioning market. But it is a fool’s dream to expect the guild…to compete on true value when the opportunity to inflate perceived value is readily available.

OK, readers, so how do I recommend we resolve this thorny issue?

Physicians should be on salaries, set in such a way that their earnings aren’t based on the procedures they order or perform. For example, I don’t think a surgeon should earn more money if he or she performs a greater number of biopsies or mastectomies, as opposed to spending time examining and speaking with patients who are contemplating those procedures. I don’t think an oncologist should earn more money by ordering or administering more infusions of chemotherapy, as opposed to recommending a “wait and watch” type approach, palliative care, or giving pills that are effective in some malignancies.

If physicians’ potential profit motive clouds the mammography debate, as the authors contend, that doesn’t mean that mammography is ineffective. Rather it signifies that doctors and scientists should analyze data and make clinical decisions in the absence of financial or other conflicts of interest.

If patients could know that their doctors don’t have a vested interest in mammograms, or any other test or procedure they might prescribe, they’d trust them more. It’s for this reason, above all, that we should disconnect doctors’ decisions from economic gain.

—-

Related Posts:

Pink Glove Dance, The Sequel

They’re back!

Now, with over 4,000 participants, slicker than last year:

Pink Glove Dance: The Sequel

—-

“…like a gift with a ribbon around it” ?

I’m not so sure about these lyrics. The featured song, “You Won’t Dance Alone” by a band called “The Best Day Ever,” is available on iTunes and Amazon.com.

This year’s video’s stated goal is to raise awareness, and money for breast cancer care and research through donations to the National Breast Cancer Foundation. The pinkglovedance.com website is sponsored by Medline Industries, Inc., said on its website to be the nation’s largest privately held manufacturer and distributor of health care products.

I encourage all of my readers to support breast cancer education and research. A local favorite, dear to my family, is the Breast Cancer Research Foundation.

—-

Related Posts:

News, and Thoughts, on Comparative Effectiveness Research

Last week the U.S. Government Accountability Office (GAO) appointed several new members to the board of the Patient-Centered Outcomes Research Institute (PCORI). This group will oversee the application of comparative effectiveness research in health care reform. The panel’s composition was the subject of two posts today by Merrill Goozner, a long-standing observer of health care economics and journalist.

The group may play a key role in forming health care policy in the years ahead. According to the Patient Protection and Affordable Care Act, signed into law in March of this year, the PCORI is slated as a private, non-profit agency. The Institute’s purpose is:

to assist patients, clinicians, purchasers, and policy-makers in making informed health decisions by advancing the quality and relevance of evidence concerning the manner in which diseases, disorders, and other health conditions can effectively and appropriately be prevented, diagnosed, treated, monitored, and managed through research and evidence synthesis that considers variations in patient subpopulations, and the dissemination of research findings with respect to the relative health outcomes, clinical effectiveness, and appropriateness of the medical treatments, services, …”

This is no small task, to say the least.

What is comparative effectiveness research and why does it matter? The idea, basically, is to inform medical decisions with relevant data derived from well-designed clinical trials. This sort of research will provide the foundation for evidence-based medicine (EBM).

Some casual readers of this blog may not appreciate the extent to which I support these endeavors. I do, 100 percent. I say this for several reasons:

First, as a patient: We need to obtain as much information as possible on the best way to treat common and rare medical conditions that arise in humans. Knowing which treatments work, and don’t, ultimately will promote care that’s effective and diminish use of procedures, devices and medications that are ineffective and/or harmful.

Second, as a citizen: We don’t have a surplus of health-care resources to throw around, so to speak, in vain efforts to treat things that can’t be fixed.

As an oncologist, I was not one to give treatments that I didn’t believe would help a patient to feel better and, ideally, get well. Rather, I think the value of chemotherapy and other treatments needs be demonstrated in clinical trials prior to their routine administration, or that their worth be measured otherwise – such as by careful testing of how a drug might work in a subset of patients or even in a particular patient who has a condition like breast cancer and, upon careful monitoring, it is apparent that nothing helps except Avastin in her case.

Third, as a doctor: I would never want to cause avoidable harm to a patient. If a drug, device or procedure is ineffective or damaging, we shouldn’t often prescribe that, if ever. The best way to help patients get well, and harm them less, is by knowing what works and what doesn’t in most cases and by knowing the limitations of aggregated, statistically-valid data applied to individual patients.

Cost analyses are needed, too, but that’s a somewhat separate issue.

I wish the panel good luck in their important work ahead.

Related Posts:

What’s Missing in the Recent Mammography Value Study

This week’s New England Journal of Medicine includes an article*on the potential benefit of breast cancer (BC) screening by mammography. The paper, authored by a group of epidemiologists and biostatisticians from the Cancer Registry of Norway, Harvard University, the Dana Farber Cancer Institute and Stockholm’s Karolinska Institutet, suggests that mammography has a relatively small influence on survival.

The work, along with an accompanying editorial* by H. Gilbert Welch, M.D., M.P.H., got front-page attention in the Times and elsewhere. My friends want to know why this particular research study was featured and, really, what it showed.

So here’s my take –

The analysis:

The researchers studied chronological trends in BC diagnosis and mortality in Norway. To see if mammography had an effect, they divided the country into two groups, “screening” and “non-screening,” based on when a state-sponsored mammography screening program was implemented in each of 19 counties there. The national plan, which required that each region establish a centralized, multidisciplinary BC care team before participation, began in 4 counties in 1996 and gradually expanded to include all of Norway by 2005. According to the authors, all Norwegian women between the ages of 50 and 69 years have been asked to participate in screening mammography since 2005; 77% have done so; Norway’s nationwide cancer registry is nearly 100% complete.

They evaluated a total of 40,075 women (“subjects”) who received a diagnosis of BC between 1986 and 2005.

Major findings:

For women between the ages of 50 and 69, BC-associated mortality** fell from 25.3 to 18.1 in counties where a government-sponsored mammography program was implemented early on, and from 26.0 to 21.2 in counties where mammography was not covered, over a similar time frame. Because BC-associated mortality declined in all regions of Norway, regardless of whether mammography was offered, the authors conclude that screening can’t account for all the reduction in mortality.

By their calculations, mammography accounts for roughly 10 percent of the enhanced survival. (This finding was not statistically significant.)

The authors suggest that recent progress in BC survival – which in their study improved significantly in all regions of Norway – comes, for the most part, from better care and treatments.

What’s wrong with the paper? I see several key flaws:

1.  The average follow-up is only 2.2 years after diagnosis, with a maximum follow-up of 8.9 years (“Results,” p. 1206). This is far too short a follow-up interval to measure the benefit of mammography or any sort of intervention in women with breast cancer. When BC recurs it’s often after several years and, occasionally, decades later.

2. Among women under the age of 50 there was a slight increase in BC-mortality noted: A non-significant relative increase in mortality, of 4%, after the introduction of the screening program for older women (p. 1207, Table 1). This worrisome finding is not adequately addressed by the authors; one might wonder – did fewer women in their forties go for mammograms after 1996, since they were only recommended and covered for older women? (My concern is that reduced screening, now, in younger women might lead to an increase in BC mortality.)

3. Digital mammography was not evaluated in this study.

4. The authors detected the greatest benefit of screening among women with Stage II BC; there was a “marked” 29% reduction in mortality relative to the historical counterparts for that group who were screened, as compared to only a 7% reduction in mortality for women with Stage II tumors in counties where screening was not available over the same historical interval (p. 1207). This observation suggests that mammography screening is most life-saving for women with Stage II tumors. As an oncologist, I find this highly-plausible; the purpose of mammography is to identify tumors in early stage and spare women morbidity and mortality associated from advanced disease.

5. There’s no mention of the absolute number of lives saved by the procedure according to the authors’ calculations, but I think this is an important number to keep in mind when we assess the procedure’s value. If the paper’s conclusion is true – that mammography reduces BC-associated deaths by just 10 percent – then in Norway, with a total population of 4.8 million and where some 4,791 women in the study died (p. 1206), these results support that mammography spared approximately 480 lives in those 20 years.

My spin:

Mortality in the U.S. from breast cancer has declined by roughly a third since the implementation of wide-spread mammography screening. Here, where some 45,000 women die each year of BC, we’d save 4500 lives per year if the added value of mammography is just 10 percent, as suggested by the new study. If the benefit of screening mammography is higher – in the range of 45 percent, as was supported by a 2007 paper, also published in the NEJM – then the value would exceed 20,000 women’s lives per year. If the benefit is only 25 percent in terms of reduced mortality, that would result in over 11,000 lives saved, per year in the U.S.

As for the editorial, first I’ll say that the opening statement – that “no screening test has ever been more carefully studied than screening mammography,” is misleading. While this was, indeed a well-organized and careful study, among other issues it was far too short in patients’ follow-up to measure the impact of mammography on BC survival. The Annals papers, which caused so much controversy last year, relied heavily on old data and did not at attempt to examine the efficacy of digital mammograms.

What’s needed, still, for public health policy in the U.S. is evidence regarding the long-term outcomes after digital mammography performed in FDA-regulated, modern facilities by skilled, board-certified radiologists applied every other year in women who are over the age of 40 in the context of modern, adjuvant treatments and current pathology methods.

—-

As for the analysis by G. Kolata in the Times, where she wrote:

“…it indicates that improved treatments with hormonal therapy and other targeted drugs may have, in a way, washed out most of mammography’s benefits by making it less important to find cancers when they are too small to feel.”

I’d say the opposite is true:

It’s precisely because there are effective treatments for early-stage disease that it’s worth finding breast cancer by mammography. Otherwise, what would be the point?

Metastatic breast cancer is quite costly to treat and, even with some available targeted therapies, remains incurable.  Despite so many advances in treating early-stage BC, the survival rate at 5 years is under 25 percent for women with Stage IV disease.

—-

*abstract available, otherwise by subscription

**mortality rates: per 100,000 person-years

Related Posts:

Stepping Back, and Thinking Forward to October

Before jumping into the raring-to-go, already re-sparked debate on the value of breast cancer screening, or not, I thought I’d step back today and consider the background of what I won’t call the “mammography wars.”

So here’s the first point on this subject: happily, it’s not a war. This is about medical progress, or lack of progress, and what may or may not spare women morbidity (what doctors and stats types call illness) and mortality (death). This is not a battle by any sane measure.

The discussion should be civil and fair, ideally without assumptions and bias. (Of course it’s impossible for me to be unbiased on this subject – I am a trained oncologist, I’ve treated women with breast cancer who responded to chemotherapy drugs and witnessed their subsequent well-being, and I’ve seen women with metastatic breast cancer who died with disease eating through their bones and brains. And I had breast cancer, and was lucky that mine was found in an early stage. So far am well in that respect, eight years ago next month and counting -)

So I have bias, yes, but my second point is that we all do. Every journalist does, whether or not they’re up front about their life-experiences and community background, and so does every medical researcher and public health official who authors a paper and is not a robot. We delineate stories, analyze and represent data based on points we want to make and, in some medical journals – even for some trials that aren’t funded by drug companies – publish to serve academic, career or even what we think are idealistic, cost-saving, pain or procedure-sparing often high-minded goals.

Data can be very tricky to sort out and among.

The problems with mammography studies, as partly-detailed in the current NEJM paper, is that it’s considered impossible to do a controlled, randomized trial in which half of the women undergo screening and half don’t. (I might argue this assumption; now, that so many women and doctors are questioning the merit of mammography, maybe we could do a truly randomized trial involving a few thousand women in the U.S.) This issue relates to the important topic of ethics and real-life dilemmas oncologists face when they set up studies for evaluation or treatment of humans some of whom are likely to die.

But before we delve into the details of this study, and next month (October), what I hope is that we’ll keep some facts in mind:

The National Cancer Institute estimates that 40,000 women will die of breast cancer this year in the U.S. There’s been dramatic progress in how we manage this once-dreaded condition: prior to 1926, fewer than 20 percent of women survived for 5 years after diagnosis. By 1950, the overall 5-year survival rate among Caucasian women with breast cancer was 60 percent. Now, the overall 5-year survival rate is around 91 percent.

A question central to today’s discussion – which does at least acknowledge the decline in breast cancer mortality – is the extent to which mammography is responsible for this trend, as opposed to other factors such as increased awareness about cancer, better cancer treatments and other variables.

What concerns me is the tone of the debate on mammography, that it shouldn’t subtly or not-so-subtly, denigrate women’s valid concerns about their health. The quote with which today’s front-page story ends is this: “I think we have to respect what women want to do.” The way the story is framed insinuates that a decision to undergo mammography is based on something other than reason.

To be clear: this is not about what women want. It’s not about emotional turmoil, comfort, stress or people’s feelings. It’s about the efficacy of state-of-the art mammograms and whether or not those, when taken by skilled, well-trained radiologists in carefully-regulated modern facilities save women’s lives, at a reasonable cost (however we might calculate that) and reduce illness by detecting breast cancer in its early stages.

Related Posts:

Wednesday Web Sightings

(Snakes on the Net, and OR games!)

Today ML read in the WSJ that a 3-foot long yellow corn snake popped up in a 19th floor Bronx apartment yesterday. It seems the resident was so surprised to find the snake in his toilet, and shaken, that he dialed 911 more than once.

a corn snake (Wikimedia)

Medical tidbit: according to the Journal, corn snakes are not usually harmful to humans.

The above would have been the most curious Internet medical finding of the day, except for an ad I found on a blog for a short adventure called OR Games. (The video is sponsored by Kimberly-Clark, a medical supply company.)

Serious stuff tomorrow!

Related Posts:

No More Clipboards

“This kind of aid to decisions, spread across tens of thousands of decisions every day, leads to much, much lower costs with no intrusion on clinical autonomy”

- David Blumenthal, M.D., M.P.P., National Coordinator for Health Information Technology, U.S. Department of Health & Human Services, speaking at Weill-Cornell Medical College, Sept. 15, 2010

Last week I had the opportunity to hear Dr. David Blumenthal, head honcho for national health IT development, give a presentation on electronic health records (EHR), the future of health information technology (IT) and meaningful use. He spoke at a health policy colloquium at Weill-Cornell Medical College.

The points he made were clear – to begin, that health IT encompasses three areas of value:

1. Electronic Health Records (EHR)

2. Health Information Exchange (HIE)

3. Clinical Decision support (CDS)

Notes on Electronic Health Records:

“What do patients think if you ask them what’s the value of an EHR?” he half-asked the audience, a group of 250 or so, mainly physicians. He answered: “No more clipboards.”

The problem is that doctors are reluctant to take on electronic systems. Prior to the Health Information Technology for Economic and Clinical Health Act (or HITECH Act) of 2009, only 6 percent of physicians used fully-functional EHRs; only 20 percent used any EHR in their practices. Hospitals weren’t doing any better, he said. We’re running far behind our European counterparts in this arena.

“What’s the barrier?” he asked, and answered: “Money is numbers one through eight. Then logistics, technical problems and fear.”

Physicians, and hospitals, are concerned about buying into systems that will become obsolete. “And it’s a psychological issue,” he added. “That comes in whenever you’re asking people to change.”

The benefit of EHRs will be more efficient care and better health outcomes. As things stand, the quality of a nurse or doctor’s care is limited by the information they have about the patient in front of them. If providers have access to a patient’s compete record – say a cardiologist who’s evaluating a patient with notes provided by the primary care provider (PCP) – that’ll save both parties (patient and doctor) time, reduce duplication of tests and facilitate the coordination of care.

But the real value in EHR, he suggests, is in the system’s potential output of information: “Cutting down the rate of dissemination of conventional information from 17 years down to months to weeks.” What he means, as I understand it, is that by having current findings – published data and recommendations – linked to a patient’s electronic chart and diagnostic codes, the doctors will see and (hopefully) read new materials having to do with their patients’ medical conditions.

Notes on Health Information Exchange:

This component of health IT relies on having a common language. “If physicians don’t speak the same language, their computers won’t either.” He emphasized the need for standard terms using this example: there are many ways to say “high blood pressure” in English; for a health information exchange to be functional, we need to agree on one term and use it.

“Information exchange is a team sport,” he considered further. “It’s a challenge, and we’re not always good at voluntary collaboration.”

Next, he listed some HIE essentials:

- a vocabulary and statistics (I’m not sure why these were paired as he spoke);

- a way of packaging information that it can travel on the Internet;

- electronic directories, so information goes where it should;

- protocols for delivery;

- security in transit.

What will it take to get physicians on board with this kind of plan? “We need economic incentives, trust, and requirements,” he indicated. Mandates might, in the future, involve licensing boards and other accrediting agencies that would test physicians’ competency in HIT. In his view, managing electronic information is an essential skill for doctors to provide effective medical care.

Notes on Clinical Decision Support:

First, he listed the four elements – algorithms, statistics, guidelines and institutional policy – that can be incorporated into CDS to promote evidence-based practice.

Second, CDS allows for Computerized Provider Order Entry (CPOE). This is a huge benefit at several levels, but the surprise for some in the audience is that the process becomes an opportunity for continuing medical education (CME).

(The idea for doctors’ learning as they enter orders, which I think important, is this, and  I’ll provide my own example here: if you’re a physician about to order a drug, like a new antibiotic for a patient who’s got liver failure, the computerized system might have a pop-up message that says “are you sure you want to order that? Antibiotic X is metabolized by the liver…” And so you’d learn that the new antibiotic X is metabolized by the liver, and you might then look over a readily-accessible table of alternative antibiotics.)

“This caught my interest because it doesn’t diminish physicians’ autonomy,” he said. It just enables them to make decisions for their patients in the context of additional, current information. “The end goal is not to adopt technology, but to improve care.”

Notes on Meaningful Use:

He listed “five pillars of meaningful use”:

1. patient and family enlightenment

2. coordinated care

3. quality, safety and efficiency

4. privacy and security

5. improved public and population health

Then the discussion turned to some big money questions – including a critical issue having to do with companies competing to develop better EHR systems and physicians’ incentives. If you’d like to read more details about meaningful use, I recommend a recent perspective in the New England Journal of Medicine and a brief in the August issue of Health Affairs, to which Blumenthal referred during his informative talk.

As for me, I’m tired of clipboards. I look forward to expanded health IT and EHRs. In the future, patients will be more informed in their decisions, and doctors will be in theirs. Maybe, sometime…

Related Posts:

Notes on Oliver Sacks, on Prosopagnosia

The New Yorker, Aug 30, 2010

Today while on a train I caught up on some reading, including the August 30 issue of the New Yorker. I learned about prosopagnosia – the inability to recognize faces and some aspects of place/orientation.

The information came through a typically curious article by Dr. Oliver Sacks, a neurologist, author – of The Man Who Mistook His Wife for a Hat fame – and New Yorker, whom I met briefly one day in a class at Columbia University. Unabashedly, Sacks details his own mishaps in recognizing people he’s met and finding his way; it’s a life-long, inherited affliction that requires he remember individuals by things other than how they look. Half-generously, the magazine provides non-subscribers a detailed abstract of Face Blind that’s far less charming than the original piece but relatively rich in the medical details:

Severe congenital prosopagnosia is estimated to affect two to two and a half per cent of the population—six to eight million people in the United States alone.

Hard to believe? yes. Interesting? yes. Will Dr. Sacks recognize me if we meet again? Unlikely -

Related Posts:

Hot Topics: This Week’s Med-Blog Rounds at Bedside Manner

I’m may be out of sync on this, but it’s not too late to visit this week’s med-blog Grand Rounds posted yesterday at Bedside Manner.

The theme is medical communication – appropriate to a blog authored by Julie Rosen, Executive Director of Boston’s Kenneth B. Schwartz Center. There are some nice photos, linked entries, and a final connection to a collection of poems (and prize) at The Examining Room of Dr. Charles.

As a doctor-patient, or patient-doctor, I’m fond of Bedside Manner – a relatively new blog, and of the Schwartz Center goals in general – to help patients and doctors communicate effectively, which should improve the quality of patients’ experiences in the health care system regardless of the usual measured outcomes.

Definitely worth a look!

Related Posts:

newsletter software
Get Adobe Flash player