Apparently, computer-based diagnostic algorithms provided cheaply via a smartphone aren’t reliable. Who would have guessed? There’s a slew of new apps available that allow you to take a picture of your weird mole or mysterious skin lesion, and they’ll then scan it and tell you whether you’ve got melanoma or not. You should be wary. When real doctors actually test their competence, the dermatology apps fail miserably.
Dermatologists are less than thrilled. In fact, they say, the apps are worthless. Writing in JAMA Dermatology, a team of physicians from the University of Pittsburgh put four melanoma apps to the test against 188 clinical images—pictures they’d taken of patients’ skin lesions and later determined, via biopsy, to be malignant or benign. How would a machine stack up against a board-certified dermatologist?
Not so well. Of the three auto-diagnosing apps, the best program missed malignant growths 30 percent of the time; a second performed only slightly better than flipping a coin.
One app, instead of using an algorithm, simply forwarded the photo to an accredited dermatologist, who responded with his considered opinion 24 hours later. At five dollars per lesion, this was the most expensive program, though the e-doctor misdiagnosed just one in 53 melanomas.
They’ve got testimonials from users praising the results, which makes me wonder…if your phone told you you didn’t have cancer, how the hell would you know if it was right or not?
It’s the oldest principle of quackery: tell the patient what they want to hear, and they’ll reward you with agreement.
People with cancer will be misdiagnosed — and it will probably go the other way, too. Wait for all the “I cured my cancer with elderberry wine, baking soda, and positive thinking” testimonials.
I think I’d get a second opinion from a genuine doctor.
“Doctor, my Blackberry says it’s cancer, but my iPhone thinks I have lupus. I’m worried.
Should I buy a Galaxy?”
The one that sends it to a real doctor actually might be useful, especially for poor folks without insurance. Rather than drop $200+ just to walk in the door and be told that it’s fine, you can spend $5 and have a pretty good shot at an accurate diagnosis. If you sent 3 pics in, a week apart, that’s only $15, and assuming the doc is blind to the origins of the pictures, a pretty low error rate for guiding you to make an appointment or not.
Yeah, the real doctor one sounds perfect. It’s an actual expert opinion for $5 and 5 minutes of your time.
On the other hand, taking a picture of a funny spot and sending to your doctor could be useful, as can taking pictures of your skin every six months and comparing them.
Perhaps the false positives are the only way that it can honestly be claimed that cell phones are giving people cancer, albeit virtually.
I’m just amazed that we have moved so far into the future that people are willing to trust phone-apps on life-or-death stuff, when autocorrect is still a major joke. People seem to treat their phone (or computer for that matter) as an extra limb. It’s weird. I mean, I love my computer, I love that the internet allows me to read stuff from all over at a click. But it’s still just a tool.
“Siri, does this look infected to you?”
“I knew my film school degree would come in handy! My doctor wants me to come in and do a biopic. Also, these weird moles are just some form of canker, like the kind you get in your mouth, I guess.”
“I knew my film school degree would come in handy! My doctor wants me to come in and do a biopic. Also, these weird moles are just some form of canker, like the kind you get in your mouth, I guess.”
Ahahhaah.. Thanks. You made my midnight.
“Siri, does this look infected to you?”
Siriously ™.
Anthony wins the thread. +10 internets to you.
All part of the mobile app gold rush. At $1 per download, a person can make a lot of money for even the dumbest of applications. They can make even more, if the app collects personal information and sends it back to be data mined or exploited. Many of the free or lower cost apps do that, without telling the user or burying the fact in the license agreement.
Too many people have no clue about internet security. User information is the new gold; I wouldn’t be surprised if the users of these skin cancer apps get targeted advertisements about sun screens, cancer clinics, etc.
Of course, the big boys like Google provide easy means for you to search and store your data in the cloud, while mining your data and searches for the benefit of their customers (advertisers).
As an extension to the above, imagine insurance companies paying these app providers to clue them in on people who have or might have skin cancer. Although they now (theoretically) can not deny coverage based on prior conditions, they can sure make signing up with them harder.
Anthony K:
What diagnosis does one get from a Galaxy?
{I was just thinking today that I hadn’t seen a post from you in a little while. Good to see you buddy.}
“What diagnosis does one get from a Galaxy?”
That you should’ve brought a towel?
@4 mokele: That assumes that all three diagnoses are independent, which likely isn’t the case in reality. If the doctor misses a certain malignant mole, it could either be due to something like accidentally missing a feature, or it could be that this mole is malignant but happens to look a lot like a benign mole. In the latter case, the doctor would probably miss it every time.
Of course, without socialized healthcare in the US, I still agree that this one app (but not the automated ones) is probably a good thing for poorer people. Even if it doesn’t catch 100% of the malignant moles, a 98% hit rate is still pretty damn good for only $5.
I’d like to see the accuracy of the real dermatologist diagnosing the images with photos by taken with a phone by actual lay people. I doubt they’d be as accurate as they were with the nice images they got with the clinical photos the authors in the JAMA article have been sending.
Yet even the app with the real doctor behind the curtain missed one of the melanomas, but I guess nothing’s perfect. A dermatologist in the room probably misses one now and again.
Faith in technology is as misguided as faith in god, the pope, Billy Graham…or living doctors. I wouldn’t trust any phone app to diagnose my melanomas at this stage of the technology, any more than I would trust a map app or automobile navigation device to get me through a remote location.
“Faith in technology is as misguided as faith in god, the pope, Billy Graham”.
Nope. Technology is continually being improved.
Hey now, robro, that’s a bit strong. Technology and doctors might make mistakes, but they happen to actually do something, unlike the Pope and Billy Graham, and actually exist, unlike God. You’re comparing something that might help to something that can’t possibly help, and will charge you for the privilege of being lied to.
And these apps make it through Apple’s stringent walled-garden review process? Really Apple, fuck you. Fuck you a whole bunch. This just takes the biscuit.
#9 beat me to it. YOU BASTAAARD!
Being vaguely into mycology – my spouse is an avid mushroom hunter – I can say that no picture truly gives you the detail you can get just looking at things. There’s time and space, as well as translucency that just doesn’t get copied over.
But it should be able to get you a start. I’d expect some to be missed by any sources – but more I’d think you’d want lots of false positives since it’s a first pass cursory tool.
Re: #23 F [nucular nyandrothol] 4 February 2013 at 7:40 pm (UTC -6)
Exactly how does Apple’s policies have to do with the veracity of a program? Are they, with their simple $100 fee once, supposed to test every possible permutation of an application’s accuracy?
Look, they’re looking for the program not to crash their platform, not to steal data or hijack phones. And that’s a near impossible task right there. Now you want them to be an arbiter of what is and isn’t accurate?
This might be a cool idea, if it was actually validated as a legitimate diagnosis tool. Patient goes to the clinic, worried about a weird lesion. Technician takes a digital picture in prescribed lighting conditions, feeds it to the computer, refers patient to a dermatologist if algorithm determines that there is a significant risk. It could be used for screening.
Lots of cool things can be done with vision algorithms. I don’t know the extent of the libraries that are available for app development, but I’m sure a dedicated team could come up with something with much better sensitivity and specificity, given time and a bit more computing power than that of a mobile device (vision algorithms tend to be heavy computationally speaking).
There’s legitimate AI research being done in this, especially since you can use it to test a computer’s ability to do fuzzy logic. But it’s still way off and what little has been done hasn’t been commercialized.
Hang about:
Does that mean the program got it right 70% of the time? I’m not saying that anyone should trust their health to a machine diagnostic with that level of accuracy, but I would say that 70% diagnostic accuracy is pretty impressive for a first-generation application.
Compare it to the current generation of human papilloma virus vaccines, which are only effective against about 70% of HPV strains in the wild. Would you say that we should shun HPV vaccination because it “only” deals with 70% of infection vectors?
Obviously, “email a competent human diagnostician” remains the most sensible option.
timothya1956:
No. It means that if you do have a malignant growth, it will get it right 70% of the time, but it says nothing about its efficacy when you don’t have one.
Poor comparison; if the vaccine is only effective against 70% of the strains, then the appropriate comparison would be against how effective it is if you do have one of those strains.
No, but as noted above, the comparison would be appropriate only if the program were as efficacious against 70% of melanoma types as the vaccine is against 70% of HPV strains.
#23
Bit quick of the mark, aren’t you? I appreciate that the linked article is titled “iPhones make lousy dermatologists”, but of 3 apps mentioned, 2 are also on Google Play. The one I couldn’t find on Google Play is SpotCheck, which is the one that sends the image to a ‘Board Certified Dermatologist’ and also happens not to be available in the UK’s iOS App Store.
You would appear to be a lazy, Apple bashing troll rather than someone interested in the genuine issues raised by this post and the linked article.
John Morales posted this:
Wrong, That isn’t how statistical analysis works. Statistics tells you nothing about the accuracy of an individual prediction. It tells you about the reliability of predictions conducted across all samples conforming to the underlying protocol.
If all diagnostics conducted according to an experimental protocol (say, the one involved in the machine diagnostic offered by the relevant application) yields a result that is significantly out of whack with the odds of tossing a coin, then you should pay attention to the result.
timothya1956, no, not wrong. Look at what you quoted:
Try re-reading my first paragraph above; note the target population for the apps is those who think they may have malignant melanoma, but the sampled population is those who do have malignant melanoma.
(You cannot compare the two directly unless you know the likelihood that someone who thinks they may have melanoma actually does have it)
[addendum]
By sampled population above, I refer to the quote, not to whatever actual sampling was done; I presume they also gathered data on false positives as well as false negatives.
The question is not what the audience believes about the truth of the application’s efficacy (for ferx’s sake, haven’t we studied the placebo effect enough yet already?), but whether or not it actually diagnoses melanoma accurately.
According to the evidence of the application, it does so in 70 per cent of tests. Not brilliant, and not as well as a human diagnostician performs. But good enough to be worth further testing as an automated adjunct to human intelligence (to stop wasting the diagnostician’s time).
The question of whether a person is likely to believe that they have melanoma is (or at least should be) irrelevant to the diagnosic method involved.
timothya1956, this will be my final comment on this issue.
You do realise that you are in effect claiming that the app is correct 100% of the time when the sampled image is not that of a malignant melanoma?
(Think about it)
No. Not claiming 100% accuracy for or against a specific diagnosis. Such a claim would be ridiculous (for statistical reasons).
The application simply claims to accurately diagnose a pathological melonoma in 70% of test cases. If that is true, then we should take the claim seriously for scientific reasons.
[Bah! My intention was good but I am totally wrong in #36, so this will be my final comment]
Given N samples, where X are samples with actual malignancy and Y those without, then N = X + Y.
Consider further that X = X1 + X2, where X1 is the number of erroneous diagnoses and X2 is the number of correct diagnoses, and similarly for Y = Y1 + Y2.
Then, the error rate for X is E(X) = X1 / X, and similarly the error rate for Y is E(Y) = Y1 / Y, and the error rate for N is E(N)= (X1 + Y1) / N.
What I’m claiming is that you are confusing E(X) with E(N), but that you cannot know E(N) without knowing E(Y).
(Note that what you quote provides no information about E(Y))
How do these apps not run afoul of practicing medicine without a license? (Except the one that sends the picture to a dermatologist)
Whilst we’re on the subject of digital quackery, how about
the world’s first downloadable medicine.
Scienceblogs article on the above link here
Apologies for funny looking formatting, preview doesn’t.
26 Chrissa
No, they regularly and arbitrarily refuse to allow apps into their appstore on a variety of grounds having absolutely nothing to do with code issues. So yes, they should not allow permutations of complete BS to be officially hosted, unless they want to keep providing evidence that they are completely hypocritical. (And as if noticing obvious bullshit requires “testing”.)
31 jaggington
You’ll have to explain where I was quick off the mark, lazy, or an Apple-bashing troll. I also don’t give a flying fuck what other app repositories host the code, as none of them are claimed to be walled gardens of purity that Apple does. Perhaps you should follow Apple’s history of denying apps entrance into their holy sanctuary. Maybe you are the lazy, knee-jerk defensive Apple fan? I mean, if you want to sling insults rather than provide any sort of argument against my comment or anything.
I will admit, however, that Apple is in a race to the bottom with Microsoft in terms of my interest in their products or corporations. Google, surprisingly, yet manages to lag behind these two somehow, although Oracle is probably on par with them.
Apple doesn’t claim its App Store provides”… walled gardens of purity…”, you make that claim on their behalf. Apple has many reasons for curating the apps in their App Store but I have never seen anyone before claiming that Apple claims to protect users from their own gullibility and irresponsibility. Of the apps I found on the App Store, all except one described themselves in their blurb as aids for people to better understand the signs of skin cancer and to provide a pictorial record of moles and other skin blemishes to assist with early detection in conjunction with yearly / regular visits to a dermatologist. The exception, Skin Vision, has such a blurb on their website.
Perhaps you should expound on this with examples.
Maybe I am, but you started it, so nyeh nyeh nyeh.
The only apps they haven’t allowed that I know of ran afoul of their monetary licensing – pre-existing contracts – or content warnings – which are pretty clear compared to what the courts have given us.