A dermatologist tested GPT-6 Astra and Claude on 200 skin spots. A 2015-design model beat them, and set to the same caution, it also sent fewer melanomas home.

On 25 October, a London dermatologist published a test that every person who has ever sent a photo of a mole to ChatGPT should read. Dr Magnus Lynch, a consultant dermatologist at Guy's Hospital, took 200 skin spots where a biopsy had already confirmed the answer. He showed them to OpenAI's GPT-6 Astra, Anthropic's Claude Fable 5.1, and a small image model built on a design Google published in 2015. The old model named the right diagnosis 125 times out of 200. Both chatbots managed it only 84 and 85 times, which is 42 percent. Lynch tested the GPT-6 Astra model directly, not the ChatGPT app, which adds its own instructions and gives many users a different model.

The post went to Hacker News, where it gathered 668 points and 273 comments, and the headline everyone shared was simple. A ten-year-old network beat the newest frontier AI at spotting skin cancer. I read the post, the longer version Lynch published two weeks earlier, the model's own documentation, and the prompts and replies he made public. The headline holds up better than many of the replies to it. The one number people used to defend the chatbots doesn't say what they think it says. And the way the chatbots fail matters most if your mole is the one being checked.

How the test worked

The 200 spots came from MILK10k, a public set of skin images released in 2025 by the International Skin Imaging Collaboration. Each one has two photos, a normal close-up and a dermoscopic image taken through a skin microscope. Half of them should never be sent home without treatment or follow-up. That half included 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas and 20 actinic keratoses, which are rough patches that can turn into cancer. The other half was harmless, with 50 ordinary moles and 50 seborrhoeic keratoses, the waxy brown growths many older people get.

Every spot had been cut out and checked under a microscope, so these were not easy cases. As Lynch puts it, a mole that someone decided to cut out is one that worried a clinician.

Both chatbots got the two photos and nothing else, with no age, no body part and no history. They had to pick one of six diagnoses, give a confidence score and a second choice, and say whether the spot was safe to send home. The old model is Inception v3. It is the same design that a famous 2017 paper in Nature used to show that AI could classify skin cancer as well as dermatologists. Lynch fine-tuned it on 68,011 images from the public ISIC archive in about 20 minutes on a single graphics card. So the story here isn't really that old beats new. It is that a recipe from 2017 still works.

What each one got right

The chart shows that nobody was good at melanoma, the deadliest form of skin cancer. Out of 30 melanomas, GPT-6 Astra named melanoma first for 8, Claude for 11 and the old model for 13. Even the winner missed more than half.

The chatbots also had strange habits. GPT-6 Astra called 75 spots basal cell carcinoma when only 25 were, and that included 14 of the 30 melanomas. Claude called 53 spots actinic keratosis when only 20 were. Their answers weren't random guesses, since with six choices a guess would be right about 17 percent of the time. But each model had a favourite answer that it reached for again and again.

The number people used to defend the chatbots

Here is the number the headline leaves out. The test also asked which spots were safe to send home. At its preset setting, the old model would have sent home 18 of the 30 melanomas, while GPT-6 Astra sent home 3 and Claude sent home 4. Many readers took this as proof that the chatbots were safer. A Hacker News commenter called speerer wrote that the chatbots were better at avoiding false negatives. In their view, that made a specialised old network one that "is a worse public facing tool than a general frontier LLM".

I read the table differently, because the chatbots kept almost everything, and they sent home only 32 and 22 of the 100 harmless spots. At its preset setting the old model was far bolder, since it sent home 76 harmless spots. So the two were never set to the same level of caution.

The table also gives a second, "matched" setting for the old model. At that setting it kept 98 or 99 of the 100 dangerous spots and sent home only 2 or 1 melanomas, which is fewer than either chatbot. The post doesn't spell out what was matched. But the numbers only add up one way, which is that the old model was set to send home about as many spots as each chatbot did. Set to the same caution, the old model was the safer one too. The 18 is what you get from a cut-off nobody tuned for this job.

There are two warnings to keep in mind here. Lynch hasn't published the old model's answer for each case, so these figures can't be checked one by one. And the model's own page calls its cut-offs research settings, not a safe line for real patients. One AI news summary said the old model "made fewer misses" than the chatbots. That is true for naming the diagnosis and true at the matched setting, but not at the preset one.

Fluent, confident and wrong

The scariest part of this test isn't the 42 percent. It is how the wrong answers sound. Lynch's first example is an invasive melanoma on a man's ear. Both chatbots called it a seborrhoeic keratosis, which is harmless, and both said it was safe to send home. The old model called it a melanoma. In his earlier write-up, Lynch says that in his opinion no human dermatologist would miss it.

GPT-6 Astra didn't just get it wrong. It described the melanoma in proper medical words as a "raised brown lesion with a lobulated cerebriform surface, fissures and ridges, and small keratin-like pale dots". That is how a textbook describes a harmless growth. A patient or a busy general doctor reading that would find it very hard to tell it is wrong.

The replies Lynch published show more of this. GPT-6 Astra gave 90 percent or higher confidence on 10 answers that were wrong, while Claude did that on none. GPT-6 Astra's confidence averaged 82 when it was right and 79 when it was wrong, so its confidence told you almost nothing. It also sent home one melanoma that was 1 to 2 millimetres thick, with 89 percent confidence that it was harmless. On three of the four melanomas Claude sent home, it listed melanoma as its second choice and still said the spot was safe to send home.

Asked like a patient, they told everyone to see a doctor

Lynch also asked the chatbots the way a worried person would, with one photo and the question of whether they needed to see a doctor. For a further 100 melanomas, both chatbots advised seeing a doctor in almost every case, 99 times for GPT-6 Astra and 98 for Claude. That sounds reassuring until you look closer. They named melanoma as a possibility in only 60 and 44 of those replies. In 14 of Claude's 100 melanoma replies, a grading model judged the advice to be only a routine or optional check. One of them said to see a doctor at "your next routine appointment". In one reply, Claude said the simplest explanation for a bleeding melanoma was "a shaving nick or abrasion". It told the person to see a doctor only if it hadn't healed in three to four weeks. In another case, both chatbots took a small melanoma for an attached tick.

The chatbots sent harmless spots to a doctor too. Shown two photos of each of the 100 harmless spots, GPT-6 Astra raised skin cancer for all 100 and Claude for 79. So the safety came from telling nearly everyone to see a doctor, not from knowing who needed one. Lynch made the same point about the clinician test, where the models were safe mainly because they kept nearly every spot, which doesn't help anyone decide anything.

Where the test is weak

This is a blog post by one doctor, not a peer-reviewed paper, and the sample is small. Lynch says himself that 200 spots are enough to show a pattern but not to rank close results. He adds that 30 melanomas leave wide uncertainty on the number that matters most. Almost half of those melanomas, 14 of 30, were in situ, meaning very early and very hard to spot. Each chatbot ran once per case at default settings, with very little thinking time. GPT-6 Astra used about 88 reasoning tokens per case on average, and neither model got the patient's age or body site, which real doctors always use.

The old model also had a home advantage. Its training images came from the same archive as the test set, including other images from the same MILK study team, though the 200 test spots themselves were removed. A Hacker News commenter called vikramkr called it a "home field advantage". They also argued that frontier models should get image tools in an agent loop instead of being asked for a one-shot answer. Lynch also says openly that Claude Code helped him with the analysis and the first drafts. The post says the old model scored 0.96 on ordinary archive images at sorting spots to keep from spots to send home. The model's own page gives 0.944 and 0.934 for a slightly different task, telling cancer from non-cancer, and never lists 0.96. None of this changes the main finding that chatbots can't be trusted with this, but it does mean the old model's win needs checking by someone else.

What the companies say, and what people do anyway

OpenAI and Anthropic both say medical diagnosis with their AI needs a professional involved. OpenAI's usage policy bars "tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional". Anthropic's policy counts medical diagnosis as a high-risk use and says a qualified professional must review the answer. Both companies have updated these rules, and the new versions take effect in November. When OpenAI launched ChatGPT Health in January, it said the product "is not intended for diagnosis or treatment."

People use it for diagnosis anyway. OpenAI says over 230 million people ask ChatGPT health questions every week. A Pew Research Center report in August found that 34 percent of American adults use AI chatbots for health. A quarter of all American adults use them to work out what their symptoms mean. Nobody reads a usage policy before uploading a photo of a mole at midnight.

What people are saying

Many Hacker News commenters said this was obvious. "And a calculator outperforms LLMs at multiplication and division," wrote MitPitt. Others said the reminder was still needed. A researcher called jszymborski wrote that colleagues increasingly ask "why bother training new models when you can just ask an LLM?", whatever the task. One commenter, JPLeRouzic, said two doctors had looked at a wound and said nothing about it, and it turned out to be a carcinoma. In their view ChatGPT would have flagged it years earlier. Another, gchamonlive, replied that ChatGPT could just as easily have had "10 other benign anomalies" cut out along with it. A third, kaikai, put the patient's side simply: "Better to have ten small false positives than a single miss."

On Lemmy, a user called brucethemoose wasn't surprised at all, since those old models "are tuned specifically for the task." A related point came up last year, when a London hospital started using an AI skin tool from Skin Analytics. On Hacker News then, a commenter called degamad wrote that the real question isn't whether you have skin cancer but "can you safely go home without seeing a doctor?" That is exactly the question this test asked, and the chatbots answered it badly in clinician mode and over-cautiously in patient mode.

"Chatbots were never meant for this, and they were the safer ones"

This is the strongest defence of the chatbots. Their makers say not to use them for diagnosis, and they weren't given the context a doctor would have. And when they had to decide who to send home, they sent home far fewer melanomas than the specialised model did at its preset setting. For a tool that millions of people use without a doctor, sending everyone to the clinic is the safe way to fail.

I don't think that defence holds up, for three reasons. First, the chatbots only look safer next to an old model set far more boldly than they were. At the same caution, the old model sent home fewer melanomas than either of them. Second, the safety was a reflex, not a skill. A chatbot that tells nearly everyone to see a doctor, and names melanoma for only 60 or 44 of 100 real melanomas, is not telling you much. When it does name something, it can be confidently and fluently wrong. Third, in the clinician setup both chatbots sent real melanomas home with confidence scores of 65 to 89 out of 100. GPT-6 Astra did it for one that was 1 to 2 millimetres thick. A general doctor using a chatbot as a second opinion is exactly the person who would trust that answer.

The honest summary is that neither the chatbots nor Lynch's research model is a skin cancer tool for patients. Proper tools exist, they are tested in trials, and they are regulated, such as Skin Analytics' DERM. NHS clinics use it with trained staff taking the photos, and it is not an app you download and use on your own.

The India angle

Skin cancer is much rarer in India than in Europe or Australia. Data from India's cancer registries, analysed by ICMR researchers, put melanoma at its highest in North India, at 1.62 cases per 100,000 men a year. Rare doesn't mean safe, though, because in India melanoma often shows up on the feet, palms or nails, where people rarely look. And the Indian problem with AI like this isn't how common skin cancer is. It is who the AI was built for.

The MILK10k images came from clinics in Austria, Turkey, Australia, the United States and North Macedonia, and its organisers say most patients had skin tones in the middle range. The old model's own page lists "few images of darker skin types" as a known limit. In people with darker skin, melanoma more often starts on the soles of the feet, the palms and under the nails. None of the 30 melanomas in Lynch's test was marked as being in those places. A 2022 study in Science Advances found that leading skin AI models performed clearly worse on dark skin, and that fine-tuning them on diverse images closed the gap.

At the same time, India has very few dermatologists, about 0.5 to 1 for every 100,000 people according to the Skin Observatory's global map. And Indians use AI for health more than almost anyone. A BCG survey this year found that 85 percent of Indian consumers use AI tools for their health, against 50 percent in the US. That combination of few specialists, heavy use of AI for health and training data that doesn't look like Indian skin is the real risk.

The good news is that India's drug regulator finalised guidance in July that requires makers of medical AI software to disclose how their models perform across population groups. Researchers at IIT Hyderabad and medical colleges in Karnataka have also built DermaCon-IN, with 5,450 photos from 3,002 Indian patients. It fills a gap for Indian skin, though it covers common skin problems more than cancer. General chatbots, though, aren't registered medical devices in India, and we found no Indian rule that stops them from answering a photo of a mole.

Who I'm calling out

My callout is for OpenAI, because ChatGPT gets more health questions than any doctor alive. When the test framed GPT-6 Astra as a dermatologist's assistant, it was 90 percent or more sure on ten wrong answers, and Claude did that on none. Doctors are exactly the people who would trust a number like that. When a patient asked, GPT-6 Astra did tell almost everyone to see a doctor, and it gave no percentages. OpenAI should make sure its model never sounds that sure about a skin spot, whoever is asking.

My prediction is that general chatbots won't become approved skin cancer tools any time soon. By 30 June 2027, none of ChatGPT, Claude or Gemini will be cleared by the US FDA or carry a European medical-device mark for diagnosing skin cancer from photos. If I am wrong, I will say so here.

What to do if a mole worries you

Don't ask a chatbot to decide. See a doctor, ideally a dermatologist, if a spot is new, growing, changing shape or colour, bleeding, itching or not healing. Doctors often use the ABCDE rule. It looks for an Asymmetrical shape, an uneven Border, more than one Colour, a Diameter bigger than a pencil eraser, and Evolving size or look.

ABCDE doesn't catch every melanoma, though. Also see a doctor for a firm lump that keeps growing or a pink or red spot that won't go away. The same goes for a new dark line in a nail, or a dark patch on the sole that isn't a bruise. If you have brown skin, check the soles of your feet, your palms and your nails as well. A chatbot can help you understand what a doctor told you, but it can't tell you whether to go.

Questions people are asking

Can ChatGPT detect skin cancer from a photo? It can't do it reliably. In Lynch's test of 200 biopsy-proven spots, GPT-6 Astra and Claude Fable 5.1 each named the right diagnosis 42 percent of the time. They named melanoma first for only 8 and 11 of the 30 melanomas.

Is AI better than a dermatologist at spotting skin cancer? Specialised, regulated tools tested in clinical trials can match doctors on some tasks, and some are already used inside NHS clinics with trained staff. General chatbots aren't in that group.

Did the old model really beat the chatbots? At naming the diagnosis it did, with 125 of 200 against 84 and 85. At its preset threshold it would have sent home 18 of 30 melanomas, against 3 and 4 for the chatbots. Set to send home about as many spots as each chatbot, it sent home only 2 or 1.

Does AI skin cancer detection work on brown skin? Most skin image datasets come mainly from lighter-skinned patients, and the model in this test lists "few images of darker skin types" as a known limit. In India, melanoma more often starts on the soles, palms and nails, places people rarely check.

What should I do if I'm worried about a mole? See a doctor if a spot is new, changing, bleeding, itching or not healing, or if you find a growing lump or a new dark line in a nail. Don't rely on a chatbot's "safe to leave alone".