- Dermatologist Magnus Lynch pitted two AI chatbots against a simple program on 200 skin lesions.
- The program got the first diagnosis right 62% of the time, the chatbots just over 40%.
- At its preset threshold the program still cleared 18 of 30 melanomas as safe.
In short: dermatologist Magnus Lynch (Guy’s Hospital, King’s College London) tested two large AI chatbots, GPT-6 Astra and Claude Fable 5.1, on 200 biopsy-proven skin lesions. He compared them with a simple image-recognition program, a convolutional network (the Inception v3 model). He published the post on 25 October 2026 on his lab’s website. It is not a peer-reviewed paper.
What was tested
- 200 lesions: 100 that must not be sent home without further checks (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, 20 actinic keratoses) and 100 harmless ones (50 naevi, 50 seborrhoeic keratoses).
- Each case had a clinical photo and a dermoscopic image. The chatbots picked one of six diagnoses and said whether the lesion could safely be left alone.
- The program was trained on 68,011 images from the ISIC Archive. The test lesions and related images were excluded from training.
Results
- First diagnosis correct: GPT-6 Astra 84 of 200 (42%), Claude Fable 5.1 85 of 200 (42%), the program 125 of 200 (62%).
- Melanoma as first choice (out of 30): 8, 11 and 13.
- Harmless lesions judged safe (out of 100): 32, 22 and 76. The chatbots sent almost everything onward, so they were cautious but not very helpful.
- The program has a preset decision threshold, and at it the error went the other way: it judged 18 of 30 melanomas safe (chatbots: 3 and 4).
In a separate test using clinical photos only, both chatbots advised seeing a doctor in 99 and 98 of 100 melanoma cases, but gave similar advice for most harmless lesions.
How to read this
The author lists the limits himself: a small sample (only 30 melanomas), each chatbot run once, every lesion already biopsied and so harder than typical, and the chatbots may have seen the public dataset during training (the program had not). The program is a simple baseline, not a state-of-the-art model. He also notes that one of the tested chatbots is made by Anthropic and that he used Claude Code for the analysis.
This is not medical advice: show a worrying mole to a doctor. I write more about AI research in the research and papers hub.