AI outperformed doctors in Harvard Medical School tests

The results of a recent study show a shift that until recently seemed theoretical: language models are beginning to compete with doctors in the tasks of primary diagnosis. In conditions of scarcity of time and information, for example, in emergency departments, algorithms demonstrate accuracy comparable to or even higher than that of specialists. It is not about a futuristic scenario, but about specific data obtained in clinically similar conditions.

How the capabilities of AI in medicine were tested

The study, published in the journal Science, was conducted by scientists at Harvard Medical School in collaboration with Beth Israel Deaconess Medical Center. Models of the OpenAI company took part in the testing – in particular, o1 and 4o.

The experiment was built as close as possible to real practice. 76 clinical cases of patients who sought emergency care were taken. Diagnostics took place in parallel: on the one hand – therapists, on the other – algorithms.

Further, the results were evaluated by independent specialists who did not know who exactly – a person or a machine – formed this or that diagnosis. Such a “blind” format made it possible to minimize bias.

The key point: the models worked with raw data from electronic medical records – the same that is available to the doctor at the time of the appointment. This excludes the effect of “prepared conditions” and makes the results more indicative.

Where the algorithms were more effective

The most pronounced advantage of AI was manifested at the stage of triage – the initial assessment of the patient’s condition. This is the most difficult moment: there is a minimum of information, and a decision must be made quickly and with a high cost of error.

Model o1 demonstrated an accurate or close to correct diagnosis in 67% of cases. For comparison:

• one of the doctors showed a result of about 55%
• the second – about 50%

And it is not about isolated luck: the model stably kept the level not lower than the human level even at the following stages of symptom analysis.

Researchers separately note that the algorithms have proven to be particularly strong in pattern recognition — they quickly “assemble” a picture from scattered features, even if the data is incomplete or noisy.

Restrictions: why doctors will not disappear anywhere

Despite the impressive numbers, it is premature to talk about the replacement of doctors. The authors of the study emphasize that the current results are a demonstration of potential, not a ready-made clinical solution.

There are a number of fundamental limitations:

• there is no legal and professional responsibility for AI decisions
• algorithms do not take into account the entire context of the patient (social, behavioral, psychological)
• large-scale prospective trials in real hospitals are needed

In addition, even with high accuracy, models can make errors in non-standard ways – and such errors are more difficult to predict.

What it changes for medicine

Harvard Medical School’s work actually captures the new role of AI: from a data analysis tool to a participant in clinical thinking.

In practice, this may lead to the emergence of a hybrid model of medicine, where:

• AI takes over the initial processing and hypotheses
• the doctor performs the function of checking, interpreting and making the final decision

The main effect is not competition, but strengthening. Where a person is limited by time and the amount of information, algorithms can be a “second opinion” in time.

And if earlier the question sounded like “will AI replace the doctor”, now it is formulated differently: how much faster and more accurate medicine will become when a person and an algorithm start working as a single system.