GPT-4 performance comparable with physicians on official medical board residency examinations. Model performance near or above official passing rate in all medical specialties tested
GPT versus Resident Physicians — A Benchmark Based on Official Board Scores | NEJM AI
What would be much more useful is to provide a model with actual patient files and see what kills more people, doctors or models.
I would watch that show.
Like “Is it Cake”
But life or death is on the line….
“Is it Lupus?” Or “Are you Dying?”
I wonder why nobody seems capable of making a LLM that knows how to do research and cite real sources.
I mean LLMs pretty much just try to guess what to say in a way that matches their training data, and research is usually trying to test or measure stuff in reality and see the data and try to find conclusions based on that so it doesn't seem feasible for LLMs to do research
They maybe used as part of research but it can't do the whole research as a crucial part of most research would be the actual data and you'd need a LOT more than just LLMs to get that
Yup! LLMs don't put facts together. They just look for patterns, without any concept of what they are looking at.
Because the inherent design of modern AIs is not deterministic.
Adding a progressively bigger model cannot fix that. We need an entirely new approach to AI to do that.
For me the biggest indicator that we've barking up the wrong tree is energy consumption.
Consider the energy required to feed a human with that required to train and run the current "leading edge" systems.
From a software development perspective, I think machine learning is a very useful way to model unknown variables, but that's not the same as "intelligence".
Cohere's command-r models are trained for exactly this type of task. The real struggle is finding a way to feed relevant sources into the model. There are plenty of projects that have attempted it but few can do more than pulling the first few search results.
Neat but I don't think LLMs are the way to go for these sort of things
I don’t mind so long as all results are vetted by someone qualified. Zero tolerance for unfiltered AI in this kind of context.
This research has been done a lot of a times but I don't see the point of it. Exams are something I would expect LLMs, especially the higher end ones, to do well because of their nature. But it says next to nothing about how reliable the LLM as an actual doctor.
But it says next to nothing about how reliable the LLM as an actual doctor.
Yet these tests say anything about how a human would be as an actual doctor?
Google started killing the Dr industry (gp) Ai will finally be the nail in the coffin except Drs will never give up the power to prescribe
LLMs can't design experiments or think of consequences or quality of life.
They also don't "learn" from asking questions or from a 1-time input. They need to see hundreds or thousands of people die from something to recognize the pattern of something new.
It's just a multiple choice test with question prompts. This is the exact sort of thing an LLM should be very good at. This isn't chat gpt trying to do the job of an actual doctor, it would be quite abysmal at that. And even this multiple choice test had to be stacked in favor of chat gpt.
I, too, can pass the Boards if you remove all the questions I don't understand.
I’d be fine with LLMs being a supplementary aid for medical professionals, but not with them doing the whole thing.