Skip Navigation

Technology @beehaw.org

liv @beehaw.org

5mo ago

OpenEvidence Sounds Promising, but is it Reliable?

Just a moment...

Technology @lemmy.world

5mo ago

OpenEvidence Sounds Promising, but is it Reliable?

medium.com /@underwaterchimp/openevidence-sounds-promising-but-is-it-reliable-0b5f49fb8d5c

11 comments

Article about an AI that aims to give treatment suggestions to doctors, with some alarming results.
When OpenEvidence took the US Medical Licensing Exam recently, it was wrong 9% of the time
The passing score for USMLE is ~200 out of 300... how many "wrong times" is that?
https://www.kaptest.com/study/usmle/passing-scores/
- When we look at passing scores, is there any way to quantitatively grade them for magnitude?
  Not all bad advice is created equal.
  
  The grading is a mess. It goes about qualitative, quantitative... and statistical corrections "to make it fair".
  Anyway, there is ~30% margin on the scores for passing, so chances are that 9% is better than the worst doctor who still "passed".
It's not reliable. The name itself is misleading. The "evidence" is apparently already open. The article doesn't seem to say whether the statistical model is open. My guess would be no.
More accurate name: ClosedSummary

11 comments