I am happy for solving protein folding for a step towards more rational drug design and additionally, eventually applying computer vision on organoids to shift to a factory-style high throughput screening for new pharmaceuticals - that’s rapid genuine progress if folks understood what actually was possible with a CS, computational bio/biochem, and ML/“AI” background.
I’m also incredibly happy there is a lot less of an “AI winter” relatively speaking for radiology as Medicare Technology payments are back in full swing. I have no qualms about what could potentially be possible in healthcare “AI”.
The problem is many people really do fall for what is termed “AI snake oil” in your link far too often, as you probably already know is fairly rampant in SV to raise funds “Theranos style” with only an Excel sheet in the back doing linear regression. At least the potential for human harm is less so with non-healthcare related AI.
The key part of the Theranos scam was there wasn’t any investors who were familiar with biotech and it was literally impossible for some of the things Holmes claimed…seen it way too often in a similar fashion for “AI” - people with no knowledge or context making far too optimistic predictions.
When you tell people off for Theranos and Watson Health at the time and explain directly why some things are literally impossible and other parts incredibly implausible, you’ll get the same responses.
Absolutely - very few VCs have the necessary expertise to really do good due diligence on these types of startups, and with FOMO (fear of missing out), they just forge ahead with investments and hope that the lead investor knows what they are doing.
AI in healthcare is going to be a very slow development and adoption process… I think.
The FDA will have a lot of issues with certifying things where you can’t identify the exactly inputs (and the rationale) for a given output. I’ve worked in regular Tech (with the typical 6 month to 9 month product development cycles) and also in digital health. Digital health (that involves anything clinical or FDA) is just unbelievably slow by traditional tech traditions; years instead of months for clinical trials, FDA reviews, etc… And thats probably appropriate given the risks and consequences, but its a very different mindset and timeline for people coming out of consumer or business tech industries.
BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e., BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large scale biomedical literature. We evaluate BioGPT on six biomedical NLP tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our larger model BioGPT-Large achieves 81.0% on PubMedQA. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms. Code is available at this https URL.
There are likely going to be many different vertical markets where this tech can be applied (consumer, medical (Radiology/cardiology/neurology, etc.), chemical, biological, etc.). And the quality is largely based on how large the dataset is (and the quality of the dataset is) that you use to train the AI system - so people with the most good data will likely be the winners).
We’ve talked about this market for years in the silicon valley… most of us always assumed that Google is the natural company to dominate many of these AI markets because they have by far the most data to train their AI systems on (billions of users searching on everything for 15+ years). But there is always the “innovators dilemma”, issue of making obsolete your own products, which causes issues in companies.
Medical records are largely Silo’d in different medical record systems with all sorts of protections, etc… Its complex. We’ll see where this goes. It will be interesting to see how it evolves.
But yes, like PDF and many tech markets, these will likely be “winner take all” type of markets. The key question I have is how much protection / independence will different vertical markets have from each other. If a company “wins” in the consumer market, does that mean they are likely to win in a business or science or medical vertical market? That gets harder to call and will depend on how different the markets and data is.
The AI market could evolve like the voice recognition software market - I’m not very familiar with that market, nor with Dragon Systems (I think thats consumer oriented isn’t it?)… Here is one market share breakout I saw on that market… I wouldn’t be surprised to see the AI market evolve similarly.
I use Dragon Medical for transcription daily, it’s very error-prone, and you must double-check it thoroughly, but I create long narratives. Grammarly catches many errors (and also makes them). Imagine excellent transcription software coupled with a real-time AI; it might end up predicting fairly accurately what I am going to say next. I am pretty certain 2023 is going to be the year AI takes a quantum leap. My advice is to adapt and learn to use AI to our advantage, but don’t surrender to it.
I did more different tests and not as optimistic as the author portrayed even though it seems the diagnosis process was more similar - inaccurate results and making stuff up. I suspect the author, which appears to be a medical student, doesn’t have a CS & ML/NLP type of “AI” background, so may have interpreted the results too optimistically and used a methodology that may have not been targeted towards more accurate testing, on top of perhaps a lack of clinical experience to get enough context on what the applications are. I would note it’s likely mnemonics would be a great option though. It’s just that even errata in textbooks can happen fairly often (when I was younger and slightly more petty, I’d actually pick out First Aid errata to prove how some of these “shortcut” texts weren’t all that great) let alone internet sources, so ChatGPT in medical education seems like a bad idea for now until there is more accuracy than current standard “shortcut” texts and the far more dense standard medical texts that one would expect for someone to really drill in depth.
Also, keep in mind the USMLE exam is not quite simple - it had about the same accuracy as PubMedGPT before, and even if they did random spot checking to reduce the risk of it simply copying answers from questions - a lot of high similarity or paraphrased clinical vignettes which may prompt the answer are actually available on the internet as practice questions, especially I suspect recently that may be the case. It’s not surprising that the average USMLE scores have gone up over time as students are just spamming qbanks such as UWorld, rather than starting with say “Big Robbins” and all the other generally recommended (and extremely dense) textbooks to get all the pathology details with full understanding, and I don’t blame them - it’s a lot of material to be frank and it requires 80+ hours per week level dedication on average. I suspect many students these days will put true understanding for the backburner during med school years until they catch up later.
The “shortcut” to medical education that is generally heavily discouraged by professors are just keep doing practice questions. But likely not happening from small sample surveying.
Even then with such benchmark testing and assuming ChatGPT actually had some improved model somehow compared to PubMedGPT despite lack of specific model training, “within passing threshold” is actually not particularly impressive. Basically 99% pass rate unless they didn’t study for US MD grads, slightly less for everyone else. Basically “within passing” for Step 1 and Step 2 these days could mean somewhere between a lack of residency options and the rare flunking out of school.
I’ll also mention I can probably get the answers to “barely pass” medical boards from Google if I didn’t study at all at the time and had unlimited time to take the exam. So again, it’s actually not super impressive to me as opposed to random people. Although this is somewhat off-topic - I can see why Alphabet’s CEO appears to be kinda worried overall when it comes to ChatGPT in general since they probably can’t greenlight their LLM as quick due to concerns of misinformation and reputation risk. Frankly, I wonder whether TikTok should be more concerning to them despite apparent regulatory pressure, as Google’s LLM seems to be doing alright relatively speaking.
There is also the huge issue of how do you monetize AI generated responses. A large part of the market may go to verbal searches on mobile, or even if its an AI generated text response on desktop (similar to ChatGPT) to a question, it doesn’t lend itself much to people clicking on ads… so its in Google’s interest to slow down the adoption of AI-based search as much as possible, to slow the negative impact on their revenue/profits.
I have no idea how they’re planning to exactly monetize but my current working hypothesis is Google does a lot of things for free to lock people in their ecosystem and suck up as much personal data as possible when they keep within it - which is used for ad revenue as you know. The ads don’t have to be initially on the LLM part until it’s clear they’re going to blow competitors away. Google has done just fine with ads on other websites from the start.
Subjectively, I feel Google’s search result quality has been deteriorating (partly from higher quality being unprofitable and aggressive SEO optimization) but not yet to the point where people will stop using Google.
Hence, if people find a much much better option for “discovery” - they may use something else as their “search”. I’m aware of several search engine startups including a few stealth ones with limited public information. I don’t think DuckDuckGo or Bing really are going to take much of Google and it’s hard to tell for startups in general. I don’t know for sure - but I think ChatGPT is what they should not be concerned about so far and there will be a slightly clearer picture when Google releases their LLM. I doubt ChatGPT is anywhere near 10x better than Google’s LLM.
If one uses Google’s LLM instead of ChatGPT - they could suck in more user data which would be used for ad revenue presumably. I believe recalling the founders once considered starting a hedge fund, but decided ultimately against it due to regulatory issues and ads were a much better biz model for them.
They’re offering ChatGPT Plus for$20 a month. Am considering it, event though I have no intended use yet. I have made more foolish purchases. I do not know if I can get into the waitlist.
Maybe like L_Hayes says, order it (if that can be done) to source its opinion/answers only from peer-reviewed publications.
This is what I am using; I have spent 0.28 $ these last two months and have been able to ask as many questions as I desired. Sometimes it makes up things that are not true.
