I have a preprint out estimating how many scholarly papers are written using chatGPT etc? I estimate upwards of 60k articles (>1% of global output) published in 2023. https://arxiv.org/abs/2403.16887
How can we identify this? Simple: there are certain words that LLMs love, and they suddenly start showing up *a lot* last year. Twice as many papers call something "intricate", big rises for "commendable" and "meticulous".
@generalising
Very cool!
from what I understand, if a paper contains both "“intricate” and “meticulous” or
“intricate” and “notable” there is a very high chance it is by LLM (although, of course that's not a proof..especially now that your work will be noticed).
Is there hope to get automatic LLM with some~99% confidence? Of course the entire point is that LLM mimic human language, but you are showing that language correlations appear and are strong indicators, and they may hard to polish.
@franco_vazza there's always a baseline of usage, but that pair together has suddenly become a lot more common.
I don't think this approach is great for detecting LLM involvement in any individual paper (there are *much* more sophisticated tools for that) but it works OK for estimation at a much broader scale.
