I have a preprint out estimating how many scholarly papers are written using chatGPT etc? I estimate upwards of 60k articles (>1% of global output) published in 2023. arxiv.org/abs/2403.16887

How can we identify this? Simple: there are certain words that LLMs love, and they suddenly start showing up *a lot* last year. Twice as many papers call something "intricate", big rises for "commendable" and "meticulous".

Very cool!
from what I understand, if a paper contains both "“intricate” and “meticulous” or
“intricate” and “notable” there is a very high chance it is by LLM (although, of course that's not a proof..especially now that your work will be noticed).

Is there hope to get automatic LLM with some~99% confidence? Of course the entire point is that LLM mimic human language, but you are showing that language correlations appear and are strong indicators, and they may hard to polish.


@franco_vazza there's always a baseline of usage, but that pair together has suddenly become a lot more common.

I don't think this approach is great for detecting LLM involvement in any individual paper (there are *much* more sophisticated tools for that) but it works OK for estimation at a much broader scale.

· · Web · 0 · 1 · 2
Sign in to participate in the conversation

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!