How to Detect AI-Written Texts?

One common approach to detecting AI-generated text is using perplexity. Perplexity helps determine how likely a model would recognize text as probable. If a text is probable, it means the model could generate it. A recent paper titled 'GHOSTBUSTER: Detecting Text Ghostwritten by Large Language Models' uses this concept. The main goal was a robust and adaptable system for detecting AI-generated texts that addresses the limitations of current AI text detection methods, which often work poorly across various text domains and styles.

Methodology

Ghostbuster is a linear classifier. Despite its simplicity, it achieves high accuracy and generalizability in detecting AI-generated text. How? Feature selection. The process involves two main steps: generating token probabilities using selected language models, and training the linear model to classify the input.

Initially, they collected both human-authored and AI-generated texts. They processed these using simpler language models like GPT-2 or n-grams, which are smaller and less advanced than current popular LLMs. AI models tend to use some phrases more than humans, and humans can create sentences that AI models would not. For example, if you frequently use ChatGPT, you might have noticed phrases like "in the realm", "elevate", "as we continue to harness the power of". While encountering one such phrase might be normal, a text densely packed with them could raise suspicions.

Hence, for each word or token in the documents, they calculated the likelihood of its occurrence based on the simpler language models. These probabilities aid in understanding the contextual meaning of each word, crucial for distinguishing between human and AI writing.

Then, they used a combination of vector and scalar operations to create a broad range of features from the probability vectors. Features were generated by combining up to three vectors of probabilities using vector functions, then reducing them to a scalar using scalar functions. These features were then used to train a linear classifier.

Results

The classifier demonstrated high accuracy, achieving a 99.0% F1 score in domain classification. This performance significantly surpassed existing models like DetectGPT and GPTZero by an average margin of 23.7% F1.

Advantages and potential limitations

Ghostbuster does not need access to token probabilities from the target model, making it effective against black-box models. Moreover, the authors made the code for experiment and a useful demo publicly available. Upon examining the demo, you'll find that it requires a minimum of 100 words and performs best with texts over 250 words. This is logical, as differentiating texts based on a single sentence would be nearly impossible.

The structured approach to feature selection and the use of specific datasets could lead to overfitting. The system might excel with texts similar to its training examples but struggle with significantly different texts. Additionally, the reliance on predefined vector and scalar operations might limit its ability to detect more complex or subtle characteristics of AI-generated text.

Conclusion

Ghostbuster presents an effective approach to detecting AI-generated text. By combining simpler language models, structured feature selection, and a linear classifier, it achieves a high accuracy and adaptability.

Still, we should remember that AI writing styles are continuously changing. Adapting to these remains a challenge. Moreover, many texts, due to their formal style and repetitive nature, may never be distinguishable. When we are exposed to so many AI-written texts, our writing styles might also evolve, potentially closing the line between human and AI writing styles.

Learning more

If you are interested in a hand-picked, brief list of recently presented papers, check out the Warsaw.AI Newsletter