Open AI has trained a classifier to distinguish between text written by a human and text written by AIs. They are making this classifier publicly available to get feedback on whether imperfect tools like this one are useful.

While this resource is focused on educators, the company expects it to have an impact on journalists, mis/disinformation researchers, and other groups.

AI Classifier for Indicating AI-written Text

In their evaluations on a “challenge set” of English texts, the classifier correctly identified 26% of AI-written text (true positives) as “likely AI-written. However, it incorrectly labeled human-written text as AI-written 9% of the time (false positives). 

Start off your week with tech news recap and trending conversations in your inbox

Enter valid email-id

Subscribe

I prefer it on Telegram

The classifier’s reliability typically improves as the length of the input text increases.

Just like ChatGPT, this classifier is still in its infancy. It has several serious limitations. It should not be used as a primary decision-making tool, but instead as a complement to other methods of determining the source of a piece of text.

Other limitations include:

1. The classifier is very unreliable on short texts (below 1,000 characters). Even longer texts are sometimes incorrectly labeled by the classifier.
2. Sometimes the human-written text will be incorrectly but confidently labeled as AI-written by the classifier.
3.  It performs significantly worse in other languages that are not English and it is unreliable on code.
4. Very predictable text cannot be reliably identified. For example, it is impossible to predict whether a list of the first 1,000 prime numbers was written by AI or humans because the correct answer is always the same.
5. AI-written text can be edited to evade the classifier. 
6. The classifier is sometimes extremely confident in a wrong prediction.

Those affected by the issues addressed by this classifier are invited to do direct feedback on the preliminary resource. 

Microsoft and OpenAI-the parent company of chatbot GPT- have recently announced a new phase of their partnership. This is with a multi-year, multi-billion dollar investment to accelerate AI advancements.

The above classifier could be one of the many developments that we will see out of this new partnership.