3 steps to help you work out whether something was written by a bot or a human

If you spent time on social media today, you probably came across that script for an Olive Garden commercial allegedly written by an artificial intelligence algorithm. The commercial is a hilarious trip into the absurd in which patrons enjoy classic staples of the kind of Italian dining we’ve come to expect from America’s 15th favorite chain restaurant, such as “warm and defeated pasta nachos,” secret soup, Italian citizens, and “unlimited stick.”

The commercial was also, unfortunately, likely not written by AI at all. Instead, it was probably just a boring old human who claimed to have used a neural net for some sweet, sweet social media fame.

Last night, engineer Janelle Shane took to Twitter to lay out some of the telltale giveaways that the script was written by a person pretending to be an AI algorithm for kicks. You may recognize Shane as the person who trains neural nets to create jokes that devolve into nonsense or paint colors that almost sound real after being trained on thousands of actual examples. Yes, the AI-generated results are absurd, but they also highlight one key fact — the neural nets have no clue what the hell they’re talking about.

So how do you spot something written by an AI, anyhow?

“I’d say the clearest giveaways are a really short memory (maybe just a couple of sentences long) and a lack of understanding of meaning and context,” Shane told Futurism. “One characteristic of neural net text is it’ll tend to mimic the surface appearance of things without really getting the meaning behind them.”

The fun parts of these bot-written passages are the parts where it creates impossibly-surreal scenarios, but one tell-tale sign that something was actually written by a person is when those individual images still fit together. For instance, the cast of the Olive Garden commercial remains consistent; if an actual neural net had been the author, characters would have been introduced and abandoned willy-nilly.

A neural net trained just on commercials wouldn’t understand how to put together a cohesive narrative, but a human writer would keep an eye out for these things.

Take, for example, this example of a recipe that was written by a neural net trained on recipes:

The instructions, you notice, have absolutely nothing to do with the ingredients listed.

Right now, artificial intelligence excels at incredibly-narrow tasks. It can generate cohesive grammar at the sentence level, but something like a script is still too complex.

“For many years yet, it will be the case that if you see a well-written story with a coherent plot and clever wordplay, it will be because a human did most of the work,” Shane said.

AI-written text will continue to grow in sophistication and prevalence, even if it’s mostly a source of humor today, Shane added. And that means it can grow to be more misleading. Today we’re just talking about who or what wrote a funny commercial online, but as artificial intelligence becomes more sophisticated, it could be used to write misinformation like convincing (but fake news) articles. That could have very real consequences for people who fall for it.

Now, based on what Shane told us, we’ve come up with a list of tell-tale signs to look for if you want to know whether a particular text was penned by human or a bot.

Did it make sense?If something looks like it matches a classic joke convention but the content seems totally garbled, it likely wasn’t written by a person.How’s their attention span? If the author seems to have forgotten what they were talking about part of the way through, then you’re likely witnessing a neural net’s inherently-short attention span. Meanwhile, if the text seems sophisticated and clever or it seems as though someone actually put care into the structure of their sentences, then you’re looking at signs of a human’s touch.Did they show their work? As Shane pointed out, whoever managed to write an Olive Garden commercial with nothing but an AI algorithm would be gloating about it much more — we’d see a whole lot more about how they trained their neural net and how they managed to make everything come together. The fact that no one is geeking out about the technical side of this neural net suggests that it doesn’t exist.

The truth at the bottom of all this? We need to know what AI is actually capable of. Because how can we appreciate a parody if we don’t understand what it’s mimicking?

“There’s definitely a place for parodies of AI-generated text,” added Shane, “but the parodies only work if you know what the real stuff is like.”