Why AI still has less common sense than a toddler

The way artificial intelligence is progressing, you might think that a robot takeover of the human race is right around the corner. Artificial brains can now drive cars, do legal research, recognize faces, and beat the best human players at games like Go. Prominent technologists keep warning that AI poses a fundamental threat to humanity.

The good news for us humans is that computers still have less common sense than a toddler.

Just look at computer programs that are supposed to understand and process human language. These bots, like Siri, Alexa, and others, frequently trip over sentences a three-year-old would have no trouble with. “I sneezed the other day and Alexa thought I was saying her name,” said my colleague Mike Murphy. The three-year-old would have said “bless you.”

Thus the Winograd Schema Challenge was developed. It’s a way to test just how much common sense these kinds of bots have. And, as expected, the bots aren’t doing so well.

A Winograd Schema question is one that is extremely easy for humans to answer, but defies the cold logic of computers. Take the following example: “The man couldn’t lift his son because he was so weak. Who was weak, the man or his son?” In this case, “he” could logically refer to either the man or his son. But as humans, we know it would be silly to mention that the son was weak in this context. For computers, the “he” is equally valid for both.

The challenge, therefore, is to build programs that can answer these kinds of questions with a success rate around that of a human’s. Six programs, submitted by independent students and researchers, competed in the latest challenge, held last month at the International Joint Conference on Artificial Intelligence in New York. These six were right no more than half of the time. That’s the same as guessing randomly, since the questions only have two possible answers. The human subjects asked the same set of questions got over 90% of the questions right.

None of the heaviest hitters in language processing—Google, Baidu, and the like—participated, but one of the submissions did employ the sophisticated machine-learning techniques that these companies have embraced for such tasks. So it’s not likely that even Google has bridged the gap between the robot and human scores.

Here is a sampling of a set of Winograd questions published by Ernest Davis, a professor of computer science at New York University who specializes in “common sense testing.” These easy questions should make you feel better about your prospects in the “robot economy” we’ve all been warned about.

#1. The city councilmen refused the demonstrators a permit because they advocated violence.

Q: Who advocated violence?
Answers: The city councilmen/the demonstrators

#2. The trophy doesn’t fit into the brown suitcase because it’s too small.

Q: What is too small?
Answers: The suitcase/the trophy

#3. Joan made sure to thank Susan for all the help she had received.

Q: Who had received help?
Answers: Susan/Joan

#4. Paul tried to call George on the phone, but he wasn’t successful.

Q: Who was not successful?
Answers: Paul/George

#5. The lawyer asked the witness a question, but he was reluctant to answer.

Q: Who was reluctant to answer the question?
Answers: The witness/the lawyer

#6. The delivery truck zoomed by the school bus because it was going so slow.

Q: What was going so slow?
Answers: The truck/the bus

#7. Frank felt vindicated when his longtime rival Bill revealed that he was the winner of the competition.

Q: Who was the winner of the competition?
Answers: Frank/Bill

#8. The man couldn’t lift his son because he was so weak.

Q: Who was weak?
Answers: The man/the son

#9. The large ball crashed right through the table because it was made of steel.

Q: What was made of steel?
Answers: The ball/the table

#10. John couldn’t see the stage with Billy in front of him because he is so tall.

Q: Who is so tall?
Answers: John/Billy

#11. Tom threw his schoolbag down to Ray after he reached the top of the stairs.

Q: Who reached the top of the stairs?
Answers: Tom/Ray

#12. Although they ran at about the same speed, Sue beat Sally because she had such a bad start.

Q: Who had a bad start?
Answers: Sue/Sally

#13. The sculpture rolled off the shelf because it wasn’t level.

Q: What wasn’t level?
Answers: The sculpture/the shelf

#14. Sam’s drawing was hung just above Tina’s and it did look much better with another one below it.

Q: Which looked better?
Answers: Sam’s drawing/Tina’s drawing

#15. Anna did a lot better than her good friend Lucy on the test because she had studied so hard.

Q: Who studied hard?
Answers: Anna/Lucy

#16. The firemen arrived before the police because they were coming from so far away.

Q: Who came from far away?
Answers: The firemen/the police

#17. Frank was upset with Tom because the toaster he had bought from him didn’t work.

Q: Who had bought the toaster?
Answers: Frank/Tom

#18. Jim yelled at Kevin because he was so upset.

Q: Who was upset?
Answers: Jim/Kevin

#19. The sack of potatoes had been placed below the bag of flour, so it had to be moved first.

Q: What had to be moved first?
Answers: The sack of potatoes/the bag of flour

#20. Pete envies Martin because he is very successful.

Q: Who is very successful?
Answers: Martin/Pete

#21. I was trying to balance the bottle upside down on the table, but I couldn’t do it because it was so top-heavy.

Q: What was top-heavy?
Answers: The bottle/the table

#22. I spread the cloth on the table in order to display it.

Q: To display what?
Answers: The table/the cloth

#23. The older students were bullying the younger ones, so we rescued them.

Q: Whom did we rescue?
Answers: The younger students/the older students

#24. I poured water from the bottle into the cup until it was empty.

Q: What was empty?
Answers: The cup/the bottle

#25. Susan knows all about Ann’s personal problems because she is nosy.

Q: Who is nosy?
Answers: Susan/Anne

#26. Sid explained his theory to Mark but he couldn’t understand him.

Q: Who did not understand whom?
Answers: Sid did not understand Mark/Mark did not understand Sid.

#27. Susan knew that Ann’s son had been in a car accident, so she told her about it.

Q: Who told the other about the accident?
Answers: Susan/Ann

#28. Joe’s uncle can still beat him at tennis, even though he is 30 years older.

Q: Who is older?
Answers: Joe’s uncle/Joe

#29. The police left the house and went into the garage, after they found the murder weapon.

Q: Where did they find the murder weapon?
Answers: In the garage/in the house

#30. The painting in Mark’s living room shows an oak tree. It is to the right of the bookcase.

Q: What is to the right of the bookcase?
Answers: The painting/the tree

#31. There is a gap in the wall. You can see the garden behind it.

Q: You can see the garden behind what?
Answers: The gap/the wall

#32. The drain is clogged with hair. It has to be removed.

Q: What has to be removed?
Answers: The drain/the hair

#33. My meeting started at 4:00 and I needed to catch the train at 4:30, so there wasn’t much time. Luckily, it was delayed, so it worked out.

Q: What was delayed?
Answers: The meeting/the train

#34. There is a pillar between me and the stage, and I can’t see it.

Q: What can’t I see?
Answers: The stage/the pillar

#35. They broadcast an announcement, but a subway came into the station and I couldn’t hear over it.

Q: What couldn’t I hear over?
Answers: The announcement/the subway

#36. In the middle of the outdoor concert, the rain started falling, and it continued until 10.

Q: What continued until 10?
Answers: The rain/the concert

#37. I used an old rag to clean the knife, and then I put it in the drawer.

Q: What did I put in the drawer?
Answers: The knife/the rag

#38. Ann asked Mary what time the library closes, because she had forgotten.

Q: Who had forgotten?
Answers: Mary/Ann

#39. I took the water bottle out of the backpack so that it would be handy.

Q: What would be handy?
Answers: The backpack/the bottle

#40. I couldn’t put the pot on the shelf because it was too tall.

Q: What was too tall?
Answers: The shelf/the pot

#41. I’m sure that my map will show this building; it is very famous.

Q: What is famous?
Answers: The building/the map

#42. Bob paid for Charlie’s college education. He is very generous.

Q: Who is generous?
Answers: Bob/Charlie

#43. Bob paid for Charlie’s college education, but now Charlie acts as though it never happened. He is very ungrateful.

Q: Who is ungrateful?
Answers: Bob/Charlie

#44. Bob was playing cards with Adam and was way ahead. If Adam hadn’t had a sudden run of good luck, he would have lost.

Q: Who would have lost?
Answers: Bob/Adam

#45. Adam can’t leave work here until Bob arrives to replace him. If Bob had left home for work on time, he would be gone by this time.

Q: Who would be gone?
Answers: Bob/Adam

#46. If the con artist had succeeded in fooling Sam, he would have lost a lot of money.

Q: Who would have lost the money?
Answers: The con artist/Sam

#47. It was a summer afternoon, and the dog was sitting in the middle of the lawn. After a while, it got up and moved to a spot under the tree, because it was cooler.

Q: What was cooler?
Answers: The dog/The spot under the tree

#48. The cat was lying by the mouse hole waiting for the mouse, but it was too impatient.

Q: What was too impatient?
Answers: The mouse/the cat

#49. Anne gave birth to a daughter last month. She is a very charming baby.

Q: Who is a very charming baby?
Answers: Anne/Anne’s daughter

#50. Alice tried frantically to stop her daughter from chatting at the party, leaving us to wonder why she was behaving so strangely.

Q: Who was behaving strangely?
Answers: Alice/Alice’s daughter

#51. I saw Jim yelling at some guy in a military uniform with a huge red beard. I don’t know why he was, but he looked very unhappy.

Q: Who looked very unhappy?
Answers: The guy in the uniform/Jim

#52. The fish ate the worm. It was hungry.

Q: What was hungry?
Answers: The worm/the fish

#53. I was trying to open the lock with the key, but someone had filled the keyhole with chewing gum, and I couldn’t get it out.

Q: What couldn’t I get out?
Answers: The key/the chewing gum

#54. The dog chased the cat, which ran up a tree. It waited at the bottom.

Q: Which waited at the bottom?
Answers: The cat/the dog

#55. In the storm, the tree fell down and crashed through the roof of my house. Now, I have to get it removed.

Q: What has to be removed?
Answers: The tree/the roof

#56. The customer walked into the bank and stabbed one of the tellers. He was immediately taken to the police station.

Q: Who was taken to the police station?
Answers: The teller/the customer

#57. John was doing research in the library when he heard a man humming and whistling. He was very annoyed.

Q: Who was annoyed?
Answers: John/the hummer

#58. John was jogging through the park when he saw a man juggling watermelons. He was very impressive.

Q: Who was impressive?
Answers: John/the juggler

#59. Bob collapsed on the sidewalk. Soon he saw Carl coming to help. He was very concerned.

Q: Who was concerned?
Answers: Bob/Carl

#60. Sam and Amy are passionately in love, but Amy’s parents are unhappy about it, because they are 15.

Q: Who are 15?
Answers: Amy’s parents/Sam and Amy.

#61. Mark told Pete many lies about himself, which Pete included in his book. He should have been more skeptical.

Q: Who should have been more skeptical?
Answers: Mark/Pete

#62. Joe has sold his house and bought a new one a few miles away. He will be moving out of it on Thursday.

Q: Which house will he be moving out of?
Answers: The old house/the new house

#63. Many people start to read Paul’s books and can’t put them down. They are gripped because Paul writes so well.

Q: Who or what are gripped?
Answers: The readers/the books

#64. Mary took out her flute and played one of her favorite pieces. She has loved it since she was a child.

Q: What has Mary loved since she was a child?
Answers: The piece/the flute

#65. Sam pulled up a chair to the piano, but it was broken, so he had to sing instead.

Q: What was broken?
Answers: The chair/the piano

#66. Since it was raining, I carried the newspaper in my backpack to keep it dry.

Q: What was I trying to keep dry?
Answers: The backpack/the newspaper

#67. Sara borrowed the book from the library because she needs it for an article she is working on. She reads it when she gets home from work.

Q: What does Sara read when she gets home from work?
Answers: The book/the article

#68. This morning, Joey built a sand castle on the beach, and put a toy flag in the highest tower, but this afternoon a breeze knocked it down.

Q: What did the breeze knock down?
Answers: The flag/the sand castle

#69. Jane knocked on Susan’s door, but there was no answer. She was disappointed.

Q: Who was disappointed?
Answers: Susan/Jane

#70. Jane knocked on the door, and Susan answered it. She invited her to come in.

Q: Who invited whom?
Answers: Jane invited Susan/Susan invited Jane.

#71. Sam took French classes from Adam, because he was known to speak it fluently.

Q: Who was known to speak French fluently?
Answers: Sam/Adam

#72. The path to the lake was blocked, so we couldn’t reach it.

Q: What couldn’t we reach?
Answers: The lake/the path

#73. The sun was covered by a thick cloud all morning, but luckily, by the time the picnic started, it was out.

Q: What was out?
Answers: The cloud/the sun

#74. We went to the lake, because a shark had been seen at the ocean beach, so it was a dangerous place to swim.

Q: Which was a dangerous place to swim?
Answers: The beach/the lake

#75. Sam tried to paint a picture of shepherds with sheep, but they ended up looking more like dogs.

Q: What looked like dogs?
Answers: The sheep/the shepherds

#76. Mary tucked her daughter Anne into bed, so that she could sleep.

Q: Who is going to sleep?
Answers: Anne/Mary

#77. Fred and Alice had very warm down coats, but they were not enough for the cold in Alaska.

Q: Who or what were not enough for the cold?
Answers: The coats/Fred and Alice

#78. Thomson visited Cooper’s grave in 1765. At that date he had been traveling for five years.

Q: Who had been traveling for five years?
Answers: Cooper/Thomson

#79. Jackson was greatly influenced by Arnold, though he lived two centuries later.

Q: Who lived later?
Answers: Arnold/Jackson

#80. Tom’s daughter Eva is engaged to Dr. Stewart, who is his partner. The two doctors have known one another for 10 years.

Q: Which two people have known one another for 10 years?
Answers: Tom and Dr. Stewart / Eva and Dr. Stewart.

#81. I can’t cut that tree down with that axe; it is too small.

Q: What is too small?
Answers: The tree/the axe.

#82. The foxes are getting in at night and attacking the chickens. I shall have to guard them.

Q: What do I have to guard?
Answers: The chickens/the foxes.

#83. The foxes are getting in at night and attacking the chickens. They have gotten very bold.

Q: What has gotten bold?
Answers: The foxes/the chickens

#84. Fred covered his eyes with his hands, because the wind was blowing sand around. He lowered them when the wind stopped.

Q: What did Fred lower?
Answers: His eyes/his hands

#85. The actress used to be named Terpsichore, but she changed it to Tina a few years ago, because she figured it was too hard to pronounce.

Q: Which name was too hard to pronounce?
Answers: Tina/Terpsichore

#86. Fred watched TV while George went out to buy groceries. After an hour he got up.

Q: Who got up?
Answers: Fred/George

#87. Fred was supposed to run the dishwasher, but he put it off because he wanted to watch TV. But the show turned out to be boring, so he changed his mind and turned it on.

Q: What did Fred turn on?
Answers: The dishwasher/the television

#88. Fred is the only man still alive who remembers my great-grandfather. He is a remarkable man.

Q: Who is a remarkable man?
Answers: Fred/my great-grandfather

#89. Fred is the only man alive who still remembers my father as an infant. When Fred first saw my father, he was 12 months old.

Q: Who was 12 months old?
Answers: Fred/my father

#90. In July, Kamtchatka declared war on Yakutsk. Since Yakutsk’s army was much better equipped and 10 times larger, they were defeated within weeks.

Q: Who was defeated?
Answers: Yakutsk/Kamchatka

#91. Elizabeth moved her company from Sparta to Troy to save money on taxes; the taxes are much lower there.

Q: Where are the taxes lower?
Answers: In Sparta/in Troy

#92. Esther figures that she will save shipping costs if she builds her factory in Springfield instead of Franklin, because most of her customers live there.

Q: In which town do most of Esther’s customers live?
Answers: Springfield/Franklin

#93. Look! There is a minnow swimming right below that duck! It had better get away to safety fast!

Q: What needs to get away to safety?
Answers: The minnow/the duck

#94. There are too many deer in the park, so the park service brought in a small pack of wolves. The population should decrease over the next few years.

Q: Which population will decrease?
Answers: The wolves/the deer

#95. Archaeologists have concluded that humans lived in Laputa 20,000 years ago. They hunted for deer on the river banks.

Q: Who hunted for deer?
Answers: The prehistoric humans/the archaeologists

#96. The scientists are studying three species of fish that have recently been found living in the Indian Ocean. They appeared two years ago.

Q: Who or what appeared two years ago?
Answers: The fish/the scientists

#97. The journalists interviewed the stars of the new movie. They were very persistent, so the interview lasted for a long time.

Q: Who was persistent?
Answers: The stars/the journalists

#98. The police arrested all of the gang members. They were trying to run the drug trade in the neighborhood.

Q: Who was trying to run the drug trade?
Answers: The gang/the police

#99. I put the cake away in the refrigerator. It has a lot of butter in it.

Q: What has a lot of butter?
Answers: The cake/the refrigerator

#100. Sam broke both his ankles and he’s walking with crutches. But a month or so from now they should be unnecessary.

Q: What should be unnecessary?
Answers: The ankles/the crutches

#101. When the sponsors of the bill got to the town hall, they were surprised to find that the room was full of opponents. They were very much in the majority.

Q: Who were in the majority?
Answers: The opponents/the sponsors

#102. Everyone really loved the oatmeal cookies; only a few people liked the chocolate chip cookies. Next time, we should make fewer of them.

Q: Which cookie should we make fewer of, next time?
Answers: The oatmeal cookies/the chocolate chip

#103. We had hoped to place copies of our newsletter on all the chairs in the auditorium, but there were simply too many of them.

Q: There are too many of what?
Answers: Chairs/copies of the newsletter

#104. I stuck a pin through a carrot. When I pulled the pin out, it left a hole.

Q: What left a hole?
Answers: The pin/the carrot

#105. I couldn’t find a spoon, so I tried using a pen to stir my coffee. But that turned out to be a bad idea, because it got full of ink

Q: What got full of ink?
Answers: The coffee/the pen