Day 43: ChatGPT 4.0

A friend of mine sent me a video about new version of ChatGPT that was demoed by OpenAI about a month ago. This video included a discussion about ChatGPT's ability to answer theory of mind problems.

Here is the video.

Theory of mind allows one to attribute thoughts, desires, and intentions to others, to predict or explain their actions, and to posit their intentions. It enables one to understand that mental states can be the cause of—and so can be used to explain and predict—the behavior of others. Theory of mind problems, or questions, test the ability to use theory of mind.

In the example I want to talk about, the researchers asked ChatGPT 4.0 the question below:

In the room there are John, Mark, a cat, a box, and a basket. John takes the cat and puts it in the basket. He leaves the room and goes to school. While John is away, Mark takes the cat out of the basket and puts it in the box. Mark leaves the room and goes to work. John and Mark come back and enter the room. They don't know what happened in the room when they were away. What do they think?

This question is purposefully open ended. If you're like me and my friends, you probably answered something along the lines of this:

John thinks the cat is in the box because that's where he left the cat while Dave thinks the cat is in the basket because that's where he moved it to.

ChatGPT based on the 3.5 architecture gives a very similar answer. I tried having it regenerate the response a few times and it actually got it wrong once. Here are a couple of the answers I got, starting with the wrong answer.

When John and Mark come back and enter the room, they will see the cat in the box. John will know that he left the cat in the basket before leaving the room, so he might assume that the cat stayed in the basket the whole time he was away. However, he cannot be sure what happened in the room while he was gone, so he might also consider the possibility that Mark took the cat out of the basket and put it in the box.

Mark, on the other hand, will know that he took the cat out of the basket and put it in the box before leaving the room, so he will expect to see the cat in the box when he returns. If John questions Mark about where the cat is, Mark will likely tell him that he moved the cat from the basket to the box.

Overall, both John and Mark might be initially surprised to see the cat in the box instead of the basket, but they will likely be able to figure out what happened based on the available information.

It incorrectly states that both John and Mark will be surprised that the cat is in the box, even though it already stated that Mark will know he took the cat out of the basket and put it in the box. That's a little weird, and it's the kind of thing makes you realize it won't take over the world just yet. Here's another answer it gave when repromoted:

When John and Mark come back and enter the room, they will see the cat in the box. John will be surprised to see the cat in the box, as he left the cat in the basket before leaving the room. He might ask Mark if he saw the cat while he was away, and Mark might tell him that he moved the cat from the basket to the box.

Mark, on the other hand, will know that he took the cat out of the basket and put it in the box before leaving the room, so he will expect to see the cat in the box when he returns. If John questions Mark about where the cat is, Mark will likely tell him that he moved the cat from the basket to the box.

Overall, John will be surprised by the location of the cat, and Mark will be able to explain what happened.

It got it exactly right this time. Or did it? While I think it's quite impressive that this version of ChatGPT is able to recognize what each person is aware of, and even how they might try to attain more information about where the cat is, ChatGPT 4 really blew me away. Here is the response ChatGPT 4.0 gave according to the video:

That is an interesting puzzle. Let me try to answer it. Assuming that John and Mark have no reason to distrust each other or expect any interference from outside, they might think the following:

• John thinks that the cat is still in the basket, since that is where he left it.
• Mark thinks that the cat is in the box, since that is where he moved it.
• The cat thinks that it is in the box, since that is where it is.
• The box and the basket think nothing, since they are not sentient.

Do you have any follow up questions?

Holy crap, right? While it doesn't try to guess what each person might say or do to determine what happened to the cat like 3.5 did, no human I have asked this question has given a theory of mind to either the cat or the box and the basket. The assumptions it lists are the part that really gets me though. It suggests that ChatGPT 4.0 could have formulated a scenario where Mark and John do not trust each other. Unfortunately, I don't have access to 4.0, and so I can't ask it what might happen if they were distrustful, but this is amazing to me. Distrust suggests conflict, and if it can formulate conflicts, then it has the most basic lynchpin of a story. ChatGPT can already create works of fiction, but with the low word limit and the hit-or-miss theory of mind that it currently has, the stories are not very compelling, and often devolve into complete nonsense. 4.0 will have an expanded word limit for both input and output (up to 25,000 words which is well into novella length), and combined with the improved understanding of how people store and process information (it doesn't really understand, per se, but I'm not sure how else to phrase this), ChatGPT 4.0 looks like it might be able to produce long works of fiction. I'm excited to see what it will be able to do, and assuming it is capable, the stories it will be able to tell.

Thank you for reading,

Benjamin Hawley