The release of GPT-4, which has been hailed as a significant improvement over GPT-3 prompted me to finally create an account and see what the hoopla is all about. I haven’t stretched for a Plus upgrade, so maybe I’m still on an older not-the-latest-and-greatest version. Not sure what’s behind the curtain. But I’ll work with what I have.
First, I wanted to check if GPT knew anything about me so I asked if it could provide information on Hufflepuff Hippo, and whether it gets anything from my blog. GPT is very polite, and seems to have no info on me.
GPT: I'm sorry, but I don't have any information about a "Hufflepuff Hippo." Could you please provide more context or clarify what you're referring to so that I can better assist you?
You’ve likely heard about GPT getting things wrong. That’s because it’s about pattern-matching and prediction, so that’s not a surprise. Then again, isn’t part of how humans learn also through pattern-matching and then testing predictions? We don’t know exactly how that works, and it’s likely not the whole story. In any case, since I study the origin of life, I wanted to see what GPT knew about it so I asked it to give me the main theories for the origin of life and rank them. GPT assiduously avoided trying to rank them and kept repeating that there is no scientific consensus. Here’s one of several similar responses when I tried asking in different ways to coax it to give its best guess based on the available evidence.
GPT: As an AI language model, I cannot provide a definitive answer to which theory is the most plausible as there is no scientific consensus on the origin of life. Each theory has its strengths and limitations, and the scientific community continues to explore and test multiple hypotheses to better understand how life emerged on Earth. The origin of life is a complex and multifaceted topic that involves many interdisciplinary fields, including chemistry, biology, geology, and astronomy, and it remains an active area of research.
So I tried a different tact and asked it about the weaknesses of each theory. GPT duly listed a number of them in general terms, which seemed reasonable. Here’s one of them:
GPT: It is challenging to explain how complex organic molecules could have formed under the conditions of the primitive Earth, as the atmosphere was likely reducing rather than oxidizing.
I followed up by asking: “Why is it challenging to explain how complex organic molecules could have formed under the conditions of the primitive Earth, as the atmosphere was likely reducing rather than oxidizing? Isn't it easier to synthesize complex organic molecules under reducing conditions?”
GPT: You are correct that reducing conditions can be favorable for the synthesis of organic molecules, but the challenge with explaining the origin of life under reducing conditions is that the early Earth's atmosphere was likely different from the present-day atmosphere. The primitive Earth's atmosphere was likely composed of gases such as methane, ammonia, hydrogen, and water vapor, rather than the oxygen-rich atmosphere we have today. Under these conditions, it is still unclear how complex organic molecules, such as amino acids and nucleotides, could have formed. One of the main issues is that the reducing atmosphere would not have provided the necessary oxygen to drive the formation of these molecules. Additionally, some of the most plausible mechanisms for the formation of organic molecules, such as the Miller-Urey experiment, have been challenged in recent years, suggesting that the conditions of the primitive Earth may have been more complex than previously thought.
The response is interesting. GPT says I’m correct. Then it provides information about the primitive atmosphere from the point of view of the Miller-Urey experiment. But then it errs by saying that “the reducing atmosphere would not have provided the necessary oxygen to drive the formation of these molecules.” That’s backward. GPT thinks that oxygen is needed to drive chemical reactions, which is true when you’re burning fuels for example, but it’s not generally true. After more back and forth where I give it more info, I remark: “It sounds like I'm giving you the relevant information rather than the other way around. Is that what's happening here?” GPT responds very politely again about what it can do and how it will try its best to be of help.
I’m curious whether it can generate chemical reactions, so I ask: “Can you provide a chemical equation for this experiment that shows how amino acids were formed?” GPT demurs by saying the Miller-Urey experiment doesn’t provide this, so then I ask: “Do you provide chemical equations for any chemical reactions?” GPT is measured in its response.
GPT: Yes, as an AI language model, I can provide chemical equations for various chemical reactions. However, the level of detail and complexity of the equations will depend on the specific reaction and the information available. If you have a specific reaction in mind, feel free to ask me, and I will do my best to provide you with a relevant chemical equation.
Now let’s try it out based on info from the answers GPT had provided. I ask: “How would CH4, NH3, H2O and H2 lead to an amino acid such as glycine? Could you provide a chemical equation?” GPT provides a bunch of text that I’m not showing here that repeats things you might find from a Wikipedia page, but it also obliges with the following:
GPT: Here is an example of a chemical equation that shows one possible pathway for the synthesis of glycine from CH4, NH3, H2O, and H2:
· CH4 + NH3 + 2O2 → HCN + 3H2O
· HCN + H2O + NH3 → glycine + CO2
Me: “The second equation isn't balanced. Could you balance it for me?”
GPT: I apologize for the error in the previous equation. Here's the balanced equation for the formation of glycine from HCN, water, and ammonia: HCN + NH3 + H2O → NH2CH2COOH
Me: “Um, it's still not balanced. There are different numbers of atoms of different elements on each side.”
GPT: I apologize again for the mistake. Here's the fully balanced equation for the Strecker synthesis of glycine from HCN, water, and ammonia: HCN + NH3 + 2H2O + O2 → NH2CH2COOH + CO2
Notice how O2 now shows up as a reactant. And it’s still not balanced and GPT doesn’t actually understand why. But to its credit it did pull up the chemical formula of glycine. GPT can balance simpler reactions and it can tell me the predicted products for HCl or H2SO4 reacting with NaOH. How much chemistry does GPT know? I’m not sure. OpenAI claims that GPT-4 performs much better on the AP-Chemistry exam than its predecessor among other things (see graph below from OpenAI’s website touting GPT-4).
Why does GPT behave the way it does? I’ve read a number of articles over the last couple of months about this, and the one I like the most that’s also a quick read is Tim Harford’s “Why chatbots are bound to spout BS”. I like his point that GPT aims at sounding plausible. When pressed for details, that’s when it can deliver nonsense. Its aim is neither truth nor falsity. It says it’s trying to be helpful. Is being plausible helpful? I suppose that depends. I’ll quote the last few sentences of Harford’s article because they’re a good reminder.
These simple chatbots [ELIZA and others] did enough to drag the humans down to their conversational level. That should be a warning not to let the chatbots choose the rules of engagement.
Harry Frankfurt cautioned that the bullshitter does not oppose the truth, but “pays no attention to it at all. By virtue of this, bullshit is a greater enemy of the truth than lies are.” Be warned: when it comes to bullshit, quantity has a quality of its own.
That being said, I think I will go chat some more with GPT. I have some ideas of how I might be able to leverage it as a classroom tool but I’ll need to do more tests to figure out what it can do well and what it can’t. Onward!
No comments:
Post a Comment