Launched in December, ChatGPT is breaking records in popularity and becoming applied in education, business, and everyday life. Researchers from Wrocław University of Science and Technology have decided to test whether the new chatbot was indeed capable of answering any question.
ChatGPT is a generative language transformer developed by OpenAI, which is theoretically able to answer any question posed to it while maintaining a high degree of language accuracy.
A group of researchers from the Faculty of Computer Science and Telecommunications as part of the CLARIN project decided to test whether artificial intelligence is really that versatile. Their experiment involved asking it almost 40,000 questions falling into 25 categories. They published the results of their observations in a scientific paper entitled
Michał CiepielskI: The world has been enthralled by the new chatbot, with many talking about a breakthrough in the development of artificial intelligence and new possibilities for putting it to use. Did a new digital revolution begin in December 2022?
Prof. Przemysław Kazienko (Department of Artificial Intelligence, Wrocław Tech): From a technological point of view, it isn’t new. Research into artificial intelligence has been ongoing for several decades. Also, we have been using artificial intelligence for a long time, for example with the facial recognition system in our smartphones or the advanced adaptive systems in our cars. ChatGPT is based on GPT-3 – a huge language model published in 2020. Its first version was developed in 2018. These dates are important because they show the incredibly high dynamics of the development of these types of models. New solutions will emerge faster and faster and, in this respect, I believe that the artificial intelligence revolution has just begun.
Is it comparable to the industrial revolution?
Yes, it will certainly be just as big and bring just as much change. One of the ground-breaking things is the fact that the ChatGPT model was used by some 100 million users in just one month of being made available. Now the number of its users must be more than 200 million. It has also met with great interest from businesses, which have also already started to use it for their operations.
You mentioned the GPT-3 model on which ChatGPT is based. What is it?
It’s a large language model that consists of 175 billion parameters. It was used for a variety of purposes, e.g. content generation or textual content inference, but the results obtained weren’t accessible enough. This is why OpenAI decided to develop a new solution, which is precisely ChatGPT. Its parameter base is around 3.5 billion, but it’s very impressive in terms of language correctness and the tone of the content it generates. But, of course, it does make linguistic errors, especially in Polish, which is one of the more difficult languages.
And how has it been possible to reach this level?
The model uses human knowledge. OpenAI hired a lot of people whose job it was to evaluate and select the best versions of the answers generated by artificial intelligence. The software kept learning from it, preparing increasingly correct answers. Today, millions of people using ChatGPT, asking it questions, correcting it, and engaging in conversations with it provide the company with human feedback (known as Human in the Loop) and thus help it grow. This time, it’s for free.
What sources do these models use for preparing their responses?
GPT models have indexed Wikipedia, public websites, complete databases of texts and books available, including scientific ones, as well as blogs and news sites, among other sources. In total, it amounts to some 45 TB of data. It should be pointed out, however, that the data isn’t pulled in real-time, so you won’t get an answer about current events until it has been reprocessed. This is because ChatGPT can’t create knowledge from scratch on its own, although such models already exist. Instead, it only uses available data. As it prepares answers to questions, it searches for relationships between words, phrases, and expressions, using a deep neural network that works based on a principle similar to how our brain functions.
What do you want to find out through your research?
ChatGPT inspired our research primarily because it has become socially popular and, by and large, one could say that it actually works well. We made more than 38,000 queries for the AI, falling into 25 different categories. What we tested included aspects such as how it reacts to sarcasm, whether it understands jokes, and whether it can grasp a wider context of statements. We compared the quality of the responses with the performance of other, state-of-the-art, specialised natural language processing models. Remember that ChatGPT doesn’t focus on one area but generates answers on almost all topics. It, therefore, gravitates towards artificial general intelligence (AGI), which mimics broad cognitive abilities.
And what were the results?
Compared to other language solutions, it was inferior but not hopeless. It turns out to be a good tool for finding answers quickly, but if high-quality inference is important to you, the results might not be good enough. What appeared to be ChatGPT's strengths is its context awareness and ability to personalise statements. On the other hand, it was unable to cope with a quick change of topic, to give you an example. We’ve already started further research, in which we will have a stronger focus on emotions. It’s a much more difficult issue that even specialised language models find problematic.
If ChatGPT isn’t so good, are concerns about students using it to do their homework, for example, justified?
It’s one of the issues we will have to deal with. It seems to me that we need to learn to use solutions of this type as opposed to limiting them. They can, of course, be blocked in exams or when knowledge is tested in another way, just as the case is with smartphones, but on the other hand, it’s probably worth incorporating them into the teaching process. The question is whether the knowledge we are presented with is now infallible. For instance, during our research, ChatGPT confused the kings of Poland with one another. So, there may have to be specialised solutions for particular fields of knowledge, in this case, Polish history.
Can you see any other risks?
Very serious ones. Since ChatGPT provides us with knowledge and information, why use the sources? Now, when you look for information, you use a search engine and visit the site you’re interested in and so the author can make money from it through advertising or through the right to access it paid for by the user. If ChatGPT has indexed everything and now gives you the answer straight away, why go to other websites? The creators no longer make money, and the whole business model falls apart. The question then remains as to what and how much will need to be changed.
It seems that the social sphere will also be affected by the changes.
Certainly. We’re already increasingly surrounded by systems that make decisions for us, but most importantly, they do it well and their suggestions appeal to us. Examples include a GPS that shows you the way or an online store displaying product recommendations that cater to your needs. From a certain point onwards, the decisions we make are no longer our own. It brings us to the question of freedom of choice. It’s also worth remembering that you lose a skill by not using it. A good example of this is the already mentioned GPS navigation, as a result of which people are increasingly losing their orientation skills.
There are many new problems and challenges ahead, and given the pace at which technology is growing, we won’t have that much time before we face them.
We need to restructure our role in the modern world. Revolution has always torn down much in its path, eating its children in the process. Our social reflection has never kept pace with progress, and yet humanity has always managed somehow. I presume it will be the same this time. The artificial intelligence revolution is also a great new opportunity.