Interview with legendary AI reporter Cade Metz: There is no moat in the industry, and China will soon catch up with OpenAI

Author | Tencent Technology Hao Boyang

Image source: Generated by Unbounded AI

The title of "Three Big Three in Deep Learning" is certainly no stranger to readers in the upsurge of AI. But among the deep learning talents, what evaluation criteria are used to single out Hinton, Yann Lecun, and Yoshua Bengio as giants? There are no competitions, debates, or rankings. This title was determined by the most famous reporter in the field of AI, Cade Metz, as early as 18 years ago.

As a senior writer of "Wired" magazine, chief writer of "New York Times" artificial intelligence column, and author of "Deep Learning Revolution", Cade Metz basically interviewed all well-known experts in the field of AI during more than ten years of working experience, Sam Altman consulted with him before launching ChatGPT, and Hinton talked to him after leaving Google. Through contact with these core figures, he also went deep into AI giants such as Microsoft, Google, and Meta to dig out the core moments and dramatic scenes behind various core business decisions. The book "Deep Learning Revolution" written some time ago was also disclosed for the first time. The secret auction that changed the course of artificial intelligence ten years ago.

The book reports in detail the auction that took place in 2012 and had a profound impact on the entire artificial intelligence industry.

In December 2012, Hinton, who was unable to sit still because of a herniated disk, stayed with two of his students for a week at a casino hotel at the foot of Lake Tahoe's ski mountains. He and his newly established company without any products are accepting rounds of bidding from the world's most famous companies, including Microsoft, Google, Deepmind and Baidu.

The most dramatic scene appeared when Hinton was dealing with the sudden visit of Yu Kai, the then deputy dean of the research institute sent by Baidu. In order to "don't let him think that I was old", he asked his students to pack up every time in order to let him Lumbar pain relieved the canopy that was temporarily built with sofa cushions, which made them quite flustered. During a certain visit, Hinton faced Yu Kai’s left backpack and struggled with the students for a long time whether to search for information on Baidu’s reserve price. But in the end, dignity made them give up. Finally, a week later, Hinton accepted Google's offer, sold the company, which was still completely empty, to the giant for $44 million, and opened the curtain of the giant AI war.

Throughout his career, such scenes full of details and dramatic conflicts can be seen everywhere. Therefore, it is not an exaggeration to call him the deepest observer in the field of AI.

In this interview, Metz once again demonstrated his extensive knowledge of the field of AI, from the history and core moments of deep learning to the future of AI and its impact on human society. He also made his observer-style evaluation of China's artificial intelligence situation in particular.

He believes that the reason why Sam Altman can make OpenAI stand out is mainly because its strong negotiation ability has contributed to the cooperation between OpenAI and Microsoft. He also believes that the gap between China and the United States in AI is not as far as the public thinks, because the existing technology itself has no moat. Regarding the future of AI, he has similar worries as Hinton, and he will worry that the emergence of AGI will bring about huge changes in society. Bystanders are clear, the following are insights from AI observers.

Focus on:

• 1 Two important moments in the development of AI, one is the release of the AlexNet paper, allowing the academic community to understand that neural networks can achieve great success in image recognition. Another is the release of ChatGPT, which opened the eyes of the public.
• 2 It is very likely that Chinese companies will soon catch up with OpenAI in terms of language models, as the overall gap in knowledge, funding and resources between the two is not that big.
• 3 The media hype around AI does exist because people don't know much about the limitations of the system. GPT is weak in reasoning and prediction, far below the level of AGI. But the risks of AI at this stage are indeed real.
• 4 If AGI comes, the human condition will be worse. Because their value as workers would then drop to zero, and it would be cheaper to have machines do all this than to hire humans. But the current AI is still far from this goal.

The gap between China and the United States is not that big, AI has no moat

Tencent Technology:

Now that the OpenAI story has come into the spotlight, you've written a lot about its founding and journey in your book. Can you talk about Sam Altman? What qualities enabled him to lead OpenAI to where it is today?

Cade Metz:

**Sam is very ambitious and he is good at convincing people to do what he wants them to do. Frankly, he's good at building teams and putting them on a certain path. **It's not just about the skill level of development. They need a lot of money to do this. Training these types of systems costs tens of millions, if not hundreds of millions of dollars. I've said that analyzing all the text on the Internet and learning, and letting these systems learn on their own, takes months and requires massive computing systems that are actually owned and controlled by large tech companies.

Sam raised over $1 billion in roughly 2019. So a good part of his skill set is negotiating the deal. **It was a huge deal, Sam got the funds he needed, and the computing power he needed. They later raised another $11-plus billion from **Microsoft, for a total of nearly $13 billion. That's a big reason why he's so pivotal.

Tencent Technology:

You mentioned Microsoft, but it was completely behind its competitors in the last wave of AI. Then why can Microsoft seize this opportunity in this wave of AI and act so quickly?

Cade Metz:

This is an interesting move for Microsoft. They were once behind the competition, but they realized the importance of the technology they were building. They have also encountered many difficulties before. His opponent Google and other companies have a scenario where this technology can be used: Google first deployed neural networks on a large scale on Android phones and its digital assistant Google Home for speech recognition. So when you use these smart speakers in your home, they can answer your questions. Google can start deploying this technology so quickly because they have the scenarios and hardware to deploy it. Microsoft does not have this scenario.

But when Microsoft finally tried to deploy it, they ran into other problems. They launched a chatbot called Tay in the US a few years ago, and almost as soon as it hit the shelves it started generating some offensive messages, including racist ones. Microsoft can only take it down soon.

**This is one side of the story as AI technology evolves in the U.S., as these systems sometimes generate unnecessary text that is biased against certain people and generates hate speech. Therefore, it is difficult for large companies to make up their minds to launch this product, and they don't want to take that risk. But Microsoft has partnered with OpenAI, which is launching the system, so that people don't react as strongly to its flaws as they did to Microsoft. ** People will accept that these bugs come from a small and unknown company, but not from a well-known company like Microsoft.

Tencent Technology:

According to your observation, how big is the gap between China and the United States in the field of artificial intelligence? What advantages do you think China has in artificial intelligence?

Cade Metz:

China has been working on this for some time. There are many people and companies who understand the technology involved here, but there can be difficulties. The computer chips needed to train these systems are manufactured at the highest level by American companies. Now there is a trade ban, these chips cannot be sold to China. This can be a disadvantage.

We'll see how it plays out when it comes to building this technology. **Sam Altman said recently that he feels China is two years behind. This is an estimate. I think the industry as a whole can catch up to what he and his company are doing pretty quickly. We're already starting to see that, and I think it can happen in China as well. **

Tencent Technology:

Is this gap getting bigger or smaller?

Cade Metz:

Because of the trade ban, it's hard to tell. This is a difficulty China faces. As far as I know, China is working hard to build, design and supply the chips and equipment needed for data centers.

**Whether in China or in the United States, we will have to deal with the many potential roles played by trade frictions such as competition. It is now generally accepted that OpenAI's technology is currently ahead of most of its competitors. But there are plenty of other companies in the world who have enough knowledge, money, and access to the resources needed to really compete. **

Therefore, we are still in the early stages of AI competition, and there are still many roads to be explored. The result still needs to wait and see.

Tencent Technology:

You mentioned Baidu in your book. The Chinese company also began experimenting with this kind of AI technology in 2012, already in the early stages of the race. Why do you think it didn't show strength today? What do you think are the main contributing factors to this situation?

Cade Metz:

**I think ideas are driving the field. **Deep learning is famous all over the world, and researchers in China have built similar techniques. But just like in the U.S., people were amazed at how well that human feedback training step I mentioned earlier worked. **Thus, they are a step behind many in the US in applying this technique (training with human feedback) to large language models. **

Tencent Technology:

You mentioned that many other companies are also in this race, besides the big ones like OpenAI, Google, and Meta, are there any small companies worth watching?

Cade Metz:

There is a company called Anthropic, which was founded by a group of people who left OpenAI. The company is not well known, but they are going to be important in this space. They helped build much of the technology that formed ChatGPT and built their own chatbot, which has yet to be released to the general public. I estimate that its capabilities will be comparable to ChatGPT.

There is a company in Toronto called Coherence that is doing something similar; in the US there is a company called Character.AI founded by ex-Googlers; there is another company called Inflection AI, founded by one of the founders of DeepMind of. DeepMind is another important lab based in London, essentially owned by Google.

Tencent Technology:

For companies developing in the AI segment, where are their opportunities?

Cade Metz:

The way these small domains work is that once someone builds a system called a large language model, you can use it to build all kinds of other techniques. You can build a chatbot, you can build a search engine, you can build a personal tutor. **So what OpenAI is doing is they built this core system. They call it GPT-4, and then they offer this system to any other business that wants to use it. This is a way to create other applications.

That's what we're starting to see. **I think you'll see companies like OpenAI offering this kind of service. Everyone can use it to build their own applications on top of it. **So I think there are all sorts of opportunities where companies can take this core service and then build new things on top of it and sell those applications. But the most difficult thing is to build this core service, and not many companies can do this. In the United States, there are giants such as Google, Microsoft, and Meta, and only a few startups have the required capital and talent. Like I said before, you need hundreds of millions of dollars to build this core system. **So currently it is difficult for smaller companies to compete with them in terms of building the base model. **

A lot of people think that as prices come down and open source projects improve, it will become easier for people to build their own core systems, and that will eventually change. But I'm not sure.

Two milestones in the development of AI: an auction and the birth of ChatGPT

Tencent Technology:

As a senior AI author, you have basically been focusing on the field of deep learning in your ten-year career, and you have established all the important roles in this field and participated in various decisive moments in between. . What prompted you to pay attention to this field?

Cade Metz:

Around 2011 or 2012, I joined Wired in San Francisco. This is one of the areas I decided to focus on. At that time more than ten years ago, it could already be smelled that this field would become very important. That's when a couple of pivotal moments occurred that got me interested in the technology.

Hinton is the protagonist of the book "The Deep Learning Revolution" and I've written about his rise and the key concept of neural networks that has driven many of the advances in the past 10 years. He eventually joined Google in 2013.

I later learned that this was an auction between some of the world's largest technology companies, including Google, Microsoft, and China's Baidu. That was a pivotal moment, and you could see something starting to happen. Over the years, I started covering the space more and more, and got to know people like Hinton and his longtime colleague Yann Lecun, who eventually joined Facebook and now Meta, among other people in the field. We started doing a lot of coverage in Wired, and then it just got bigger and bigger. Ultimately, I decided to write a book about the field, which I continued to cover when I moved from Wired to The New York Times.

Tencent Technology:

As a reporter who has followed AI for a long time, it is like going through a cycle. You have experienced the trough period in the field of machine learning. artificial intelligence. In your opinion, what made Hinton and others persist?

Cade Metz:

Hinton started working on neural networks in 1972. At that time, almost no one believed that it would succeed, because** the entire field of artificial intelligence abandoned the direction of neural networks. But ** Hinton is a man of his own mind, who really believes in what he believes in, and he is steadfast in that direction.

**By the 1980s, due in large part to Hinton's own efforts, the technology made some major advances. Many people began to believe in this idea again. But in the early 90s, people gave up again, but he continued to work, and he has always maintained his consistent attitude. ** They believed the idea would continue to improve, and they were right. Part of what makes this story so interesting is that they keep working even in the face of so much skepticism, even from their close colleagues.

The moment that really opened the eyes of the entire industry was what is now known as the AlexNet paper. This research paper, written by Hinton and two of his students at the University of Toronto, shows that neural networks can achieve great success in image recognition, identifying objects in photos such as flowers, cars, people, animals etc.

When the paper was published in 2012, it opened the eyes of Google, Microsoft, Baidu, and eventually Facebook. We can see this war for talent, this rush to apply this idea inside some of the largest companies on the planet, not just for image recognition, but for speech recognition, translation, and so on. That paper was a pivotal moment. That's why my book starts with that paper,** that's the moment when Hinton auctioned off his company to the highest bidder, and that sparked everything else. **

From then until today, we have continued to improve. The above-mentioned awakening of industry awareness in 2012 was a critical moment, and 10 years later, ChatGPT was released, which is another critical moment. Both of these moments are very important turning points when we look back at the history of AI.

Tencent Technology:

**In your book, you describe the development of deep learning, and product milestones like AlphaGo, Deepfake, and the GPT series have sparked extensive public discussion. But until the appearance of ChatGPT, everyone really felt that the real industrial revolution was coming. What makes the GPT series so different? **

Cade Metz:

This is a great question. GPT and ChatGPT related technologies have been developed for some time. Several companies have been developing this technique over the past five years, with OpenAI eventually developing Chat GPT, and companies such as Google, Meta (formerly Facebook) and Microsoft also beginning to develop so-called large language models five years ago.

The idea behind the technology is to build a neural network, a mathematical system that can learn from data. Feed it as much text as it can, and it analyzes that text, and in the process of analyzing that text, it learns to generate the text itself. By analyzing Wikipedia articles, blog posts, chat logs and various content on the Internet, it recognized patterns in the way we put words together and learned to do so. We've seen this technology come to fruition over the past few years, and several interesting systems have been released.

OpenAI released GPT-2, GPT-3, they are all very impressive. We can see these systems start to produce text like humans. But it was the release of ChatGPT that really made the general public realize this. Other companies have released their own chatbots even months, weeks ago, like Meta (ex-Facebook) released a chatbot in the scientific community, but it was not only uninteresting to people, it was also criticized ridicule. Because it creates disinformation and people are very upset about it. Meta quickly took it down. But shortly after that, OpenAI released ChatGPT on Twitter.

**It became popular partly because of how it was released, and partly because of the company that released it. But ChatGPT does improve on some key technical aspects. Because in building these large language models learned from all over the internet, they're applying human feedback to them. They asked humans to rate the chatbot's responses. Are they asking humans to rate a response as good? is it real Does it work? They give it a like or a dislike, and then they feed those ratings back into the system and let it learn from those ratings. **

In this way, they were able to hone it to the point that almost every time the chatbot was asked, it produced convincing text. It might not always be true and still generate disinformation, but **it communicates to people in a responsive way that humans actually use. Not just with experts in the field, but with anyone. This really captured people's imaginations. **There has been a real shift in the perception of this technology, not only among ordinary people, but also among many researchers in this field of technology. The popularity of ChatGPT has ushered in a new era of this type of technology and a new race toward increasingly impressive artificial intelligence.

Tencent Technology:

So you think, the technical key factor for the success of ChatGPT is mainly RLHF (Learning with Human Feedback)?

Cade Metz:

Yes, if you use some past version, like **GPT-3, sometimes it can be impressive when you ask it in a certain way. For example, if you ask it to give a speech in the style of Donald Trump, there is about half a chance that it will produce an impressive speech. It's a bit like rolling the dice, sometimes it will give you what you want, and sometimes it won't. In this case the system does not attract the attention of ordinary people. **But they took this basic system, but every time the system produced a response, they had the annotators score it. It can take ratings from humans, see how they rate those responses, and use that to retrain the system. The annotator is telling GPT what kind of answer is good and what kind of answer is bad.

Eventually, OpenAI got a chatbot that can converse like a human almost every time. They put this system in front of ordinary people, and this heated discussion is the way people respond to its emergence. On Twitter, anyone can use it, and people really respond to it.

AGI is still far away, but its arrival will replace all human work value

Tencent Technology:

Some time ago, Musk said in an interview that AGI will be realized within 5 or 6 years. Do you think AGI will be an easy goal?

Cade Metz:

It's really hard to say, and there's a lot of controversy on the subject. **We know that today's AI systems are nowhere near that level. They can produce language in impressive fashion, but they cannot reason like humans, nor do they have the common sense of humans. **

Many people believe that we need new ways to give them this ability, and our current methods cannot make them achieve AGI. They need to know more about the physical world than just language. There's a lot of debate and disagreement about this, but it's certainly not something we need to think about today, nor are we close enough to AGI yet.

Tencent Technology:

What do you think the current system lacks compared to true general artificial intelligence?

Cade Metz:

When these systems are used, the flaws of current AI can be easily found. If you try to make them reason like humans, sometimes they can imitate, but most of the time they can't.

This is the real difficulty. **The biggest difference between them and AGI is that they generate plausible text and can actually reason. **

**These systems are very good at dealing with things that happened in the past, that is to say something that is documented on the internet. But they don't talk about the future and speculate about what might happen. **You and I can have this conversation where we're talking about the future and thinking about what might happen. These systems are not good at doing this. They are good at imitating what they have seen before. So they are very good at passing standardized tests. They do so well on tests like law and high school science and math that they get a lot of hype in the media.

**But other studies have shown that if you only give them completely new questions, questions that were written after they were trained, they don't perform as well. So while they're answering all these standardized questions, they're not necessarily reasoning. **What they do is repeat what was seen before.

Tencent Technology:

Regarding the threat of AI, like opacity, you recorded an interesting passage from Hinton in the book "People need to live with the 'black box' problem, even if you can't see the inner workings, they will do what they are supposed to do" , but recently Hinton quit Google, and you also did an interview with him, in which he expressed a lot of concerns about AI. How do you understand Hinton's change in attitude towards AI?

Cade Metz:

His thinking has definitely changed. When I published the book, he thought the risks of AI were pretty remote. But in the past year, he changed his mind when he saw the ChatGPT technology that we see now.

He came to realize that in some ways systems are more powerful than human brains. **You and I cannot comprehend the entire Internet, it is beyond human reach. We cannot learn from this much data, but the system can. **He is concerned that they are being used to spread disinformation, in this case text that is not real, image that is not real, video that is not real.

**In addition to that, he is also worried about the system starting to take people's jobs; also worried about some of the larger problems, such as automated systems being used on the battlefield, being used as weapons; he is even worried that in a longer period of time, we will lose control of the AI. **

Tencent Technology:

Hinton's longtime colleague Yann Lecun and other AI scientists have really talked a lot lately about the media's exaggeration of AI capabilities and threats. Do you think he is right? In what ways might the media possibly complicate these matters more than they really are?

Cade Metz:

**I think the media does exaggerate. When Chat GPT was first released, people didn't see its flaws, and it took a long time before they realized that it generates misinformation to mislead people. Many people, including journalists, have a hard time understanding what's going on. Therefore, They continue to mislead people as they spread. **In this case, over-hyping can easily happen.

Of course, sometimes deliberate exaggeration comes from misunderstandings, but some people hype on purpose and others unconsciously. But I think it's mostly because people don't fully understand what they're seeing.

Tencent Technology:

What do you think is the historical significance of the current AI boom? Is it the beginning of a new industrial revolution?

Cade Metz:

It is possible. I think we'll continue to see improvements to these systems. **They will be able to handle not just text, but also images. We've seen the latest version of technology built by OpenAI take in an image and describe what's in it, answering questions about it. This isn't public yet, but it's part of what they've built. It promises that more and more skills will be similar to humans and continue to change people's jobs.

So I think what we've seen in the past six months suggests that we're going to have a really big shift in the next few years. This transformation does look like it will be as far-reaching as the first industrial revolution. We are still in the early stages. These things often go slower than people think. But I think we're headed in that direction.

Tencent Technology:

How do you think AI will reshape this society? Where is the value of human beings when AGI is realized?

Cade Metz:

I think it would be a difficult situation for humans to actually have a system that can do anything the human brain can do, so-called AGI.

**If a machine could do everything a human can do, then the value of a worker would drop to zero because it would be cheaper to use a machine than to hire a human. In my opinion, this is not a good situation for human beings. ** But AI is not there yet.

Take computer programmers as an example. Today, the system can generate computer programs and computer codes very well, but the code may still have defects. It still needs an experienced human programmer to take over the code they generated, edit and integrate it. into a larger program, eventually building an application. As these systems get better over time, they will replace more and more of what humans do. The problem at this stage is that AI will soon start to replace the less experienced junior programmers, the people who are at the bottom of the programmer industry, who do the same thing as AI, write some basic code and hand it over to more experienced programmers. people to integrate.

** As AI continues to improve, machines will certainly change things more and more, but we are not at the point where AI will suddenly replace a large number of jobs. **

Another concern I have is that these systems can generate realistic text and images. They're even starting to generate video on the fly, we won't be able to tell what's real from what's not on the internet, and will absolutely have to change our mindset when browsing almost anything on the internet. You have to ask the masses if they have the power to change the way they think in general. **