The Transformer Revolution: How a Google Researcher’s 2017 Idea Sparked Generative AI and a Biotech Boom
The rise of generative AI, particularly ChatGPT, has captured the world’s attention, but the technology’s roots lie in a research paper published in 2017 by a group of Google researchers. Jakob Uszkoreit, one of the authors, reveals that the revolutionary transformer architecture, which powers these AI models, was initially developed to speed up the training of neural networks. The impact of this idea extends far beyond the realm of language models, as Uszkoreit himself is now using it to revolutionize drug development through his biotech startup Inceptive.
Key Takeaways:
- A 2017 paper by Google researchers introduced the transformer architecture, providing the foundation for generative AI. This groundbreaking work, titled "Attention Is All You Need," laid the groundwork for powerful AI models capable of generating text, images, and code.
- Google’s internal focus may have limited the initial public impact of transformers. While the technology was deployed internally, OpenAI’s ChatGPT brought the capabilities of transformers to the forefront.
- Uszkoreit, the co-founder of Inceptive, is leveraging generative AI to create "biological software." The startup, which recently secured $100 million in funding, is aiming to design RNA molecules using AI, revolutionizing drug development and potentially leading to medicines that are more harmonious with living systems.
The Transformer’s Journey from Google Labs to the World Stage
In 2017, Uszkoreit, then a researcher at Google, proposed a new approach to interpreting data – self-attention. This enabled the creation of the transformer, a neural network architecture that has since become the cornerstone of generative AI. While the transformer’s potential was recognized early on, its deployment was largely confined within Google, where it was used in a variety of applications.
"There are actually applications, for example at Google and other places, where transformers have been deployed in production long before, but to much, much less fanfare," Uszkoreit explained. The emergence of OpenAI’s ChatGPT, he says, "shined the spotlight on these applications."
Google’s Internal Focus vs. OpenAI’s Daring Innovation
Uszkoreit acknowledges that Google’s focus on internal applications may have hindered the wider adoption of transformers: "Maybe Google here hasn’t been able to be as daring as, you know, a much, much smaller company such as OpenAI when it comes to applying this technology to quite different types of products." However, he recognizes the valuable contribution Google has made: “This is something that we fundamentally have to accept and actually, in a certain sense, be maybe even grateful for because Google is providing something to the world that we all rely on day to day.”
Biotech Revolution: Inceptive’s Generative AI-Powered Drugs
In 2021, Uszkoreit left Google and co-founded Inceptive, a biotech startup focusing on developing drugs through generative AI. The company describes itself as a “biological software company,” applying AI to design and engineer molecules for therapeutic purposes. Their current focus is on RNA, using AI to design molecules that behave in specific ways within the body.
"We’re starting with a focus on RNA, whose exact composition has been designed with generative artificial intelligence, such that these molecules inside certain biological systems exhibit behaviors that ultimately are native to those systems," Uszkoreit explained.
Inceptive raised $100 million in funding, led by Andreessen Horowitz and Nvidia, highlighting the immense potential of their generative AI approach to drug development. Uszkoreit believes this technology could lead to a new era of medicines that are more harmonious with the body, addressing current limitations of traditional pharmaceuticals.
The Future of Generative AI: Beyond Language Models
The impact of Uszkoreit’s work goes far beyond the development of a language model like ChatGPT. The transformer architecture, initially designed to improve machine learning efficiency, has fundamentally reshaped the landscape of artificial intelligence. Its application in drug development, spearheaded by Inceptive, suggests that generative AI is poised to revolutionize diverse fields beyond language, image, and code generation. The future of this technology, as Uszkoreit puts it, is "to make medicine work better and with greater and greater harmony."
This story is a prime example of the convergence of innovative technologies, with revolutionary research emerging from the depths of Google’s labs and finding its way into diverse fields like biotech, impacting the future of medicines and healthcare.
