Artificial Intelligence Companies Turn to Synthetic Data for Training AI Models
Artificial intelligence (AI) companies like Microsoft, OpenAI, and Cohere are exploring the use of synthetic data to train their AI systems known as large language models (LLMs). With the increasing demand for data to develop powerful generative models, these companies are turning to computer-generated information as they reach the limits of human-made data.
The launch of OpenAI’s ChatGPT last year has sparked a wave of similar products from companies like Google and Anthropic. These products can produce text, images, or code in response to simple prompts, driving investor and consumer interest in generative AI technology.
Currently, LLMs are primarily trained by scraping the internet for data, including books, news articles, blogs, search queries, social media posts, videos, and images. However, AI companies are running out of easily accessible and high-quality data while facing scrutiny over privacy concerns.
To address these challenges, companies are turning to synthetic data. This approach allows AI models to generate text, code, or complex information related to various domains such as healthcare or finance. The synthetic data is then used to train advanced LLMs, making them more capable.
According to Aidan Gomez, CEO of Cohere, synthetic data is already being used by several companies in the field. For example, Cohere uses two AI models to have a conversation about trigonometry, with one acting as a tutor and the other as a student. Humans then correct any mistakes made by the model.
Recent studies from Microsoft Research have shown that synthetic data can be used to train models that are smaller and simpler than state-of-the-art software. For instance, one study used synthetic data generated by OpenAI’s GPT-4 to train a simple LLM capable of producing fluent and grammatically correct stories.
Start-ups like Scale AI and Gretel.ai have emerged to provide synthetic data as a service. Gretel, founded by former US intelligence analysts, works with companies like Google and HSBC to augment their existing data with synthetic versions for better AI model training.
While synthetic data offers numerous benefits, critics warn that not all synthetic data will accurately reflect or improve real-world data. Training AI models on poor synthetic data could lead to degradation over time, producing regurgitated knowledge without any new insights.
Despite the risks, AI researchers believe that synthetic data has the potential to accelerate the development of superintelligent AI systems. The goal is to enable models to teach themselves, ask questions, discover new truths, and create their own knowledge.
In conclusion, the use of synthetic data is becoming increasingly prevalent in the AI industry as companies seek alternative ways to train their models and overcome limitations in human-made data.