Deepseek Deepseek DeepSeek!
While everyone is talking about how DeepSeek has taken the US market by storm, how Chinese AI is outperforming US AI models, and how DeepSeek has built a competitive (or even superior) model to OpenAI with less time and a smaller budget, let's set aside these clickbait, politics, and world news.
Instead, let's focus on how DeepSeek made this possible. What innovations did they introduce, and which features set them apart from OpenAI and other models? Let's dive in…
The motivation behind Deepseek: Reasoning is a cornerstone of human intelligence. While current AI models demonstrate impressive reasoning skills, their development often depends entirely on supervised fine tuning (SFT) with extensive labeled datasets. This approach, though effective, is not without its limitations.
DeepSeek R1 Zero has completely bypassed supervised fine tuning (SFT) and reimagined reasoning in LLMs through pure reinforcement learning (RL):
Current AI models high dependence on SFT: OpenAI's language models need datasets where the data is carefully labeled or annotated with the correct answers or information. These high quality annotations help the model learn specific patterns, relationships, or tasks more effectively.
In simple terms, these labeled datasets act as teacher, showing the model exactly what to learn. The better the quality of these "lessons," the better the model performs.
Deepseek Bypassed SFT: Deepseek completed bypassed this SFT by Reinforcement learning. Reinforcement Learning (RL) is a type of machine learning in artificial intelligence where an agent learns to make decisions by interacting with an environment to achieve a specific goal. Instead of being explicitly told what actions to take, the agent learns through trial and error by receiving rewards (positive feedback) or penalties (negative feedback) for its actions.
By completely bypassing the SFT, which makes data collection and curation costly, time consuming, and labor intensive, the process becomes more scalable and efficient.
Deepseek’s Group Relative Policy Optimization (GRPO): In RL, there are two key types of models: the actor and the critic.The actor decides what action to take (like a player in a game).The critic evaluates how good that action was (like a coach giving feedback). The GRPO directly focuses on improving the actor. It uses clever tricks to estimate how good actions are directly from the actor's own experiences, without needing a separate critic.
Imagine training a player who learns how to play just by replaying and analyzing their own games, without needing a coach to tell them what they did right or wrong.
DeepSeek R1 incorporates a small amount of cold start data and follows a multi stage training pipeline.
Current models are not always cold start: Cold start fine tuning in AI refers to the process of training a model from scratch or with minimal prior knowledge. Fine tuning models at OpenAI can lead to challenges like carrying over biases from pre training, overfitting to specialized data, and difficulty generalizing to other contexts. It also raises privacy concerns when using proprietary data, is resource intensive, and may not achieve the same level of expertise as a model built specifically for a domain.
Deepseek’s Cold Start data: By, this way if you're working with a very niche or unique dataset that the model hasn't seen before, cold start fine tuning lets the model learn from scratch and reduces the risk of carrying over unwanted patterns or biases from previous tasks. Curated dataset with readable long Chains of Thought (CoT) is used to fine tune the base model. So, the dataset is designed to teach the model how to think in a more structured and understandable way, helping it make better decisions and explain things clearly.
DeepSeek doesn’t stop with large models; it extends its capabilities to smaller models using distillation techniques.
Deepseeks Inovative distillation technique: In AI, distillation techniques are like teaching a smaller, simpler version of a model (called the student) to learn from a larger, more complex model (called the teacher). The teacher doesn’t just transfer raw data—it shares how confident it is in its predictions, insights, patterns it has learned to the student models like Qwen and Llama series. DeepSeek R1, which is a large AI model, has generated 800k high quality examples (training samples). These examples contain useful information for the model to learn from.
DeepSeek R1 Distill Qwen 32B surpassed OpenAI’s o1 mini in reasoning tasks, with a Pass@1 score of 72.6% on AIME 2024.
Vision for the Future:
DeepSeek aims to push the boundaries of what reasoning models can achieve, not just by improving their capabilities but also by making them accessible and reliable across industries.
