Training Methodology - Large Language Models initiate from randomly initialized weights where token selection remains near uniform, resulting in the generation of gibberish text [2] - The pre-training stage consumes nearly all training compute by processing a large text corpus to repeatedly predict the next word, imparting grammar, world facts, and raw reasoning abilities [2] - The model acts as a text continuation engine during pre-training rather than directly answering questions [1][3] Fine-Tuning and Alignment - Instruction fine-tuning utilizes pairs of instructions and good responses to teach the model formatting conventions, such as answering questions or summarizing requests [3] - OpenAI leveraged roughly 13,000 human-written examples for GPT-3 during instruction fine-tuning, compared against a pre-training corpus comprising hundreds of billions of words [3] - Preference fine-tuning employs human evaluators to select the superior output between two responses, training a separate reward model to update the main language model [4][5] Advanced Reasoning and Limitations - Reasoning fine-tuning applies programmed correctness checks for domains like mathematics and code without requiring human preference signals [6] - DeepSeek trained the R1 model using GRPO, allowing emergent behaviors such as intermediate step processing and self-correction to manifest spontaneously during training [8] - Stage 4 reasoning fine-tuning remains restricted to domains where programs can automatically verify answers, contrasting with support replies or summaries that lack a deterministic correct output [8][9]

Pre-training Data Composition and Paradigm Shifts - Traditional base models relied heavily on general human knowledge, with WebText and Wikipedia comprising roughly 85% of the training mix in GPT-3 [5] - Recent models like Llama 3 allocate about 50% of their training tokens to general knowledge [5] - The proportion of raw web text has significantly decreased, dropping down to 15% in newer model architectures [12] - Early pre-training recipes lacked specific code data sets, whereas code has now become a dominating data subset [16] - Industry practices now incorporate supervised fine-tuning (SFT) and question-answer chat data early into the pre-training phase rather than keeping it strictly for post-training [14] Reinforcement Learning and Compute Allocation - Industry adoption of reinforcement learning (RL) has shifted from being a minor enhancement to dramatically improving model performance on diverse tasks, pioneered by models such as OpenAI o1 in 2024 and DeepSeek R1 in January 2025 [8] - Xiaomi's Mimo Labs allocates roughly equal compute budgets between pre-training and post-training in the final model, while some architectures sink significantly more compute into RL than supervised learning [23] - Base models are increasingly viewed as providers of atomic skills and reasoning priors to build effective foundations for large-scale reinforcement learning and agentic behavior rather than acting as static web text repositories [24][28]