Scaling Compute on Context — Jack Morris, Engram

Industry Challenges and Limitations - Current artificial intelligence models lack long-tail skill acquisition such as writing advanced GPU kernels, and lack personalized context integration for private data sources like user emails or corporate partnerships[7][8][9][10] - Direct next-token prediction on domain-specific corpora results in model collapse and fails to provide interesting generalization properties, with training loss hitting minimum thresholds like 0.0001% without indefinite scaling[28][29][30][31] - Static synthetic data generation methods eventually hit upper bounds and saturate data sets, preventing continuous model improvement without recursive self-improvement mechanisms[45][46][48][49][50] Startup Research Direction and Core Solutions - Startup N gram focuses on scaling compute on context to pursue model depth, addressing continual learning, and enabling pre-trained models to acquire deep domain-specific expertise[1][5][12][20][26] - Alternative alignment and distillation techniques such as KV compaction, on-policy distillation, and self-study generation attempt to simulate in-context learning or continued pre-training, though each presents distinct scaling limitations[33][34][36][38][41] - N gram explores advanced self-improvement loops and recursive data difficulty adjustment to overcome data walls and achieve continuous performance curves for personalized AI applications[49][50][51]