X @Avi Chawla

Hardware Memory and Precision Formats - Consumer GPUs with 12GB memory cannot hold a 7B parameter model in FP32 precision because it requires 28GB for weights alone[1] - Quantization reduces memory consumption by trading numeric detail, allowing a 4-bit build of a model to fit in approximately 4GB and run on standard hardware[1] - Halving the number of bits reduces memory by 50%, requiring tradeoffs between the exponent (value range) and mantissa (numeric precision)[8] Technical Specifications of Precision Formats - FP32 costs 4 bytes per parameter, retaining 8 exponent bits and 23 mantissa bits while preserving optimizer states during low-precision runs[3] - BF16 and FP16 utilize 16 bits, where BF16 retains 8 exponent bits with 7 mantissa bits to prevent overflow, and FP16 uses 5 exponent bits with 10 mantissa bits to provide eight times finer resolution up to a limit of 65504[3][4] - TF32 utilizes FP32 exponent and FP16 mantissa internally within tensor cores to accelerate matrix multiplications without altering memory storage[4] - INT8 and INT4 map values to 256 or 16 evenly spaced levels respectively, though outliers can cause standard values to round down to zero[5] - FP8 employs two layouts including E4M3 reaching 448 for weights and activations, and E5M2 reaching 57344 for gradients[7] - NF4 utilizes 16 unevenly spaced levels based on pretrained weight distributions, enabling QLoRA to maintain frozen base models in 4 bits while training adapters in BF16[8] Inference and Memory Optimization - KV cache memory grows with every generated token and can eventually exceed model weight size during long context processing[9] - Inference workloads divide into prefill phases that are compute-bound and process input tokens in parallel, and decode phases that are memory-bound and generate tokens sequentially[10]