- A Microsoft engineer working on the GPT-6 training cluster revealed that the company cannot physically colocate more than 100,000 NVIDIA H100 GPUs in a single U.S. state without overloading the regional power grid, forcing them to distribute the cluster across multiple regions.  
- This power constraint is driving Microsoft to provision Infiniband-class interconnects between geographically separated GPU pods, creating significant engineering pain around low-latency, high-bandwidth networking.  
- The anecdote underscores a new scaling bottleneck for frontier AI training: electrical grid capacity rather than raw silicon or capital, highlighting infrastructure limits on the path to larger models.