Much of the discussion around AI infrastructure is currently focused on scale. New projects are increasingly being discussed in hundreds of megawatts, with some proposed AI campuses extending into gigawatt territory. There are good reasons for this. Training increasingly capable foundation models requires enormous amounts of compute, and concentrating thousands of GPUs in a small […]

Read More

When discussing AI inference infrastructure, the word “scaling” can mean several very different things. We might need more GPUs because a model is too large for a single GPU, more copies of a model because concurrency is increasing, or better routing between those copies because repeated context is consuming unnecessary compute. Eventually, we may also […]

Read More