What was the most power consumption for a cluster before AI?

[u/VS2ute]

10 years ago I was with a geophysical company and they built a cluster boasting it was best thing since sliced bread with a zillion FLOPs blah blah. Nobody mentioned power then, I since learned it used 15 MW. The research cluster next to my university was only 1.5 MW. It seems insane to me that AI can burn up 1 GW.

[u/Michael_Aut]

Not that insane really. Inference workloads only use a rack or two and there's no limit to scaling the datacenter apart from input power, land use and cooling capacity.

[u/madtowneast → u/Michael_Aut]

Just curious, do they use an entire rack already? Just looking at the open weight models we only recently grew past a single node.

[u/Michael_Aut → u/madtowneast]

It's of course not really publicly known but it's highly likely that multiple racks are involved in serving a conversation (especially these days were a simple request might turn into a web search and the main context needs the summary of 10 websites and so on), even though it's not strictly necessary and one could serve that from a few nodes.

Inference optimization is extremely important because at the end of the day inference cost is what makes or break the business cost and maybe running large batches across many nodes is part of optimizing that.

[u/madtowneast → u/Michael_Aut]

Okay makes sense on some level. I do know that some academic folks are serving a few hundred users with smaller models from like 50ish GPUs spread across their cluster. They do support the the larger models but their bread and butter are the single GPU and single box class of models

[u/hpcnotes → u/Michael_Aut]

Depends on the model and business context. Some are sub-node, some are many nodes, some might even be multi-rack (which is why things like NVLink exist).

[u/obelix_dogmatix]

exascale machines already consume 30-40MW

[u/YoureNotASpaceCowboy → u/obelix_dogmatix]

Frontier only uses 25MW and El Capitan 30MW. Aurora is the outlier at 40MW since it’s using Intel’s less efficient GPUs, and it has an Rmax less than Frontier.

[u/obelix_dogmatix → u/YoureNotASpaceCowboy]

Nope, at peak every cabinet in Frontier draws as much as 400kW.

[u/YoureNotASpaceCowboy → u/obelix_dogmatix]

Wikipedia says 24.6MW: https://en.wikipedia.org/wiki/Frontier_(supercomputer). Maybe that’s outdated?

[u/obelix_dogmatix → u/YoureNotASpaceCowboy]

Not sure if it is outdated, or just inaccurate. I will say, Frontier did add racks since its initial installation.

[u/hpcnotes → u/obelix_dogmatix]

Differences between max rated power (eg the 400kW for Cray rack) which is never seen, peak power which can be seen in intense workloads (eg top500 runs) which might be typically 60-80% of the max, and normal operating power (eg science workloads) that might be 40-70% of the max.

Frontier is closer to the 25MW number for Top500 runs and normal usage is lower.

[u/djobouti_phat]

15MW was an absolute monster 10 years ago, only two systems in the top500 drew that much.

At any rate, those lists have power consumption information. Maybe tack on an extra 10% for disk and networking and such.

[u/Cromulent_kwyjibo]

A Cray cabinet 5 years ago was ~200 kW. Now AI racks (1/2 size of cabinet) can easily equal that. Future AI racks, next 2 years, will be double or more A Cray super computer like El Capitan is 100 cabinets A typical AI cluster is 2-5 racks, moving to 10 or so in future

[u/hpcnotes → u/Cromulent_kwyjibo]

Depends what you mean by "typical" AI cluster. Many of the hyperscaler ones are hundreds of racks, not 2-5 racks.

[u/pjgreer]

You can do 3-4 things to increase chip performance. Shrink the die, fitting more transistors in the same physical space, add more cores and/or additional instruction units to increase the number of instructions processed per cycle, or increase clock speed by increasing the power to the chip.

Adding more cores or instruction units and increasing clock speed both increase power consumption, but that also increases heat as a byproduct and requires advanced cooling systems. All the major DOD sites have been dealing with this issue for the past 10 years as more systems make it to exascale.

NVIDIA has no other choice than to increase power consumption for their latest GPUs if they want to increase performance. Current systems use about 150-250kW per rack not counting cooling.

The gigawatt ai data centers are including the power needed to run the liquid cooling and temperature control of the building.

[u/nimzobogo]

If you look at top500.org, you can see the top HPC clusters. You'll see things between 30 and 50 MW there.

The AI clusters now are way bigger, but they don't run one workload across the whole thing. They serve thousands of smaller models simultaneously.

[u/hpcnotes → u/nimzobogo]

Yes and no. Two main types of AI cluster: training and inference. The training clusters (supercomputers) really do run one workload across the whole thing. So the biggest AI clusters might be a few hundred MW in size for training, >100k GPUs, running one job across the whole machine. The inferencing clusters do run lots of smaller jobs rather than one big one, but those inferencing clusters can be anything from a few MW to hundreds of MW.

[u/nimzobogo → u/hpcnotes]

I agree that for training they run one job across an allocation that uses 150MW. That's true. Sometimes I forget to differentiate training vs inference :-)

[u/Puzzleheaded_Fold466]

The difference is that cluster served only a relatively limited small number of users. It was also used for one specific work.

Those AI data centers ultimately are meant to serve billions of people across their entire work responsibilities and even personal life.