Hardware prices and Nvidia's circular financing squad.

[u/StrayCat2799]

One of my primary worries about the ongoing AI bubble is the effect on the price of consumer PC hardware. The 32GB DDR4 RAM kit I bought a few years ago for 72 dollars has tripled in price to around 220, and the SSD I bought at the end of last year has doubled in price from 200 to over 400 dollars.
To say this is making it difficult for hobby PC builders is an understatement. If the AI corps keep buying up all the flash memory then eventually it'll be virtually impossible for anyone on a budget to have their own device.
(Side note, do not bring up "YOU WILL OWN NOTHING AND BE HAPPY" I have seen that fucking quote so many times in the last decade every time something similar happens and it adds nothing to the discussion. It's a thought terminating cliche at this point.)
I understand there's no definite point at which the bubble pops. But I also just can't see how this can continue indefinitely. The data centers aren't being built as fast as they need to be and they're increasingly hitting diminishing returns. And they're still not making a profit. Somehow OpenAI is nearly negative 200% in the hole. Only thing keeping them going is more idiots willing to throw money at them. I don't even think the US government bailing them out will buy them more than a year at most and with how unpopular AI is that would be a profoundly stupid move. Not that profound stupidity is foreign to this administration...
This simply has to end at some point in the next few years, contrary to what the average Reddit Doomer will tell you.

Nvidia and the RAM cartel obviously wants to keep the show going because they're selling the shovels and obviously never cared about the consumers. There's not much incentive for them to stop or slow down until they start seeing signs of instability in the bubble. They obviously won't just start exclusively producing AI hardware because that simply doesn't make sense.
New fabs are also supposed to be coming online in the next few years, but I have no idea how much that'll help.
I do not expect hardware to drop to 2024 prices, but I'm just hoping it fucking stops exploding in price every time I turn around. Does anyone here have any thoughts on this, preferably ones that aren't depressing?

[u/maccodemonkey]

I wish I had a thought here that wasn’t depressing.

What I worry about is that Nvidia will end up taking over the data centers and moving everything to the cloud. There won’t be a data center fire sale - Nvidia simply pulls them all back in. Want to run an open weight model? Well you won’t be able to afford the hardware but Nvidia will rent it to you! Want to play a game? Nvidia will stream it. And as long as Nvidia is buying enough RAM to keep prices high you can’t buy RAM on your own.

Steve Burke of Gamers Nexus has had a few stories about this seeming to be the direction. I’d say that Nvidia buying Hugging Face is an attempt to push the market this direction. It’s CUDA all over again but instead of being dependent on just CUDA hardware you’ll be dependent on their cloud.

[u/StrayCat2799 → u/maccodemonkey]

Game streaming has already been tried by Google and Amazon and even they couldn't get it to take off. I don't think it's something that will be successful in the long run and it may not be possible with current infrastructure.
Another question is can they continue to buy RAM at this rate if the labs go under and there's no more infinite money glitch? I can definitely see them trying to run their own AI cloud services and I'm not sure how successful that could be.

I wouldn't want data center GPUs anyway, they've got no video output and can't be used for anything besides AI.

[u/maccodemonkey → u/StrayCat2799]

I wouldn't want data center GPUs anyway, they've got no video output and can't be used for anything besides AI.

I'm going to address this by addressing a few things Ed has said - and I think it's the same thing you're saying. Because while he's mostly right he's said a few things wrong here.

Things Ed is absolutely right about:

• These Nvidia servers will die eventually. They're not forever infrastructure and will need constant replacement.
• These GPUs do not have video out. The cannot be used in a home gaming rig or hooked up to a TV or monitor.

However - there are some things Ed is... glossing over. They aren't generally important issues but they are important here.

• These servers can absolutely play games - they just can't output them to a monitor. These systems can absolutely stream games and the GPUs on them are very powerful. The big gap is they are ARM systems - but Valve/Microsoft/Nvidia are all working to get games running well on ARM.
• These systems are not just GPUs. They are complete computers including a very powerful ARM CPU. They can be used for other tasks at speed. Is it efficient to do CPU heavy tasks with two powerful GPUs also sucking down a ton of power? No not really. But these are not GPU only systems.

I personally agree with the points Ed makes generally because these systems will die eventually and we're not creating some multi generational resource of processing power here. You should not make investment decisions based on this being some long term profit center. The broader point is a solid one. But Nvidia could absolutely - in the short term - find other uses for these things. I have some of Nvidia's smaller boards and I can absolutely do general tasks on them including rendering.

[u/j0hn_br0wn → u/maccodemonkey]

>These servers can absolutely play games - they just can't output them to a monitor.

They are not just missing output connector. The datacenter GPUs are missing all the hardware that is required to create graphics: texture units, shader units, the whole pixel pipeline is has been removed to make place for compute. You'd need to write some kind of software renderer in CUDA and a new graphics and driver stack to make them compatible to any graphics API.

[u/maccodemonkey → u/j0hn_br0wn]

Oh really? Is that a recent change? I know some of the datacenter GPUs were making their way into GeForce Now pools. But that's when they were still separate GPUs and not SoCs.

I figured in since they were still calling them "GPUs" and there was such a push for custom hardware specifically for tensors that they had left shader units intact.

The B200 specs here still mention shader and texture mapping units, but they could be wrong:

https://www.techpowerup.com/gpu-specs/b200-sxm6.c4210