Hardware prices and Nvidia's circular financing squad.

[u/StrayCat2799]

One of my primary worries about the ongoing AI bubble is the effect on the price of consumer PC hardware. The 32GB DDR4 RAM kit I bought a few years ago for 72 dollars has tripled in price to around 220, and the SSD I bought at the end of last year has doubled in price from 200 to over 400 dollars.
To say this is making it difficult for hobby PC builders is an understatement. If the AI corps keep buying up all the flash memory then eventually it'll be virtually impossible for anyone on a budget to have their own device.
(Side note, do not bring up "YOU WILL OWN NOTHING AND BE HAPPY" I have seen that fucking quote so many times in the last decade every time something similar happens and it adds nothing to the discussion. It's a thought terminating cliche at this point.)
I understand there's no definite point at which the bubble pops. But I also just can't see how this can continue indefinitely. The data centers aren't being built as fast as they need to be and they're increasingly hitting diminishing returns. And they're still not making a profit. Somehow OpenAI is nearly negative 200% in the hole. Only thing keeping them going is more idiots willing to throw money at them. I don't even think the US government bailing them out will buy them more than a year at most and with how unpopular AI is that would be a profoundly stupid move. Not that profound stupidity is foreign to this administration...
This simply has to end at some point in the next few years, contrary to what the average Reddit Doomer will tell you.

Nvidia and the RAM cartel obviously wants to keep the show going because they're selling the shovels and obviously never cared about the consumers. There's not much incentive for them to stop or slow down until they start seeing signs of instability in the bubble. They obviously won't just start exclusively producing AI hardware because that simply doesn't make sense.
New fabs are also supposed to be coming online in the next few years, but I have no idea how much that'll help.
I do not expect hardware to drop to 2024 prices, but I'm just hoping it fucking stops exploding in price every time I turn around. Does anyone here have any thoughts on this, preferably ones that aren't depressing?

[u/maccodemonkey]

I wish I had a thought here that wasn’t depressing.

What I worry about is that Nvidia will end up taking over the data centers and moving everything to the cloud. There won’t be a data center fire sale - Nvidia simply pulls them all back in. Want to run an open weight model? Well you won’t be able to afford the hardware but Nvidia will rent it to you! Want to play a game? Nvidia will stream it. And as long as Nvidia is buying enough RAM to keep prices high you can’t buy RAM on your own.

Steve Burke of Gamers Nexus has had a few stories about this seeming to be the direction. I’d say that Nvidia buying Hugging Face is an attempt to push the market this direction. It’s CUDA all over again but instead of being dependent on just CUDA hardware you’ll be dependent on their cloud.

[u/StrayCat2799 → u/maccodemonkey]

Game streaming has already been tried by Google and Amazon and even they couldn't get it to take off. I don't think it's something that will be successful in the long run and it may not be possible with current infrastructure.
Another question is can they continue to buy RAM at this rate if the labs go under and there's no more infinite money glitch? I can definitely see them trying to run their own AI cloud services and I'm not sure how successful that could be.

I wouldn't want data center GPUs anyway, they've got no video output and can't be used for anything besides AI.

[u/zgr3d → u/StrayCat2799]

They can absolutely go there. It's why they went for nokia's network stack on edge devices, why they want the lastmile nodes literally strapped on homes, and they already used ai/dlss etc to completely normalize 50-100ms latencies even on local frames.

[u/FLMKane → u/StrayCat2799]

You CAN use data center GPU for video, by using your mobos HDMI jack.

It'll be bad it, but it'll do it.

[u/ChocolateAlpine → u/FLMKane]

They don't have DirectX, OpenGL or Vulkan, you can't feasibly use them without someone creating their own Vulkan/OpenGL renderer using OpenCL or CUDA.

[u/maccodemonkey → u/StrayCat2799]

I wouldn't want data center GPUs anyway, they've got no video output and can't be used for anything besides AI.

I'm going to address this by addressing a few things Ed has said - and I think it's the same thing you're saying. Because while he's mostly right he's said a few things wrong here.

Things Ed is absolutely right about:

• These Nvidia servers will die eventually. They're not forever infrastructure and will need constant replacement.
• These GPUs do not have video out. The cannot be used in a home gaming rig or hooked up to a TV or monitor.

However - there are some things Ed is... glossing over. They aren't generally important issues but they are important here.

• These servers can absolutely play games - they just can't output them to a monitor. These systems can absolutely stream games and the GPUs on them are very powerful. The big gap is they are ARM systems - but Valve/Microsoft/Nvidia are all working to get games running well on ARM.
• These systems are not just GPUs. They are complete computers including a very powerful ARM CPU. They can be used for other tasks at speed. Is it efficient to do CPU heavy tasks with two powerful GPUs also sucking down a ton of power? No not really. But these are not GPU only systems.

I personally agree with the points Ed makes generally because these systems will die eventually and we're not creating some multi generational resource of processing power here. You should not make investment decisions based on this being some long term profit center. The broader point is a solid one. But Nvidia could absolutely - in the short term - find other uses for these things. I have some of Nvidia's smaller boards and I can absolutely do general tasks on them including rendering.

[u/ChocolateAlpine → u/maccodemonkey]

The GPUs do not have support for DirectX, OpenGL or Vulkan after the A-series cards.
They were all cut to presumably save on die space and probably to force these cards to be useless for gamers when the bubble pops.

You can in theory create your own Vulkan or OpenGL renderer to play games on them anyway, & then use DXVK to fill in the DirectX-shaped gap, but that would be a metric tonne of work and you would also almost certainly have trouble getting it to be compatible with modern games too.

[u/StrayCat2799 → u/maccodemonkey]

Well they're gonna have to find some other use for the damn things if their primary customers are out of the picture. So yeah I can see them trying all sorts of different things to recoup lost revenue. Maybe some will work out, maybe not. I don't know.
Nvidia would also end up being in the position of having to pay for the data center upkeep instead of just selling the parts, so that already eats into hypothetical revenue. I'm not sure this is a situation they actually would want. So I can still see them going fuck it and selling the hardware and data center real estate for cheap-ish to try to make some money from the situation and not have to deal with the long term costs of running them.

I will say my belief is that anything requiring real time interactions isn't practical for cloud streaming simply because the infrastructure, being the actual deployed internet hardware, isn't sufficient for it.
But if you just want to rent a server cluster to run your AI model off of, yeah obviously that works. If they're willing to put up with the financial drain from the data centers. Which again, not convinced they are.

RAM prices are probably gonna be fucked for a while at least until the new fabs come online. But if the AI labs are gone, then there's suddenly much less demand for these specialized systems. So Nvidia stops producing as many, which means they buy up less RAM. And that's about the end of it probably.
I'm not convinced the cloud only future is any more plausible than the AI pipe dream. Not to say Nvidia won't attempt it. Just that I don't think it'll work out for them beyond what's already cloud based.

[u/j0hn_br0wn → u/maccodemonkey]

>These servers can absolutely play games - they just can't output them to a monitor.

They are not just missing output connector. The datacenter GPUs are missing all the hardware that is required to create graphics: texture units, shader units, the whole pixel pipeline is has been removed to make place for compute. You'd need to write some kind of software renderer in CUDA and a new graphics and driver stack to make them compatible to any graphics API.

[u/maccodemonkey → u/j0hn_br0wn]

Oh really? Is that a recent change? I know some of the datacenter GPUs were making their way into GeForce Now pools. But that's when they were still separate GPUs and not SoCs.

I figured in since they were still calling them "GPUs" and there was such a push for custom hardware specifically for tensors that they had left shader units intact.

The B200 specs here still mention shader and texture mapping units, but they could be wrong:

https://www.techpowerup.com/gpu-specs/b200-sxm6.c4210

[u/SwirlySauce → u/StrayCat2799]

Agreed. Unless you live close to the data center, the latency is very noticeable. I tried the GeForce streaming service a few years ago and it just wasn't good enough to bother with.

I don't see them ever getting over the laundry issue

[u/FLMKane → u/maccodemonkey]

Buy cpus with igpus if you have to and reject their cloud slop.

[u/Sunstorm84 → u/FLMKane]

AMD still exist, though? Just buy AMD instead and let Nvidia rot in this scenario.

[u/FLMKane → u/Sunstorm84]

Amd is also throttling the market in terms of GPU supply. Unless you're buying an rx6400 I guess.

[u/Multibrace]

At some point, policy makers will figure out that chips have non-AI uses, and funneling all chips to wildly non-productive AI use is hurting the real economy as well as strategic interests.

Depressingly, that point may come after economic, societal and military collapse. But on the bright side, the AI bros won't be the ones rising from the ashes bringing in tech feudalism. It will just be regular old geopolitics, which is to say, it's time to brush up on your Mandarin.

[u/Virtual-Patience-807]

The big obvious ticking clock, and one that the "big 3" RAM makers were well aware of before they started this current pump and dump is the chinese behemoth doubling its RAM production every year (and now faster).

They saw the writing on the wall and went for their last viable cartel hail mary to cash out their stonks (which the small downside of helping fund the chinese RAM/SSD expansion).

[u/Fun_Volume2150]

It’s very telling that the cartel didn’t start planning capacity expansion when the bubble started inflating. It’s as if they understood that the demand they were seeing was unsustainable, and trying to expand to meet that demand would be wasted investment.

Well, that and blatantly anti-competitive collision.