Would having an expandable RAM help in running non training workloads like inference for LLMs / Stable Diffusion models?
In: 6
For the most part it would just make things worse and only help in a very narrow set of circumstances
Graphics memory is running at extremely high speed and is very sensitive to latency.
Not enough memory or memory too slow for the GPU and performance is significantly reduced.
Faster memory or more memory? Doesn’t really matter because you’re limited by the processing
There are some pro level cards with crazy amounts if memory of your data set requires it, but then you’re not worried about cost or graphics. Nvidia sells GPU Accelerator cards with up to 80 GB of memory and a 5120 bit wide connection to get some crazy memory speeds out of HBM but it needs to be right next to the core for those speeds
Latest Answers