Researchers from NVIDIA, IBM, and Cornell University have been working on a potential solution for fast, fine-grain access to large amounts of data storage for GPU-accelerated applications called Big Accelerator Memory (BaM). BaM is an architecture that includes a software component for turning an SSD into shared memory for the GPU via direct access. The research paper can be found here.
We propose a novel system architecture called BaM (Big accelerator Memory). The goal of BaM’s design is to extend GPU memory capacity and enhance the effective storage access bandwidth while providing high-level abstractions for the GPU threads to easily make on-demand, fine-grain access to massive data structures in the extended
memory hierarchy.
The idea is to replace current forms of shared pooled memory that GPUs presently use, which incorporate system memory via the CPU or additional VRAM when multiple graphics cards are linked together. When it comes to data-centric GPU applications, memory is often a limitation, and these current solutions tend to be inefficient. BaM hopes to mitigate those limitations by providing an API for NVMe direct storage access.
An existing solution to the problem of insufficient GPU memory capacity for these massive data structures is to pool together the memory capacity of multiple GPUs to meet the capacity requirement [12], [36], [38], [41], [56] and use fast shared memory interconnects like NVLink [1] for the GPUs to access each other’s memory. The entire data structure is first sharded into the GPU memories. The computation then identifies and accesses the portions that are actually used. This approach has two drawbacks. First, the entire data structure needs to be moved from storage to the GPU memories even if only a portion might be accessed, which can add significantly to the application start-up latency. Second, the data structure size determines the number of GPUs required for the application, which may far exceed the compute resource requirement of the workload.
Using the host memory, whose capacity typically ranges from 128GB to 2TB today, to help hold the sharded data structure can reduce the total number of GPUs used [35]. We will refer to the use of host memory to extend the GPU memory capacity as the DRAM-only solution. Because multiple GPUs tend to share the same CPU and thus the host memory in data center servers, such DRAM-only solutions tend to add only a fraction of the host-memory size to each GPU’s memory capacity. For example, in NVIDIA DGX A100 systems, each host memory is shared by 8 GPUs. Thus, using host memory only extends the effective size of each GPUmemory by 1/8 of the limited size of the host memory.
It is hoped that by mostly removing the CPU from memory access, there will be much less I/O overhead, resulting in faster transfer speeds while freeing up the processor for other tasks.
Unfortunately, such a CPU-centric strategy causes excessive CPU-GPU synchronization overhead and/or I/O traffic amplification, diminishing the effective storage bandwidth for emerging applications with fine-grain data-dependent access patterns like graph and data analytics, recommender systems, and graph neural networks.
The research team is planning on making their results open-source so others can develop BaM to their own needs. They’ve already tested it using off-the-shelf NVMe drives and GPUs.
This will not be the first time GPUs have had an SSD solution, as AMD once tried it with its Radeon Solid State Graphics. We’ve also seen fantastic transfer speeds with the PlayStation 5, while Microsoft has gone on to incorporate its own means of communication between the GPU and SSD with the Direct Storage API in Windows 10/11.
One thing that is certain is that as NVMe transfer speeds increase, from data-centric applications to console and PC gaming, everyone is wanting a slice of the NVMe pie now.
Source: Cornell University (via The Register)
Discussion (8 replies)
Join Discussion →Seems like a stop-gap solution, when the real solution should be to decrease latency and increase bandwidth between the GPU and mass storage.
Granted it may also be necessary for a generation or two - asking host bus (chipset / northbridge / whatever) controllers to supply that much bandwidth while cutting the CPU out of the equation to reduce latency is perhaps a big step that likely wouldn't provide much gain elsewhere.
I’m wondering how far we are from just having the GPU take over the main parts and the CPU just being a co-processor
For those companies with integrated IP, i.e. Apple (and other ARM users), AMD, Intel, Nvidia maybe someday, this is already a reality, right?
Imagine a <5nm update to the console APUs using Zen 4 cores as an example closer to what we see on desktops.
And Apple's M1 family as probably the best modern example of a high-performance SoC with both DRAM and NAND on package.
When this happens PC's will just become glorified consoles. You have a chassis, perhaps a cooling and power delivery solution. Everything else is on the 'card' and your Motherboard is just a techie version of a high speed port replicator. Upgrades... Sure you can buy a new card. Oh you wanted to upgrade your RAM.. sorry no closed box. Storage... Sure you can ADD a storage card to your motherboard. But it won't be as fast as native. New video processors... nope that's a new card and it comes with new storage, and Ram too.
I don't want that future. I'm not naive enough to think it's not coming. But I think that will be as NUC like device, and bigger badder compute will still be in the realm of DIY.
Soon enough we will have to have licenses to have a DIY system. Just wait.
I think it's even more interesting than that - I find our current 'solution', that being all manner of 'chassis' that we put mostly standardized parts into - to be fairly inelegant and necessary largely due to a lack of collective will to standardize.
I'm sitting here next to a Lian Li O11 Dynamic XL, which houses my personal desktop. It's the largest desktop case I've owned in the thirty years I've been building them, and I went with it for one reason only: to have more room to work inside the case.
As I've been able to pick up a 3080 and have started venturing down the Alder Lake path, the larger Lian Li case has proven to be a great companion.
Thing is, if you don't need to work in a system, and you don't need the highest-TDP parts, well, you could get away with an SFF the size of a toaster!
Now, to the meat.
With the proliferation of fast network interconnects as well as a wide variety of virtualization advancements, I can see 'desktop computing' going a similar direction that 'mobile computing' has already gone, that is, punted off to some big iron on the other end of a network link. Everything from 400Gbps fiber to variable-frequency wireless links can bring latency down to manageable levels with the 'local machine' being used primarily for the user interface and feedback-intensive work.
And if you consider what is within the realm of possibility today in say a <100W package, something that can be reasonably inconspicuous in terms of visual and aural footprints (small and quiet), I'm thinking that we can get pretty far toward 'computers' really being tuned for whatever I/O they need to service in terms of user interaction. Like phones, you really wouldn't need to upgrade them very often, and they'd really be more like appliances with decade-long or longer service lives being a given, like laundry, refrigeration, televisions before the digital age and now into the 4k age, and so on.
Very likely few will actually 'need' significant local compute resources. Gaming is literally the main consumer workload that has fairly absolute interactive latency limits, so it's likely to remain a bit of a special case.
For any other 'heavy' processing, whatever that might be, it can either be done on a local compute node (in a closet, etc.) or farmed out to a cloud environment of some sort.
We or even maybe I am the minority here. I don't want to pay a monthly subscription to have a computer to use. I would much rather own it and have a portal to access my computer from or stream steam games or whatever from remotely rather than pay a monthly subscription to whosit online for twenty bucks a month. Oh you want the newest video card features and more performance... that's MORE per month.
No thank you to ALL of that. I would much rather own my PC and game how I want and use it how I see fit without some 3rd party assigning services based on a monthly fee.
I'm not going to break out my hardware spend for my compute costs. I already know I'm paying more for personal hardware than I would on a month to month basis. Part of that is I enjoy the hardware journey it also feeds directly into my career.
I can honestly say my year to year salary growth is more than I spend year to year on computer parts.
This is a threat, sure; how much of one I think depends on where consumers go.
For example, Adobe has been making money hand over fist by 'renting' their software. It's to the point that actually buying their software is either extremely difficult or outright impossible.
Some have complained. However, for most, it's been an improvement. The cost is low, as low as buying updates would be or lower, and quality and features have steadily improved. Video editing and digital photography, for example, haven't changed that much in the last decade, but the equipment very much has, both in terms of the capture chain and the editing chain. Cameras and lenses need profiling, codecs need to be integrated for stills and video, and so on. Adobe keeps up with that stuff better than many / most.
On the other hand, I'm not big on not owning the hardware, and I think many consumers feel the same way, whereas there's definitely a business case for leasing at the enterprise scale.
I was really just thinking the GPU would be the main attraction on the motherboard and the CPU would go on a daughter card (as opposed to the current thinking which has that flipped), but the SOC examples are probably much more pertinent.