Conclusion
There’s the next generation of Gen4 NVMe SSDs out, those with NVMe protocol 1.4, new faster controllers, and NAND flash. These SSDs are pushing new performance boundaries of 7000MB/s+, providing incredible read and writing performance, and there are even faster drives on the horizon.
With performance, this fast, PCI-Express 4.0 x4 is finally being pushed to its limits with SSDs. However, how those PCI-Express lanes are divided up varies from chipset to chipset and motherboard to motherboard. Often typical, you will find a couple of PCI-Express 4.0 x4 NVMe M.2 sockets available on your motherboard. While they look the same, and both support PCI-Express 4.0 x4 connections, they can very much differ on where those lanes come from. That can add latency, and cause performance differences on the fastest SSDs today.
We decided to throw on a very fast SSD, and simply switch it between the Primary M.2 socket which pulls its lanes from the CPU and the Secondary M.2 socket which pulls its lanes from the X570 chipset. Both support PCIe 4.0 x4, but they differ on where the lanes come from.
Performance
Using our MSI SPATIUM M480 2TB HS PCIe 4.0 Gen4 NVMe SSD we tested it on both M.2 sockets on our ASUS TUF GAMING X570-PLUS (WI-FI) motherboard, a very typical setup. We ran benchmarks, and compared performance.
We figured there would be a performance difference, but we weren’t prepared for how large those could be. Every benchmark, every result, was slower on the Secondary M.2 socket by varying degrees. Some benchmarks were not as large differences, but some were larger.
On the application performance benchmarks, we saw around a 3% difference. The file copy tests were also not extremely large differences, but they could add up. We saw a greater performance difference with files of mixed sizes. A lot of little file sizes seemed to cause the most difference in performance. Game load times were not that much affected.
What was affected big time is the raw throughput of performance in both read and write. Sequential performance being the number one performance difference between both sockets. This was very significant, as much as 10% difference in bandwidth throughput, which could be noticeable.
That put the performance class of the SSD into a different class on the Secondary M.2 socket. There was also a large difference in random 4K write performance, the Secondary M.2 socket was a big 13% slower, this is very significant. We also saw a significant difference in workstation-class applications.
Final Points
What have we learned from all this? If you are purchasing one of the new breeds of SSDs with performance in the 6000MB/s-7000MB/s range, or higher, install your SSD into the M.2 socket that connects its PCI-Express lanes directly with the CPU.
You are spending a lot of money to get a high-performance SSD, in the case of the MSI SPATIUM M480 2TB HS PCIe 4.0 Gen4 NVMe SSD, $449.99. You want to make sure you get all that performance you are paying for, and not leaving any on the floor. Therefore, install the SSD into an M.2 slot that has its PCIe lanes coming from the CPU, not the chipset, else it will be holding back the potential of your SSD, and wasting your money.
This is going to be ever-important as we look forward to the future of PCI-Express 5.0, and even faster SSDs which have now been revealed to be upwards of 14GB/s from KIOXIA. With the evolving landscape of SSDs, ever continuing advancements, and performance improvements, the disadvantages, and latencies associated with PCI-Express lanes connected via the chipset will potentially only widen and be exposed even more. Therefore, if you have a fast PCIe 4.0 Gen4 SSD, stick that SSD in an M.2 socket that shares its lanes with the CPU.