3DXP Products Tomorrow, Rumors and Questions Now
Intel is sampling 3DXP DIMMs today, and will offer selected developers access to cloud-based systems in June. Intel will be shipping products this year (read as a 4Q18). However, it is unlikely that customers will have access outside of the cloud until 2019, given the schedule for Cascade Lake, the typical qualification cycles for OEMs, and the novelty of 3DXP DIMMs.
One consequence of Intel’s excellent software and ecosystem enabling strategy is that many key players within the industry have access to information about (or samples of) 3DXP and 3DXP DIMMs ahead of public availability. Naturally a lot of information concerning 3DXP DIMMs has leaked out from third parties. Some of this information is accurate, but some leaked information has become stale as plans have changed. Moreover, some of the analysis and interpretation is simply incorrect.
Intel’s recent briefing and announcements are partial disclosures and resolve some of this uncertainty, but also leaves many questions unanswered about 3DXP DIMM hardware, systems, and software.
First, Intel has not revealed the exact latency for 3DXP DIMMs, let alone a full characterization sweeping across different scenarios. It is likely that the Cascade Lake memory controllers use DRAM for caching persistent memory, so in some scenarios the latency and throughput will mimic DRAM. Unloaded access latencies are expected to be at the microsecond level, but more details would be helpful.
Second, Intel did not disclose power consumption and thermal dissipation of 3DXP DIMMs. This implies the power draw will probably be greater than a conventional DRAM-based DIMM; it is also possible that the product is still subject to tuning. If the power is great enough, it could require that customers redesign their boards to meet power delivery and cooling requirements. The power consumption could also impact the suitability of 3DXP DIMMs for notebooks and other power-limited platforms. Similarly, Intel did not share any endurance ratings, except to say that the DIMMs would be guaranteed for the lifetime of the part. Given the improvement in 3DXP SSD endurance, it is nearly certain that Intel is continuously improving the endurance.
Third, Intel has not disclosed pricing. Given that there is no product yet, this is entirely reasonable. Overall, the company has positioned 3DXP between DRAM and NAND, so it is reasonable to expect pricing to fall within this range. The first 3DXP SSDs were priced at about the midpoint of NAND and DRAM. It is only logical that 3DXP DIMMs would be priced a bit higher – perhaps at 75-80% of comparable DDR4 memory. Historically, memory and storage have always been highly elastic – so the adoption of 3DXP DIMMs will depend on the pricing. The initial applications are all fairly high value and therefore can bear high prices, but lower prices could encourage customers to substitute away from DRAM.
Turning to systems, Intel and its partners have clearly indicated that two-socket servers will support 6TB of 3DXP, implying a single 512GB DIMM per memory channel. For many database applications such as SAP, Oracle, or SQL Server, customers use four- and even eight-socket servers. In some cases, software vendors may not even support two-socket systems. Those larger systems should offer up to 12TB and 24TB of 3DXP, but it is unclear when such configurations will be available. Historically, larger systems require one to two quarters of extra validation and this may be the case for 3DXP DIMMs.
Server systems will continue to use DRAM alongside 3DXP DIMMs, and the memory controllers in Cascade Lake probably use some portion of the DRAM to act as a cache for the 3DXP. But the available configurations are unknown, and it is likely that the recommended mix will be application dependent. For example, using persistent memory for heavy updates to a small working set should require less DRAM than a larger working set. Intel’s cloud systems for developers are configured with 1TB of 3DXP and 192GB of DRAM – a ratio of 5.3:1, so clearly a richer 4:1 mix is viable. But what about 8:1, or 16:1? It is likely that Intel will offer some guidelines, but ultimately leave the decisions up to OEMs.
Fourth, Intel has refused to discuss security and serviceability. The company has stated that data stored in 3DXP DIMMs is encrypted, and declined to elaborate further. For example, how will this encryption mechanism interact with SGX, which was designed to encrypt volatile memory? SGX was curiously missing from Skylake-SP, but presumably is enabled in Cascade Lake. We expect that regardless of the encryption scheme used, that the encryption keys will be stored in the processor’s memory controller. However, this introduces challenges for serviceability and recoverability. If a persistent DIMM fails or is moved to a different system, how will the contents be accessed? Similarly, what if the processor is removed (e.g., due to a failure)? Any in-memory database customer should demand a sensible security scheme that enables recovery and serviceability.
One of the last questions is around virtualization and multi-tenancy. In modern systems, memory is virtualized by the OS and hypervisor, while storage is typically not virtualized. Persistent memory and 3DXP DIMMs fall in between memory and storage, and are not virtualized by current hypervisors or OSes. That presents a challenge for cloud vendors and customers, many of which are accustomed to overprovisioning physical hardware. For example, to create a consistent target for developers a company may wish to standardize on VMs with 1TB of persistent memory. With current infrastructure, that requires physically having enough persistent memory in the system for each VM – which could leave valuable hardware resources under-utilized. We believe that some hypervisors will choose to virtualize persistent memory, but it is a question of when products will be released (or in the case of Xen/KVM, code released and put into production).
Conclusion
It is abundantly clear that economical persistent memory will be one of the biggest changes to computing in many years. Intel has released enough details about 3DXP DIMMs to tantalize and substantiate claims about 4X better density than DRAM, while demonstrating the performance advantage over NAND flash. More importantly, the company has described an impressive and comprehensive strategy for building the ecosystem around persistent memory. Learning from previous failures, Intel engaged with the ecosystem (e.g., existing NVDIMM-N vendors) to enable the software world. While there is still much progress to be made, the initial software support arrived before production hardware. Multiple server and client OSes and hypervisors already support persistent memory and leading developers are excited about the technology. The greatest excitement is from the database world, where the persistence and large capacity is invaluable. Beyond databases, the PMDK is ready for use, and will enable popular languages like Java to benefit from 3DXP DIMMs. Products will be out later this year, and we are quite excited to test out 3DXP DIMMs when they arrive.