Counterpoint Analysts were invited by Applied Materials, one of the leading semiconductor manufacturing equipment vendors, to participate in the AI Design Forum at SEMICON West in California, USA recently. The event was hosted by SEMI Americas, the Electronic System Design Alliance and Applied Materials. The keynote speakers included key executives in the semiconductor industry from Applied Materials President and CEO Gary Dickerson to other CEOs from AMD, Synopsys and Xilinx, along with technology presentations from arm, Google, Qualcomm, and the Embedded Vision Alliance.
We interviewed a number of key stakeholders driving the AI in the cloud and at the edge and understand the challenges the semiconductor industry is about to face as Moore’s Law, the rule of progress in semiconductor design, is no longer valid. As a result, traditional chip design and the associated manufacturing processes cannot fulfill the fast-developing demand for Artificial Intelligence (AI) and will have to change.
During the meeting, Victor Peng, the President & CEO of Xilinx, said the processor performance is saturating and is facing three major problems:
- Power density increases
- End of Power, Performance & Area (PPA) improvements
- Multicore hits a limit
Given these facts, semiconductor technology must adapt to meet the growing needs of emergent technology areas such as AI & IoT.
In addition, Gary Dickerson, chief executive of Applied Materials, discussed the data explosion that AI and the Internet of Things will cause, thanks to the expected hundreds of billions of connected devices by 2030.
The amount of data generated by a person is currently about 1GB per day. The amount of data generated by an autonomous vehicle in 2023 will reach 4000GB per day with average annual growth rates of about 5% and 70% between 2018 and 2023, respectively.
Also important, but often overlooked, is the energy consumption needed to train AI models. This is expected to increase exponentially. It is estimated that training a single AI model can emit as much as carbon as five cars in their lifetime. Energy consumption will likely become a key constraint to the growth of AI unless more energy-efficient computing paradigms are developed.
And while the power consumption of most IoT edge devices is low, the total energy consumption is largely due to the massive number of them. For example, each IP camera consumes only 5 to 8 watts, but in 2020 all IP cameras combined will consume more power than the energy generated by a standard power plant in the United States.
Dickerson also mentioned AI will take 80 percent of the workforce in future datacenters for neural nets and training, and also consume more than 10% of global electricity supply in 2025. Therefore, in order to reduce the excessive power consumption and the surge in data volume, future semiconductor design and manufacturing must consider four key aspects: Performance, Power, Area-Cost (PPAC). And start with five new technologies:
- New architectures
- New structures/3D
- New materials
- New ways to shrink
- Advanced packaging.
In addition, semiconductor innovation is needed as you can see in the examples below in the cloud and edge.
Exhibit 1: Promising pipeline of new hardware innovation
Source: AI Design Forum at SEMICON West 2019
As pointed out by the exhibit above, the new role of memory in hardware innovation is getting important. Today’s mainstream memory consumes too much power, so a “new architecture” for chip design and new memory chips are needed. He also listed several types of memory, including MRAM, PCM, and ReRAM and demonstrated Applied Material’s capability in manufacturing equipment for memory.
Counterpoint has long-term research for emerging memory. In the past 20 years, several next-generation memories have debuted, but none can challenge the mainstream DRAM and NAND. The power consumption of volatile DRAMs is much higher than non-volatile memories. Although the production cost of NAND flash is low, its speed and delay are poor. However, as DRAM approaches its physical limits, production and R&D costs have increased significantly relative to the past. The technology lifetime of NAND flash will be extended with 3D structures. However, its shortcomings have not been addressed during the migration to 3D. RRAM, MRAM, and PCM are currently considered to be promising replacements for DRAM and NAND, but they still need to prove their capabilities in full-scale production, as well as further reduce production costs.
Exhibit 2 illustrates the current memory classification, including mass production memory, niche memory, and emerging memory.
Exhibit 2: Memory Technology Classification, 2019
Source: Counterpoint Research – Memory Research
PCM: One of the most promising PCMs is 3D XPoint which is promoted by Intel and Micron. So far, the manufacturing cost of 3D XPoint is lower than that of server DRAM, but it is still higher than NAND flash and commodity DRAM. 3D XPoint is superior to NAND flash in terms of read/write speed, latency, input/output operations per second (IOPS) and endurance. Compared to DRAM, 3D XPoint is still slower but, it has lower manufacturing costs and power consumption. As a result, 3D XPoint can be used as a non-volatile dual in-line memory module (NVDIMM), solid state drive (SSD) storage and cache in servers.
MRAM is a memory that stores binary data in an electronic spin state. MRAM is faster than DRAM and requires neither management nor refresh. MRAM is one of the most promising non-volatile emerging memory types and is now in mass production. However, it is still expensive, so its application is limited to industrial and server uses. Currently, the maximum density of a single-chip STT-MRAM is 1 Gb. Once the price falls further, it has an opportunity to be used as a cache in consumer SSDs.
The density of ReRAM could be comparable with 3D NAND because 3D structures are possible. However, it still has reliability issues and is expected to see more adoption after the production cost of 3D NAND increases significantly.
In conclusion, one of the biggest challenges to implement AI today is to overcome the memory speed and power bottlenecks in the current architecture to achieve faster data access while reducing energy costs. Adopting the new non-volatile memory, MRAM, PCM and RRAM, can make it possible to achieve large-scale energy savings and performance improvements, thus greatly extending battery life and better user experience.
