Top

Guest Post: AI Ecosystem – The Race Begins

While everyone is going to be focused on AI inference in 2024, the real action is going to be in the ecosystem where new players and old will slug it out to ensure that users and developers use their models.

Foundation model is to AI what OS is to smartphone

  • The players are all working to get their ducks in a row to be ready to launch two things in 2024:
    • First, software development kit (SDK): This is a piece of software or an interface that makes it easy for developers to retrain a foundation model in order to perform a certain service.
    • This is the AI equivalent of the SDKs that are used to write apps for iOS, Android, Windows and so on.
    • It turns out that the foundation model is becoming a control point in the AI ecosystem as they are difficult and expensive to create and difficult to change once one has based a service on one particular model.
    • Hence, it looks like the foundation model will have the same strategic importance as the operating system in consumer devices like smartphones and tablets.
    • Second, AI store: This is the equivalent of an app store on iOS or Android and provides a marketplace for developers to sell the services that they create on the foundation model of the store’s owner.
  • This is a precise repeat of the strategy that we saw in 2008 from Apple and in 2012 from Google. It allowed the smartphone ecosystem to grow into the behemoth that it is today.
  • It is also how the owners of the foundation models intend to grow and monetize the AI ecosystem. If a platform can become the go-to place to create, buy or sell a service then a lot of money can be made.
  • This is why during 2024 we will see a lot of launches of both SDKs and stores as the contenders for the AI ecosystem begin jostling for position.
  • The outcome of this battle will define who wins and who loses in the AI ecosystem. And there are vast amounts of money at stake, given how useful generative AI can be.
  • The early leader is OpenAI which seemed to have the race sown up but immediately after launching its SDK and GPT Store, a total self-immolation which is far from resolved has opened the door for everyone else.
  • Many developers who have already committed to using GPT are now much less certain about their choice. The corporate instability raises questions about the long-term viability of GPT as a development platform.
  • Most of the other players are simple, for-profit companies, which means that committing to use Gemini from Google as the foundation is immediately less risky.
  • Furthermore, I remain far from convinced that OpenAI is massively better than anyone else. It did come to market first but appears to have subsequently lost its lead.
  • Hence, I think that the AI ecosystem remains wide open and just like the smartphone ecosystem, I suspect that there will be 3 to 5 large players who take most of the market with a sprinkling of smaller niche players around the edge.
  • It is this battle to be one of the 3 to 5 that is likely to commence this year and we will see this played out at developer conferences and events where the AI ecosystem tools will be launched.
  • OpenAI’s competitors are much more attractive than they were a few months ago as their governance structures are much less flawed.
  • Hence, as a developer, I would have far more confidence that Google and Meta will not blow up in the same way that Open AI did, although they still make plenty of silly mistakes just like everyone else.
  • 2023 was the year of training but I think this will begin to give way to inference in 2024 as algorithms begin to be deployed and the battle for the AI ecosystem heats up.

(This guest post was written by Richard Windsor, our Research Director at Large. This is a version of a blog that first appeared on Radio Free Mobile. All views expressed are Richard’s own.)

Over 1 Billion Generative AI (GenAI) Smartphones to be Shipped Cumulatively During CY2024-2027

  • The smartphone industry is set for a revolutionary change with the introduction of Generative AI-based devices. 
  • The share of GenAI smartphones will be 4% of the market in 2023 and is likely to double next year.
  • By 2027, we expect the GenAI smartphone share to reach 40% and surpass half a billion in shipments.
  • Samsung will capture half of this market next year followed by key Chinese OEMs like Xiaomi, vivo, HONOR and OPPO.
  • Qualcomm and Samsung are immediate leaders as current product offerings and capabilities position them as first movers.
  • Qualcomm is likely to capture over 80% of the GenAI smartphone market for the next two years. MediaTek is likely to catch up with its Dimensity 9300-based devices.

Seoul, Beijing, Boston, Hong Kong, London, New Delhi, Taipei, Tokyo – December 20, 2023

The coming year will be pivotal for Generative AI (GenAI) smartphones with preliminary data projecting their shipments to reach over 100 million units in 2024, according to an upcoming report, “GenAI Smartphone Shipments and Insights”, from Counterpoint Research’s Smartphone 360 Service. By 2027, we expect GenAI smartphone shipments to reach 522 million units, growing at a CAGR of 83%.

Counterpoint Research defines GenAI smartphones as a subset of AI smartphones that uses Generative AI to create original content, rather than just providing pre-programmed responses or performing predefined tasks. These devices will run size-optimized AI models natively and come with certain hardware specifications. Our short-term GenAI landscape sees OEM roadmaps touching on four main areas – info provisioning, image building, live translation, and personal assistant applications.

Samsung and Qualcomm are immediate leaders as current product offerings and capabilities position them as first movers.

Similar to what it did with foldables, Samsung is likely to capture almost 50% share for the next two years, followed by key Chinese OEMs like Xiaomi, vivo, HONOR and OPPO. Samsung earlier highlighted the use case of its Galaxy AI on smartphones. This is one example of how OEMs are geared to differentiate their upcoming smartphones and GenAI will play a key role in that differentiation.

Qualcomm is likely to capture over 80% of the GenAI smartphone market for the next two years. MediaTek is likely to catch up with its Dimensity 9300-based devices.

Research Director Tarun Pathak said, “The share of GenAI smartphones in the overall smartphone market will be in single digits through next year. But those numbers will not accurately reflect the amount of excitement and marketing hyperbole we are expecting to see.”

Pathak added, “We are working to a standard definition developed with our clients and partners, reflecting inputs from across these key players. Next year is about learning and we expect GenAI smartphones to hit an inflection point in 2026 as the devices permeate through the broader price segments.”

Global GenAI Smartphone Share and Forecast, 2023-2027

Global GenAI Smartphone Share and Forecast, 2023-2027

Source: Counterpoint Research Smartphone 360 Service, GenAI Smartphone Shipments and Insights Report

VP & Research Director Peter Richardson said, “AI has been a feature of smartphones for the last few years. We now expect to see the emergence of smartphones optimized to run GenAI models in addition to the normal use of AI in smartphones. The likely use cases will include creating more personalized content, smarter digital assistants with unique personalities and conversation styles, content recommendations, and more. However, this will also require addressing issues like running into memory constraints and will likely lead to a hybrid approach in some cases. Nonetheless, one thing is for certain – we are entering an era where smartphone users no longer need to align to their devices; it will be the other way around with GenAI smartphones.”

Background

Counterpoint Technology Market Research is a global research firm specializing in products in the TMT (technology, media and telecom) industry. It services major technology and financial firms with a mix of monthly reports, customized projects, and detailed analyses of the mobile and technology markets. Its key analysts are seasoned experts in the high-tech industry.

Follow Counterpoint Research

press(at)counterpointresearch.com

Guest Post: Meta’s Instagram Fee Proposal Priced to Fail?

Meta doesn’t expect anyone to pay for Instagram.

Meta’s proposal to charge $10.5 per month for Instagram looks deliberately priced to fail, creating a strong incentive for users in the EU to explicitly allow targeted advertising which will also test just how valuable the service is to its users. 

Meta Platforms is currently embroiled in negotiations with the EU with regard to its business model where the regulators’ view is that just because users clicked “Agree” to an agreement, this does not give Meta the right to target them with advertisements. This is an ongoing issue as Meta was fined €390 million by Ireland’s Data Privacy Commissioner for precisely this activity and was told it had to think of something else. 

The monetization model for digital ecosystems can have three methods – hardware, advertising and subscription. Advertising and subscription are mutually exclusive and the option for users to choose one or the other is now commonplace across many types of digital services. 

RFM Research conducted a study in 2018 that examined the viability of digital ecosystems switching from advertising to subscription as their source of revenue. The results of the study indicated that while X and Snap could probably make the switch without risking damage to their revenue base, the same was not true for the larger players such as Google and Meta Platforms. This is because although they generate an average revenue per user (ARPU) of around $3.00-$3.60 per month, the distribution of ARPU within the user base is very large. 

For example, even though the US makes up a small portion of the user base, it often accounts for as much as 50% of revenues, indicating that US users generate far more advertising revenues per user than non-US users. This creates a pricing problem for subscription because in order to ensure that revenue is not lost, the price would have to be so high that no one would ever pay it. 

The $3.00-$3.60 ARPU includes all of Meta’s properties and so it is easy to deduce that ARPUs for Instagram in the EU are likely to fall well short of $1 per user per month. Hence, a price of $10.5 per user per month represents a price increase of more than 950% and it’s pretty clear that no user in the EU will be willing to pay it. 

It seems that this is precisely what Meta Platforms is aiming at. In order to keep the regulator happy, it provides an option that no one will want, and assuming that the users still want to use Instagram, they will sign up for targeted advertising in a way that satisfies the EU. 

The risk of this strategy is that users decide that Instagram is actually not that important and stop using it entirely, although its engagement and user metrics indicate that this is a very small risk. Instead, what can be expected is that Meta makes this offer and almost all of its users sign up for targeted advertising and life returns to normal. 

Hence, there doesn’t seem to be any meaningful implications from this issue, although there may be another fine in the works for infractions that Meta may have committed in the past. 

(This guest post was written by Richard Windsor, our Research Director at Large.  This first appeared on Radio Free Mobile. All views expressed are Richard’s own.) 

Related Posts

Guest Post: OpenAI Haymaker?

OpenAI makes hay while the sun shines.

When a company that has issues with making profits can raise money at a valuation of $85 billion, it becomes abundantly clear that investors in generative AI have taken leave of their senses. 

Open AI is reportedly raising money at a valuation of $80 billion to $90 billion. This looks like an opportunistic event for two reasons. 

First, doubts over whether Open AI actually needs the money. It was only nine months ago that Microsoft invested $10 billion in OpenAI, meaning that if it has run out of money already, then it has a cash burn of $1.1 billion per month. This is Reality Labs’ levels of cash burn which with 400 employees amounts to $2.75 million per employee per month.  

The vast majority of this spend will be going to compute costs where even with 100 million users making 30 requests per day this is an uneconomic level of spending. This would mean that ChatGPT and generative AI generally can never become a viable business or generate a positive ROI and so one suspects that OpenAI has in fact got plenty of money left. 

Second, virtually free money. In the market’s mind, OpenAI is the leading generative AI company in the world (which is debatable). Furthermore, generative AI is the hottest theme in the technology sector by a wide margin, meaning that OpenAI sits at the pinnacle of what the market wants to own. This in turn means that OpenAI can sell far fewer shares for the money it wants to raise, and its existing shareholders can also register large unrealized gains on their balance sheets. Consequently, I think that this raise is opportunistic in that the market has given OpenAI an opportunity to capitalize on its fame and popularity. 

However, most telling of all is that employees will also have an opportunity to sell some of their shares as part of this transaction. Insider stock sales are often an indicator of the insiders’ view that the valuation of the shares has hit a peak. At $85 billion, this is pretty hard to argue against. 

OpenAI is supposed to earn revenues of $250 million this year and $1 billion next year, putting the shares on over 80x 2024 revenues. This is very high even in the best of times, but the plethora of start-ups and the thousands of models being made available for free by the open-source community leads one to think that competition is on the way. 

Hence, price erosion is likely which in turn could lead to OpenAI missing the $1-billion revenue estimate for 2024 and burning through even more cash than expected. OpenAI will not be alone, and many start-ups will suffer from price erosion that will cause their targets to be missed. This could well be the pin that pricks the current bubble, causing enthusiasm to wane and valuations to fall. 

OpenAI may not be worth $85 billion but the timing of the raise is perfect.

(This guest post was written by Richard Windsor, our Research Director at Large.  This first appeared on Radio Free Mobile. All views expressed are Richard’s own.) 

Related Posts

Podcast #69: ChatGPT and Generative AI: Differences, Ecosystem, Challenges, Opportunities

Generative AI has been a hot topic, especially after the launch of ChatGPT by OpenAI. It has even exceeded Metaverse in popularity. From top tech firms like Google, Microsoft and Adobe to chipmakers like Qualcomm, Intel, and NVIDIA, all are integrating generative AI models in their products and services. So, why is generative AI attracting interest from all these companies?

While generative AI and ChatGPT are both used for generating content, what are the key differences between them? The content generated can include solutions to problems, essays, email or resume templates, or a short summary of a big report to name a few. But it also poses certain challenges like training complexity, bias, deep fakes, intellectual property rights, and so on.

In the latest episode of ‘The Counterpoint Podcast’, host Maurice Klaehne is joined by Counterpoint Associate Director Mohit Agrawal and Senior Analyst Akshara Bassi to talk about generative AI. The discussion covers topics including the ecosystem, companies that are active in the generative AI space, challenges, infrastructure, and hardware. It also focuses on emerging opportunities and how the ecosystem could evolve going forward.

Click to listen to the podcast

Click here to read the podcast transcript.

Podcast Chapter Markers

01:37 – Akshara on what is generative AI.

03:26 – Mohit on differences between ChatGPT and generative AI.

04:56 – Mohit talks about the issue of bias and companies working on generative AI right now.

07:43 – Akshara on the generative AI ecosystem.

11:36 – Akshara on what Chinese companies are doing in the AI space.

13:41 – Mohit on the challenges associated with generative AI.

17:32 – Akshara on the AI infrastructure and hardware being used.

22:07 – Mohit on chipset players and what they are actively doing in the AI space.

24:31 – Akshara on how the ecosystem could evolve going forward.

Also available for listening/download on:

 

Guest post: AI Business Model on Shaky Ground

OpenAI, Midjourney and Microsoft have set the bar for chargeable generative AI services with ChatGPT (GPT-4) and Midjourney costing $20 per month and Microsoft charging $30 per month for Copilot. The $20-per-month benchmark set by these early movers is also being used by generative AI start-ups to raise money at ludicrous valuations from investors hit by the current AI FOMO craze. But I suspect the reality is that it will end up being more like $20 a year.

To be fair, if one can charge $20 per month, have 6 million or more users, and run inference on NVIDIA’s latest hardware, then a lot of money can be made. If one then moves inference from the cloud to the end device, even more is possible as the cost of compute for inference will be transferred to the user. Furthermore, this is a better solution for data security and privacy as the user’s data in the form of requests and prompt priming will remain on the device and not transferred to the public cloud. This is why it can be concluded that for services that run at scale and for the enterprise, almost all generative AI inference will be run on the user’s hardware, be it a smartphone, PC or a private cloud.

Consequently, assuming that there is no price erosion and endless demand, the business cases being touted to raise money certainly hold water. While the demand is likely to be very strong, I am more concerned with price erosion. This is because outside of money to rent compute, there are not many barriers to entry and Meta Platforms has already removed the only real obstacle to everyone piling in.

The starting point for a generative AI service is a foundation model which is then tweaked and trained by humans to create the service desired. However, foundation models are difficult and expensive to design and cost a lot of money to train in terms of compute power. Up until March this year, there were no trained foundation models widely available, but that changed when Meta Platforms’ family of LlaMa models “leaked” online. Now it has become the gold standard for any hobbyist, tinkerer or start-up looking for a cheap way to get going.

Foundation models are difficult to switch out, which means that Meta Platforms now controls an AI standard in its own right, similar to the way OpenAI controls ChatGPT. However, the fact that it is freely available online has meant that any number of AI services for generating text or images are now freely available without any of the constraints or costs being applied to the larger models.

Furthermore, some of the other better-known start-ups such as Anthropic are making their best services available online for free. Claude 2 is arguably better than OpenAI’s paid ChatGPT service and so it is not impossible that many people notice and start to switch.

Another problem with generative AI services is that outside of foundation models, there are almost no switching costs to move from one service to another. The net result of this is that freely available models from the open-source community combined with start-ups, which need to get volume for their newly launched services, are going to start eroding the price of the services. This is likely to be followed by a race to the bottom, meaning that the real price ends up being more like $20 per year rather than $20 per month. It is at this point that the FOMO is likely to come unstuck as start-ups and generative AI companies will start missing their targets, leading to down rounds, falling valuations, and so on.

There are plenty of real-world use cases for generative AI, meaning that it is not the fundamentals that are likely to crack but merely the hype and excitement that surrounds them. This is precisely what has happened to the Metaverse where very little has changed in terms of developments or progress over the last 12 months, but now no one seems to care about it.

(This guest post was written by Richard Windsor, our Research Director at Large.  This first appeared on Radio Free Mobile. All views expressed are Richard’s own.) 

Related Posts

TSMC Bullish on AI in Long Term

Weaker-than-expected macroeconomic situation continued to weigh on TSMC’s Q2 2023 business performance. Muted smartphone and PC/NB demand negatively impacted the overall utilization rate during the quarter. Though largely expected by the market, the company further cut its fullyear revenue guidance on the weaker end demand expected for H2 2023. However, TSMC projects a strong AI demand in Q3 2023 and, going forward, sees itself as the key enabler for AI GPUs and ASICs that require a large die size. We give our takes on the key points discussed during the earnings call: 

Is AI semiconductor demand real?

  • Chairman (Mark Liu): Neither can we predict the near future, meaning next year, how the sudden demand will continue or will flatten out. However, our model is based on the data center structure. We assume a certain percentage of the data center processors are AI processors and based on that, we calculate the AI processor demand. And this model is yet to be fitted to the practical data later on. But in general, I think the trend of a big portion of data center processors will be AI processors is a sure thing. And will it cannibalize the data center processors? In the short term, when the capex of the cloud service providers is fixed, yes, it will. It is. But as for the long term, when their data service – when the cloud services have the generative AI service revenue, I think they will increase the capex. That should be consistent with the long-term AI processor demand. And I mean the capex will increase because of the generative AI services.
  • Adam Chang’s analyst take: Supply chain checks reveal that cloud service providers such as Microsoft, Google, and Amazon aggressively invest in AI servers. NVIDIA is continuing to add orders for the A100 and H100 to the supply chain, echoing the strong momentum for AI demand. TSMC holds a significant market share in AI semiconductor wafer production, mitigating the risk of misjudging CoWoS capacity expansion concerning AI demand.
  • Akshara Bassi’s analyst take: Over the medium term, as hyperscalers continue to develop their own proprietary AI models and look to monetize through AI-as-a-Service and simiilar models, the infrastructure demand should remain robust.

Can AI semiconductor demand offset short-term macro weakness?

  • CEO (Che-Chia Wei): Three months ago, we were probably more optimistic, but now it’s not. Also, for example, China economy’s recovery is actually also weaker than we thought. And so, the end market demand actually did not grow as we expected. So put all together, even if we have a very good AI processor demand, it’s still not enough to offset all those kinds of macro impacts. So, now we expect the whole year will be -10% YoY.
  • Adam Chang’s analyst take: Although there is a lot of promise around AI, it would only account for around 6% of total revenues in 2023. Therefore, AI is not a panacea for broader short-term demand weakness.

Is TSMC CoWoS capacity enough to fulfill current AI demand?

  • CEO (Che-Chia Wei): For AI, right now, we see very strong demand, yes. For the front-end part, we don’t have any problem to support, but for the back end, the advanced packaging side, especially for the CoWoS, we do have some very tight capacity to — very hard to fulfill 100% of what customers needed. So, we are working with customers for the short term to help them to fulfill the demand, but we are increasing our capacity as quickly as possible. And we expect these tightening will be released next year, probably toward the end of next year. Roughly probably 2x of the capacity will be added.
  • Adam Chang’s analyst take: Due to TSMC’s CoWoS capacity constraints, the company is finding it challenging to fulfill the strong AI demand from customers,, including NVIDIA, Broadcom, and Xilinx, at the moment. NVIDIA is actively seeking second- source suppliers as TSMC looks to outsource some of its production.

N3E/N3/N2 status

  • CEO (Che-Chia Wei): N3 is already involved in production with good yield. We are seeing robust demand for N3 and we expect a strong ramp in the second half of this year, supported by both HPC and smartphone applications. N3 is expected to continue to contribute mid-single-digit percentage of our total wafer revenue in 2023. Our N2 technology development is progressing well and is on track for volume production in 2025. Our N2 will adopt a narrow sheet transistor structure to provide our customers with the best performance, cost, and technology maturity.
  • Adam Chang’s analyst take: Apple is the sole customer expected to adopt TSMC’s 3nm technology in its A17 Bionic and M3 chips during 2023. The Qualcomm Snapdragon 8 Gen 4 processor is also anticipated to join the TSMC 3nm family (N3E) in 2024. Moreover, Intel is likely to adopt TSMC’s 3nm technology for its Arrow Lake CPU, scheduled to launch in H2 2024. 

Results summary

  • Q2 2023 results beat slightly: TSMC reported $15.67 billion in sales, slightly above the midpoint of guidance. EPS beat consensus due to higher non-operating income. Both GPM and OPM slightly beat guidance thanks to favorable FX and cost control efforts.
  • Q3 2023 guidance in line: The management guided $16.7-$5 billion (+9% QoQ at midpoint), gross margin in the range of 51.5%-53.5%, and operating margin in the range of 38%-40%. The gross margin dilution resulting from the N3 ramp-up would be 2-3 percentage points in Q3 2023 and 3-4 percentage points in Q4 2023. This impact would persist throughout the entire year of 2024, affecting the overall gross margin by 3-4 percentage points. Notably, this dilution is higher than the 2-3 percentage points gross margin dilution experienced during the N5’s second year of mass production in 2021.
  • 2023 revenue guidance revised down but expected: TSMC revised down the full-year revenue guidance to -10% YoY. The management sees weaker-than-expected macroeconomics in H2 2023 affecting the demand for all applications except for AI.
  • Strong AI demand, 50% revenue CAGR forecast: AI revenue currently makes up 6% of TSMC’s total revenue. The company anticipates a remarkable compound annual growth rate (CAGR) of nearly 50% from 2022 to 2027 in the AI sector. As a result of this significant growth, the AI revenue percentage share in TSMC’s total revenue is projected to reach the low teens by 2027.
  • CoWoS capacity expected to double by 2024 end: TSMC is experiencing strong demand in the AI sector, with sufficient capacity for the front-end part but facing challenges in advanced packaging, particularly CoWoS.It is working with customers to meet demand in the short term while rapidly increasing capacity which it expects to double by the end of 2024, easing the current tightness.

Related Posts

AI Drives Cloud Player Capex Amid Cautious Overall Spend

  • Cloud service providers’ capex is expected to grow by around 8% YoY in 2023 due to investments in AI and networking equipment.
  • Microsoft and Amazon are among the highest spenders as they invest in data center development. Microsoft will spend over 13% of its capex on AI infrastructure.
  • AI infrastructure can be 10x-30x more expensive than traditional general-purpose data center IT infrastructure.
  • Chinese hyperscalers’ capex is decreasing due to their inability to access NVIDIA’s GPU chips, and decreasing cloud revenues.

New Delhi, Beijing, Seoul, Hong Kong, London, Buenos Aires, San Diego – July 25, 2023

Global cloud service providers will grow capex by an estimated 7.8% YoY in 2023, according to the latest research from Counterpoint’s Cloud Service. Higher debt costs, enterprise spending cuts and muted cloud revenue growth are impacting infrastructure spend in data centers compared to 2022.

Commenting on the large cloud service providers’ 2023 plans, Senior Research Analyst Akshara Bassi said, “Hyperscalers are increasingly focusing on ramping up their AI infrastructure in data centers to cater to the demand for training proprietary AI models, launching native B2C generative AI user applications, and expanding AIaaS (Artificial Intelligence-as-a-Service) product offerings”.

According to Counterpoint’s estimates, around 35% of the total cloud capex for 2023 is earmarked for IT infrastructure including servers and networking equipment compared to 32% in 2022.

Global Cloud Service provider's Capex
Source: Counterpoint Research
2023 Capex Share
Source: Counterpoint Research

In 2023, Microsoft and Amazon (AWS) will account for 45% of the total capex. US-based hyperscalers will contribute to 91.9% of the overall global capex in 2023.

Chinese hyperscalers are spending less due to slower growth in cloud revenues amid a weak economy and difficulties in acquiring the latest NVIDIA GPU chips for AI due to US bans. The scaled-down version – A800 of the flagship A100/H100 chips – that NVIDIA has been supplying to Chinese players may also come under the purview of the ban, further reducing access to AI silicon for Chinese hyperscalers.

Global Cloud Service Provider's AI spends as % of Total Capex, 2023
Source: Counterpoint Research

Based on Counterpoint estimates, Microsoft will spend proportionally the most on AI-related infrastructure with 13.3% of its capex directed towards AI, followed by Google at around 6.8% of its capex. Microsoft has already announced its intention to integrate AI within its existing suite of products.

AI infrastructure can be 10x-30x more expensive than traditional general-purpose data center IT infrastructure.

Though Chinese players are investing a larger portion of their spends towards AI, the amount is significantly less than that of the US counterparts due to a lower overall capex.

 The comprehensive and in-depth ‘Global Cloud Service Providers Capex’ report is available. Please contact Counterpoint Research to access the report.

Background

Counterpoint Technology Market Research is a global research firm specializing in products in the technology, media and telecom (TMT) industry. It services major technology and financial firms with a mix of monthly reports, customized projects, and detailed analyses of the mobile and technology markets. Its key analysts are seasoned experts in the high-tech industry.

Analyst Contacts

Akshara Bassi

 

Peter Richardson

      

 Neil Shah

 

Follow Counterpoint Research

press@counterpointresearch.com

Related Posts

AI Chip Market: Advanced Packaging Capabilities Key Differentiating Factor

  • The requirement of computing power, latency and higher bandwidth can be met with the combination of advanced packaging and wafer fabrication techniques.
  • One of the integration techniques widely used is CoWoS (chip-on-wafer-on-substrate), a 2.5D IC integration technology that integrates logic computing and HBM chips.
  • As AI becomes more pervasive across devices and industries, vertical players such as Apple, Tesla, Google and Meta are adding tons of AI capability in their offerings and would also demand integrated AI compute for their portfolios. CoWoS seems the best bet for now.

Advanced packaging offers a good lever to enhance overall chip performance beyond traditional geometric scaling on transistors, and extend Moore’s Law over the next decade. Advanced packaging is categorized as front-end 3D which stacks chips or wafers vertically and backend 2.5D CoWoS that interconnects dies horizontally via RDL (redistribution layer) or interposer.

HBM going mainstream with demand for AI applications

The requirement of computing power, latency and higher bandwidth can be met with the combination of advanced packaging and wafer fabrication techniques, especially for high-performance computing chips. AI accelerators used to train AI models in data centers require the highest memory bandwidth available. Unlike earlier systems, which were memory-constrained, current data center architectures use a variety of techniques to overcome memory bottlenecks. One of the solutions widely implemented to boost bandwidth and memory capacity is high bandwidth memory (HBM). AI has been a big driver of HBM in GPUs even though traditionally it has been a niche technique.

HBM technology works by vertically stacking DRAM chips on top of one another and interconnected through TSVs (Through Silicon Vias) and micro-bumps. For AI training and high-performance applications, HBM can deliver terabytes per second which is significant processing power required for AI and high-performance computing applications as the DRAMs are stacked. While HBM offers extreme bandwidth for the off-chip memory needed for data center AI accelerators, tradeoffs include HBM’s cost and thermal limitations. HBM is becoming more mainstream, with advances in the area, performance improvement and reduced power consumption with every iteration, as the DRAM stack and the SoC are placed in a single package substrate, driving its adoption in AI applications.

CoWoS major driver for packaging advanced logic with HBMs

One of the integration techniques widely used is CoWoS (chip-on-wafer-on-substrate), a 2.5D IC integration technology that integrates logic computing and HBM chips by mounting them on a silicon interposer and then placing them directly on a package substrate. TSV/RDL interposers are for extremely fine-pitch, high-I/O, high-performance and high-density semiconductor IC applications.

The logic and HBMs are first bonded side by side on the silicon interposer to form chip-on-wafer (CoW) with fine pitch and high-density interconnect routing among the devices. Each HBM consists of DRAMs with micro-bumps and a logic base with TSVs straight through them. Through-Silicon Via (TSV) is the feature that enables 2.5D and 3D advanced packaging. TSVs are electrical connection pathways that are short vertical columns running through the silicon wafer or die and enable smaller package sizes and denser interconnects, improve electrical performance by shortening the electrical distance traveled, and enable stacking of multiple chips used in products like HBM. Finally, the TSV interposer’s assembly with larger bumps on a package substrate is done.

AI chip market - CoWoS integration process
Source: Counterpoint Research

CoWoS also a bottleneck for industry

Over the years, CoWoS technology development has focused on supporting increasing silicon interposer dimensions to support processors and HBM stacks in the overall package. Today, CoW with C2 bump by TCB method is the most used assembly method for silicon-to-silicon flip chip bonding. CoW flip chip by a bumpless technique called the hybrid bonding method, currently in R&D, will gain traction over the years.

CoWoS has been the main bottleneck for AI shipments recently. Increasing CoWoS capacity to fulfill the rising demand, driven by NVIDIA, AMD, Broadcom and Amazon, has been the focus.

Addressing challenges to CoWoS growth

To address these challenges, existing players need to diversify the supply chain by increasing capacity and collaboration with players in the value chain – OSATs, interposer manufacturers, and packaging equipment vendors.

What should pure-play foundries do?

TSMC has capacity constraints with respect to CoWoS creating some bottlenecks for NVIDIA and other customers. One of the reasons here is the bottleneck from equipment suppliers and manufacturing interposers. Therefore, TSMC should look to collaborate and potentially outsource to its partners, which will help stabilize volume in the near- to mid-term and meet the growing demand for advanced packaging in AI and HPC applications. So, long term, TSMC should collaboratively work with equipment manufacturers to optimize the CoWoS processing techniques such as TCB (Thermal Compression Bonding) and hybrid bonding.

What should equipment vendors do?

Equipment players should collaborate closely with foundries and co-develop optimized processes to scale the CoWoS packaging. For example, Applied Materials is actively developing new technologies for hybrid bonding and TSV to advance heterogeneous chip integration that help chipmakers integrate chiplets into advanced 2.5D and 3D packages. Similar collaborations would be expected from other players such as KLA Tencor, Lam Research, ASMPT and BESI to drive advancements in CoWoS packaging or other competing techniques such as hybrid bonding techniques to enable improvements in fine pitch, I/O density and power consumption. Research collaborations will provide semiconductor and systems companies with a complete suite of tools and technologies for developing and prototyping various packaging designs and enable continued advances in power, performance, area, cost and time-to-market.

What should IDMs do?

Samsung and other IDMs are also actively looking to take a share of the advanced packaging market and enhance their design and manufacturing capabilities to establish a sufficient customer base for high-end solutions. For example, Samsung Electronics’ HBM-PIM technology integrates processing chips and advanced memory chips like HBM with its proprietary 2.5D or 3D packaging technology. Samsung has developed HBM-PIM as a potential replacement for its existing HBM2 product where the bottom four of eight memory chips are replaced with chips containing both DRAM and compute cores. By doing some of the compute in the DRAM, the volume of data that needs to go to the processor decreases, effectively speeding up the neural network and saving the power needed to transport data. As this technology develops, Samsung will be able to leverage its advanced packaging solutions and technology as it is the only company that has a chip business portfolio spanning memory chips to contract-based processing chip fabrication and packaging and will be able to increase its market share and compete with TSMC.

Advanced packaging is also an opportunity for IDMs that have mature node capability to enhance the performance of their existing product portfolios and capture significant market share.

What should OSATs do?

OSATs like ASE and Amkor are actively investing to expand advanced packaging offerings but it will take time to benefit from the uptrend. Yield and volume output are the keys for foundries to grow their advanced packaging platform. This also poses a barrier for most traditional OSAT companies to enter this area. However, they could collaborate with leading players and increase their capacity in back-end processes. Another option could be to increase investment in R&D for advanced packaging applications in mature nodes, which will help prepare for the next upturn, like moving away from flip chip and wire bonding.

OSATs will face rough weather if they don’t join the wave of advanced packaging to achieve PPA benefits in mature nodes. But they can get a first-mover advantage if they start investing now, especially in AI applications in the IoT, automotive and network segments.

What should fabless vendors do?

NVIDIA accounts for the majority of the current CoWoS capacity. Getting the high performance required for AI workloads is possible because of the benefits being derived from CoWoS packaging. The technology offers flexibility in terms of a wide range of interposer sizes, package sizes and a number of HBM cubes, and provides the best performance with the highest integration density required for high-performance computing applications.

As AI becomes more pervasive across devices and industries, vertical players such as Apple, Tesla, Google and Meta are adding tons of AI capability in their offerings and would also demand integrated AI compute for their portfolios. CoWoS seems the best bet for now. As other competitors including Intel expand their offerings to capture the AI demand, they would have to match the best in class, which as of now is provided through CoWoS.

From the client compute perspective, as the industry develops scaled-down AI models capable of running on smartphones and users demand AI models on smartphones, it might experiment with CoWoS. However, implementing CoWoS packaging in smartphone applications at present will not be cost-effective and alternative packaging techniques will be a better option to consider.

Related links:

Artificial Intelligence: Irrational Exuberance is in Full Swing

ABF Substrate Demand Likely to Recover in H2 2023, BT Substrate Demand to Remain Weak

Currency Fluctuation Limits Global Wafer Fab Equipment Revenue Growth to 9% YoY in 2022

Wafer Fab Equipment Revenue Tracker

Applied Materials FY 2022 Earnings Report

Applied Materials Delivers Strong FY 2022 Numbers Despite Headwinds

Data Center CPU Market: AMD Surpasses Intel in Share Growth

Global Data Center CPU Revenue Tracker: Q1 2018 – Q4 2022

Term of Use and Privacy Policy

Counterpoint Technology Market Research Limited

Registration

In order to access Counterpoint Technology Market Research Limited (Company or We hereafter) Web sites, you may be asked to complete a registration form. You are required to provide contact information which is used to enhance the user experience and determine whether you are a paid subscriber or not.
Personal Information When you register on we ask you for personal information. We use this information to provide you with the best advice and highest-quality service as well as with offers that we think are relevant to you. We may also contact you regarding a Web site problem or other customer service-related issues. We do not sell, share or rent personal information about you collected on Company Web sites.

How to unsubscribe and Termination

You may request to terminate your account or unsubscribe to any email subscriptions or mailing lists at any time. In accessing and using this Website, User agrees to comply with all applicable laws and agrees not to take any action that would compromise the security or viability of this Website. The Company may terminate User’s access to this Website at any time for any reason. The terms hereunder regarding Accuracy of Information and Third Party Rights shall survive termination.

Website Content and Copyright

This Website is the property of Counterpoint and is protected by international copyright law and conventions. We grant users the right to access and use the Website, so long as such use is for internal information purposes, and User does not alter, copy, disseminate, redistribute or republish any content or feature of this Website. User acknowledges that access to and use of this Website is subject to these TERMS OF USE and any expanded access or use must be approved in writing by the Company.
– Passwords are for user’s individual use
– Passwords may not be shared with others
– Users may not store documents in shared folders.
– Users may not redistribute documents to non-users unless otherwise stated in their contract terms.

Changes or Updates to the Website

The Company reserves the right to change, update or discontinue any aspect of this Website at any time without notice. Your continued use of the Website after any such change constitutes your agreement to these TERMS OF USE, as modified.
Accuracy of Information: While the information contained on this Website has been obtained from sources believed to be reliable, We disclaims all warranties as to the accuracy, completeness or adequacy of such information. User assumes sole responsibility for the use it makes of this Website to achieve his/her intended results.

Third Party Links: This Website may contain links to other third party websites, which are provided as additional resources for the convenience of Users. We do not endorse, sponsor or accept any responsibility for these third party websites, User agrees to direct any concerns relating to these third party websites to the relevant website administrator.

Cookies and Tracking

We may monitor how you use our Web sites. It is used solely for purposes of enabling us to provide you with a personalized Web site experience.
This data may also be used in the aggregate, to identify appropriate product offerings and subscription plans.
Cookies may be set in order to identify you and determine your access privileges. Cookies are simply identifiers. You have the ability to delete cookie files from your hard disk drive.