Semiconductor Engineering

Ultra-low-bit LLM Inference Allows AI-PC CPUs And Discrete Client GPUs To Approach High-end GPU-Level (Intel)

A new technical paper titled “Pushing the Envelope of LLM Inference on AI-PC and Intel GPUs” was published by researcher at ...

AI Infrastructure Evolution: How Better Hardware Powers The LLM Era

Running both phases on the same silicon creates inefficiencies, which is why decoupling the two opens the door to new ...

The New Frontier Of LLM Inference: Where The Next Tenfold Gains Will Come From

This brute-force scaling approach is slowly fading and giving way to innovations in inference engines rooted in core computer ...
NextBigFuture

Defeating Nondeterminism in LLM Inference by Thinking Machines

A research article by Horace He and the Thinking Machines Lab (X-OpenAI CTO Mira Murati founded) addresses a long-standing issue in large language models (LLMs). Even with greedy decoding bu setting ...
WinBuzzer

AI: Memory Bottleneck Emerges as Main LLM Inference Challenge

Google researchers have revealed that memory and interconnect are the primary bottlenecks for LLM inference, not compute power, as memory bandwidth lags 4.7x behind.
9to5Mac

Apple collaborates with NVIDIA to research faster LLM performance

In a blog post today, Apple engineers have shared new details on a collaboration with NVIDIA to implement faster text generation performance with large language models. Apple published and open ...
Business Wire

Red Hat Launches the llm-d Community, Powering Distributed Gen AI Inference at Scale

Forged in collaboration with founding contributors CoreWeave, Google Cloud, IBM Research and NVIDIA and joined by industry leaders AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI and university ...
Semiconductor Engineering

Four Architectural Opportunities for LLM Inference Hardware (Google)

“Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerbated by recent AI ...