Design of SOT-MRAM based In-Memory Computing accelerators for KV-Caching in Large Language Models

Keywords : Large Language Models, KV-Caching, In-Memory Computing, SOT-MRAM, Spintronics, AI Accelerators, Co-design, Post-CMOS electronics

Summary
Designing hardware accelerators to address the critical bottlenecks in deploying Large Language Models (LLMs) is far from trivial. In this project, we focus specifically on the Key-Value (KV) cache management during the decoding phase. These applications apply to diverse fields such as real-time conversational AI, edge-based generative content creation, autonomous systems, and energy-efficient data center operations.

The proposed approach relates to the design of In-Memory Computing (IMC) architectures to overcome the memory wall inherent in transformer attention mechanisms. Specifically, the KV-cache requires massive parallel reads and frequent sequential updates to store and retrieve context states, creating a severe bandwidth and energy bottleneck.
The memory elements are based on the Spin-Orbit Torque Magnetic Random Access Memory (SOT-MRAM), which is an ideal magnetoresistive technology: SOT-MRAM offers a unique combination of high write endurance, fast switching speeds, and non-volatility, making it a promising candidate for dynamic KV-cache updates. SOT-MRAM cells leverage the Spin Hall Effect for writing, providing distinct advantages in speed and reliability. To contain the ever-increasing power consumption of LLM inference, SOT-MRAM based IMC systems appear promising, in order to bring computation closer to data and to alleviate energy-costly on-chip data transfer. The key challenges to address are crossbar array density and the robustness of the write mechanism.

We are looking for a candidate with a profile oriented towards device-to-architecture co-design. The Post-Doc will join the Spintronic Computing & Architectures team at Spintec and benefit from the team’s design expertise in developing AI neuromorphic solutions based on spintronic devices. They will also work in close collaboration with the experimental team developing the SOT-MRAM devices, and in particular with PhDs aiming at using this technology as memories and computing elements.

The main tasks

Required skills

Imagination, inventiveness, interest towards innovative microelectronics applications, and towards scientific communication and intellectual property creation are expected.

Contacts
Jonathan Miquel – jonathan.miquel@grenoble-inp.fr
Lorena Anghel – lorena.anghel@grenoble-inp.fr
Anca Molnos – anca.molnos@cea.fr