Coffee & Code Philly

Kimi Linear: An Expressive, Efficient Attention Architecture
https://arxiv.org/abs/2510.26692

Abstract

We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule.

We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths.
To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.

This is a recurring session for IT professionals to discuss career growth and market strategy. We focus on how to build rare and valuable skills, improve professional standing, and navigate the current job market.

The Format:

• The Concept (5 mins): One core idea regarding professional development or career strategy.
• The Discussion (30 mins): The floor opens for group feedback. We share real-world experiences and peer advice to help everyone improve their career capital.
• The Next Idea (5 mins): Voting for the next core idea to discuss.

Why Join? The goal is to move beyond just writing code. We talk about what actually works in the industry right now, from technical positioning to performing better in your current role.

Join us to share your perspective and hear how others are navigating their career paths.