Flashkda Delta Attention

This is a CUDA kernel implementation for Kimi's Delta Attention architecture, built on CUTLASS and targeting H100-class GPUs. It's designed as a drop-in backend for flash-linear-attention's chunk_kda operation, handling the recurrent state updates that make delta attention work. You'll need SM90+ hardware and CUDA 12.9 to run it. The implementation is opinionated: it only supports 128-dim keys and values, and works exclusively with bfloat16 precision. The varlen batching support is genuinely useful for production inference where you're packing multiple sequences together. If you're already using flash-linear-attention for KDA models and have the right hardware, this should just plug in and speed things up.

skills.sh installs: 261
GitHub stars: 26
last updated: Jun 3, 2026

View on GitHub

Install

$ npx -y skills add aradotso/trending-skills --skill flashkda-delta-attention --agent claude-code

Installs into .claude/skills of the current project.

Files

FlashKDA Delta Attention Skill

Skill by ara.so — Daily 2026 Skills collection.

FlashKDA provides high-performance CUDA kernels for Kimi Delta Attention (KDA) built on CUTLASS. It targets SM90+ GPUs (H100/H20 class) and integrates as a drop-in backend for flash-linear-attention's chunk_kda operation.

Requirements

GPU: SM90+ (H100, H20, or newer)
CUDA 12.9+
PyTorch 2.4+
Python 3.8+

Installation

git clone https://github.com/MoonshotAI/FlashKDA.git flash-kda
cd flash-kda
git submodule update --init --recursive
pip install -v .

Install the FLA integration (optional but recommended):

pip install -U flash-linear-attention  # >= 0.5.0

Core Kernel API

`flash_kda.fwd`

The primary low-level kernel call:

import torch
import flash_kda

flash_kda.fwd(
    q, k, v, g, beta, scale, out,
    A_log, dt_bias, lower_bound,
    initial_state=None,
    final_state=None,
    cu_seqlens=None
)

Tensor shapes and dtypes:

SKILL.md on GitHub