Ultrafast LLMs: Rust and CUDA for the Future RTX 5090

Someone is already figuring out how to make AI models fly on the next generation of hardware. Avifenesh developed an LLM inference engine, dubbed bw24, which promises mind-blowing performance.

It's written from scratch in Rust and CUDA, targeting the upcoming NVIDIA RTX 5090. This means it doesn't rely on existing solutions; it's built to be as efficient as possible. Think of it like building a custom race car instead of tuning a production model. The goal is to push the capabilities of a GPU that isn't even out yet.

The bw24 project stands out for its obsessive focus on optimization, leveraging advanced techniques. The bw24 project uses NVFP4, an NVIDIA numeric format, which reduces memory footprint and speeds up calculations. Plus, it supports MoE (Mixture of Experts) architectures.

These MoE architectures allow AI models to activate only the necessary parts for a given request, making inference much faster. Imagine having a team of specialists, and you only call the one you need, not everyone at once. The engine also includes "MTP speculative decoding," a technique to accelerate model output by predicting multiple responses simultaneously.

Simple: the code has been chiseled to squeeze every millisecond from the GPU. It was "tuned" against measured hardware limits to ensure no resource goes to waste. Avifenesh tested his bw24 engine against the theoretical maximum performance of NVIDIA's next architecture, known as sm_120a.

This approach ensures that when the RTX 5090 hits the market, bw24 users will have a speed and efficiency advantage. It's like having a Swiss watch, but for AI computations. Less waiting, more immediate responses, even with complex models.