AI breakthrough: non-autoregressive models built a year ago

A year ago, while many were busy with buzzwords, Nandakishor M. was building something tangible: non-autoregressive decision models. These models are his creation, born and raised far from the spotlight of big tech companies. Then, one fine day, a "frontier lab" announced the same idea as an epoch-making discovery. Curious, isn't it?

Nandakishor M. developed non-autoregressive decision models approximately a year ago, documenting his work on dev.to. Essentially, he had already solved a problem before the "big players" even realized it existed. It's a bit like finding a treasure and then seeing someone else celebrate for discovering it on the same map.

This episode raises a question: how often do major innovations emerge quietly, only to be "rediscovered" with great fanfare later? It almost seems like marketing is sometimes faster than research itself.

Simply put, non-autoregressive models are like a team working in parallel, not one after the other. Instead of waiting for one piece to finish before starting the next, they do everything simultaneously. Imagine an assembly line producing all components at once, not one by one.

Non-autoregressive models, unlike autoregressive ones like GPT, generate output in parallel, not sequentially. This means they can be much faster and more efficient. Less waiting, more results. Perfect for situations where every millisecond counts, like in a self-driving car or a trading system.

Their efficiency can reduce AI response times, making it more practical for real-world applications. Who wouldn't want an AI that doesn't make you wait hours for an answer? A significant step forward for everyday usability.

Often, the academic research world and the industry world travel on different tracks. While individual developers or small teams experiment in the field, large labs take time to validate, publish, and then announce discoveries. Sometimes, too much time.

The original article on dev.to, published on June 12, 2024, highlighted this discrepancy. It's a reminder that the best ideas don't always come with a multi-million dollar budget or a shiny press release. In fact, they often originate in a garage, or perhaps during an engineer's spare time.

This makes us reflect on the hierarchy of innovation. Is it truly a breakthrough only when a "frontier lab" says so? Or perhaps we should give more credit to those who build and share first, regardless of the name they carry?