OpenAI-like API for open models: speed up, costs down

Imagine wanting to use a powerful AI model, maybe an open source one, but your application is already set up to talk to OpenAI's API. What do you do, rewrite everything? No thanks. Ekzhang created openjev-sglang, a solution that acts as a "universal translator" for open models.

This tool lets developers trick their apps into thinking they're talking to the OpenAI API, when they're actually querying an open source model. The benefit? It saves a ton of money and avoids the "golden handcuffs" of a single vendor. It's a bit like having a Ferrari engine that runs on supermarket gas.

Yes, but with a small asterisk. This API endpoint is designed to handle "prefill" requests super efficiently, meaning initial responses or fast text generation. It's perfect for when you need a first email draft, a summary, or a caption.

The current version of openjev-sglang, developed by Ekzhang, is "prefill-only," ideal for generating quick initial responses. It's not ideal for long, detailed conversations, like the ones you have with ChatGPT for hours. For that, you might still need to shell out more cash or wait for the next version. But for the "first punch," it's a blast.