Imagine an AI model, built for text, that can now 'look' at images. It sounds like magic, but someone found the trick to make it work in Codex.
In 30 seconds
01Anionex's proxy lets text-only models 'see' images directly within Codex.
02It enables text LLMs to use Codex's `view_image` tool without modifications.
→
💡
What this means for you
Now your text-only AI assistant can understand and describe images, without needing a super expensive model. This means smarter, more versatile AI agents for everyone, with less hassle.
Ever chat with an AI only for it to forget your last five messages? A new open-source project aims to fix that pesky short-term memory issue.
·1 min·2·Intermediate
03Builds multimodal AI agents cost-effectively, skipping complex model retraining.
0101
Can text models actually "see"?
Yes, in a way. A GitHub project by Anionex found a clever method to let purely text-based language models use Codex's vision tool. This means that even if a model wasn't trained on images, it can now "consult" them through an intermediary.
Imagine your favorite AI assistant, the one great at chatting, but previously blind as a bat. Thanks to this solution, it can now ask a "friend" to describe what's in a photo. The Anionex/codex-vision-proxy project, published on GitHub, introduces a method to integrate visual capabilities. The model doesn't grow eyes, but gains indirect access.
0202
How does this "trick" work?
The secret lies in a "proxy" and a "vision toolkit" designed specifically for text models. Basically, the proxy intercepts the model's requests. When an image needs to be "seen," it passes the request to Codex's view_image tool. Then, the result is converted back into text the model can understand.
📬 Enjoying this article?
Get the best AI news every week, straight to your inbox.
It's a bit like having a simultaneous interpreter for images. The "vision toolkit & skill" included with the Anionex project provides the model with the necessary instructions and abilities to interact with this interpreter. So, an that isn't natively can suddenly act as if it were, using external tools. Not bad for a model that only read words yesterday, right?
0303
Why should I care about this?
This means you can build more versatile and intelligent AI agents without spending a fortune training multimodal models from scratch. For developers, or anyone who just wants an AI to do more, it's a brilliant shortcut. It opens up possibilities for chatbots that understand screenshots or assistants that analyze graphs.
Instead of waiting for OpenAI or Google to release a new "omniscient" model, this solution offers a way to add visual capabilities to existing, less expensive models. The Anionex/codex-vision-proxy project, available on GitHub, allows text models to access visual tools, lowering the entry barrier for creating multimodal agents. Who knew a simple "proxy" could do so much?
