Blind Vision Assistant — describing images for blind users
Blind Vision Assistant is an accessibility project that converts an image into a spoken description. A vision-language model generates a natural-language description of the image, and a text-to-speech stage reads it aloud, so someone who cannot see the image can still learn what it contains.
The problem it addresses
Screen readers handle text well and images badly. When the web platform provides a useful alt attribute, a screen reader announces it; when alt text is missing, generic or auto-generated, the user learns nothing about the image. Alt text is also written once, for everyone, so it cannot answer a follow-up question about a detail the author did not think to mention.
A model that can describe an arbitrary image on demand changes what is possible: the description is generated for this image rather than retrieved, and it can be regenerated with a different emphasis if the first attempt misses what mattered.
How the pipeline works
The system is a three-stage pipeline, which is also why it ships three interfaces.
- Ingest — the image arrives from a file path, an upload, or a REST request
- Describe — LLaVA, a vision-language model, produces a natural-language description of the visible content
- Speak — Edge-TTS converts the description to audio, so the output is available without reading it
- Web interface — upload an image and receive both the text and the audio in one flow
- CLI — the same pipeline as a local command, useful for batch work and for testing without a browser
- REST API — the pipeline exposed as an HTTP endpoint, so other applications can call it rather than reimplementing it
Design decisions worth noting
Exposing three interfaces over one pipeline was deliberate. The interesting engineering in an accessibility tool is the pipeline itself; the interface is a distribution decision. Building the pipeline first and then wrapping it meant the model stage could be evaluated independently of any particular way of reaching it.
Using a local vision-language model rather than a hosted API keeps the project runnable without a paid key and keeps uploaded images on the machine running it, which matters for an accessibility tool where users may be describing personal photographs.
What would make this stronger
The honest gap in this project is evaluation. A description system without a measured error rate is a demonstration, not a contribution: the useful question is how often the model invents content that is not in the image, since a confident wrong description is worse for a blind user than no description at all.
Measuring that means running the pipeline against a fixed sample of images, comparing each generated description with a reference description, and reporting hallucination rate and latency alongside the qualitative examples. That evaluation is the natural next piece of work on this project.
At a glance
- Web Interface
- CLI Tool
- RESTful API
- Accessibility Focused