Skip to content
← All writing
August 15, 2026·6 min read

Multimodal AI Features: Voice and Image Input in Your App

The keyboard is the worst input device your app supports, and on a phone it's the default. Watch someone thumb-type a receipt into an expense tracker and you're watching friction win. Modern LLMs accept images directly, and speech models turn audio into clean text — which means the two fastest ways to get information into an app (point the camera, just talk) are now practical for any developer to build. Receipt scanning, "what is this?" features, voice-first capture: these used to be specialist computer-vision projects with training pipelines and labelled datasets; now they're API calls. The engineering that remains is mostly about latency, cost, and honest UX — and that's what this post covers.

Image input: the model can just look

The pattern that's changed is that you no longer need OCR pipelines or custom vision models for most tasks — you send the image to a multimodal LLM with instructions and an output schema, and get structured data back:

"Extract merchant, date, currency, line items and total from this receipt. Return JSON matching this schema. If a field isn't legible, use null."

That last sentence is the important one. Real photos are blurry, cropped, shadowed, and weirdly lit; a model told to always produce a total will invent a total, in perfect confident JSON. Giving it explicit permission to say "illegible" converts hallucination into a UX state you can design for — a highlighted field asking the user to fill in what the camera couldn't read.

What used to require training a model per document type now handles receipts, business cards, forms, whiteboards, and menus with a prompt change each. The same mechanism does description ("what is this plant?"), comparison, and screenshot understanding. If your app has any "user transcribes something they're looking at" flow, this is the feature that removes it.

Practical notes that keep image features healthy:

  • Downscale before upload. A twelve-megapixel photo is wasted tokens and wasted seconds — models read detail fine at much smaller sizes, and providers resize internally anyway. Resize to roughly a thousand pixels on the long edge (keep text-heavy images on the higher end), compress to JPEG or WebP, then send. Your users on cellular connections will feel the difference immediately.
  • Route through your backend, same as every AI call — the app uploads the image to your endpoint, your server holds the key, and you get a natural place for auth, rate limits, and logging.
  • Validate the output like any structured response. Schema-check the JSON, range-check the numbers (a receipt total of $4,500,000 deserves suspicion), and treat validation failure as "ask the user," not "crash."
  • Design the correction step. Show the extracted fields in an editable form, not a triumphant "Done!" A user fixing one wrong digit is a good interaction; discovering a wrong total three weeks later destroys trust in the whole feature. The confirmation screen isn't admitting weakness — it's what makes the feature trustworthy enough to keep using.

Voice input: transcription is the easy 80%

For most apps, "voice" means speech-to-text feeding your existing features — voice capture of an expense, a note, a search query. The architecture is refreshingly boring: record audio, send it to a transcription model (or use the platform's on-device dictation, which is free and private), then treat the text like any other input — usually piping it through an LLM to extract structure ("coffee four fifty yesterday" → amount: 4.50, category: coffee, date: yesterday's date).

That second step is where the magic actually lives. Raw transcripts are messy — "uh, add like, fifty bucks for gas, no wait, forty" — and an LLM instructed to extract the final intent handles the self-corrections and filler words that would break a regex-based parser instantly.

The traps are in the details:

  • Show the transcript. Users need to see what the machine heard and fix it before it commits — same correction-step principle as images. Voice without a visible transcript feels like dictating to someone who won't read it back.
  • Names and jargon get mangled. Most transcription APIs accept a prompt or vocabulary hint; feeding in the user's contact names, your product's terms, or the domain's acronyms fixes the most common errors cheaply.
  • Design for noise and accents, which means designing for failure: low-confidence results should ask, not guess. A "did you mean…?" prompt costs a second; a silently wrong entry costs trust.
  • Respect the microphone. Record only while the user holds or toggles a clearly-labelled button, show a visible recording indicator, and never leave a session open longer than needed. Both app stores review microphone usage closely, and users are rightly sensitive about it.

Full duplex voice conversation — the app talks back, interruptions work, latency stays under a second — is a different tier of difficulty involving realtime audio streaming APIs, echo cancellation, and turn-taking logic. It's achievable, but know the tier you're signing up for; most apps get outsized value from the boring tier and should start there.

The rules that don't change

Multimodal features are still LLM features. Everything you already do for text applies unchanged: backend proxy, structured output with validation, cost ceilings, evals built from real examples. Two things get more important:

Cost per call is higher. Images cost more tokens than text, audio bills by the minute, and both invite casual heavy use — "just photograph everything" is exactly the behaviour a good feature creates. Per-user limits and the measure-first discipline stop a delightful feature from quietly becoming your top expense. Build your eval set from real photos and real recordings taken on phones, too — pristine test images will tell you the feature works right up until real users prove otherwise.

Privacy is sharper. Photos and voice recordings are personal in a way typed text isn't — they contain faces, homes, documents in the background, other people's voices. Send media only to providers who commit not to train on your API data, don't retain uploads longer than processing requires, and say plainly in your privacy policy what happens to a photo after it's scanned. "We extract the data and discard the image within X days" is a sentence users deserve — and both Apple's and Google's data-safety declarations will require you to account for it anyway.

The takeaway

Voice and image input are the highest-leverage AI features for mobile precisely because they attack input friction — the thing phones are worst at. Architecturally they're the same discipline as any LLM feature plus three habits: shrink media before sending, let the model admit it can't read something, and always give the user a correction step before anything commits. Point, speak, confirm — that's the whole magic, and it's an API call away.

Further reading