Why I’m betting on on-device AI

I’ve been thinking a lot about what small software teams can build with on-device AI. Useful features, more personal data kept on the device, and no cloud inference charge every time someone uses them.

If you can deliver a good experience that way, I think you have room to compete with much bigger companies.

A lot of the discussion about local models focuses on how well they compare with frontier models at answering questions or handling complicated tasks.

I use frontier models constantly. Their range is what makes them useful, and I expect them to keep pushing what general-purpose AI can do.

Inside an app, though, the job is often much narrower. Turn audio into text, find a receipt, recognize a voice, suggest a few words, or find a note based on what it means.

A model that does one of those jobs well doesn’t have to be good at everything. Combine a few, give each a specific job, and you can build something more useful than any one of them would be alone.

Most users won’t care how it works. They want their thing to work, quickly, without making their computer or phone feel slow.

Ramp’s receipt matching is a good example. Its iOS app can look for receipt photos taken near a transaction and match them using the merchant, date, and amount. The matching happens on the phone. The user reviews the suggestion, and only a receipt they approve gets uploaded.

I’d feel much more comfortable giving an app access to my photos under that arrangement. It can help me find something personal while I control what leaves the device.

The same idea applies to features involving voices or personal writing. You still need clear permissions and controls, but you can do useful work with that information without sending it away for processing. That could affect whether someone is comfortable using the feature at all.

There’s a business opportunity here, too.

Cloud APIs give developers access to capable models without having to run them. They make sense for plenty of tasks. But if each use carries an inference charge, more usage brings more cost.

When a specific job runs on the user’s device, the developer can avoid that charge.

You still have to build the software, maintain it, support people, and make sure it works on their hardware. The computation uses the user’s memory, battery, and processor.

But removing a cost tied to every use gives a small team more choices. You might charge less, offer a different payment model, or include a feature that would be expensive to run repeatedly in the cloud.

If you can deliver comparable value, I think you have a real opening against an established subscription product.

I’m also interested in what we can build beyond recreating existing apps. Working offline or handling private information locally could make a feature practical that wasn’t before. Ramp’s photo search is the kind of example I want to see more of.

The experience has to hold up, though. A small model can still run too often. Several models can use too much memory. A feature that looks impressive in a demo might drain the battery or make the rest of the app sluggish.

We need to measure those costs on the devices people own. Saving a cloud bill is a poor trade if you hand the user a slow computer.

Open-weight and on-device also mean different things. Open-weight describes access to a model’s weights. On-device describes where it runs. Some local models are open-weight; others come from the operating system.

I think small teams should be trying this now. Pick a job your users need done, see whether a local model can do it well, and measure what it costs on their hardware. If it works, you have more choices about the features you build, the information you collect, and how you charge.

Source for Ramp’s receipt matching: https://support.ramp.com/auto-match-receipts-from-your-camera-roll/