Putting the Model on the Phone, Not in the Cloud

Running an AI model on a server is the easy path. The machine is powerful, memory limits do not bother you, and when you want to update the model a single deployment is enough. Even so, in most of the R&D work I have designed I planned to put the model inside the phone.

That choice is not free. I want to be as clear as I can about what I gain and what I give up in return.

First, credit where the easy path is due

Running inference in the cloud has real advantages. The size of the model barely matters; you can load a hundred-megabyte network if you want. You update in one place and the user downloads nothing. When you find a bug in the model, fixing it takes one deployment — you are not waiting in an app store review queue.

You also do not deal with device diversity. A five-year-old Android phone and a new iPhone get the same answer; you never have to think about which hardware accelerates what. These are not things to shrug off.

So I am not saying “put everything on the device.” What I am saying is narrower: the cloud is not the default, it is a choice. And in plenty of products it is the wrong one.

Four concrete wins for on-device

When I make this decision I look at four headings. Not all of them are easy to measure, but all of them are things the user feels directly.

Latency looks like a small line item on paper, but it defines the user experience one to one. In a flow that uploads a photo the user sees a progress bar, can cancel, and you are forced to answer what happens when they hit the back button. When inference runs on the device most of those screens disappear — and so does the code behind them.

Cost is the line item that decides things in the long run. When you put an image classification feature in the cloud, your expenses grow along with your free user count. On the device you break the link between user count and server bill completely. Whether a free tool survives usually comes down to exactly that.

The price you pay: size and quantization

The real price of on-device inference is this: the model has to fit inside your app. You cannot drop the network you used comfortably during training onto a phone as is.

Quantization is what steps in here. Instead of keeping the weights as 32-bit floating point numbers, you keep them as 8-bit integers. The arithmetic is simple: one byte per weight instead of four, so the space the weights take drops to roughly a quarter. Memory usage benefits from this, and on most devices so does inference time.

In return you lose accuracy. How much you lose depends on the model and the data, so there is no rule that says “it drops by this much” — you have to run the quantized model through the same test set again and see for yourself.

Size is not a single number either. The space the model takes on disk is one thing, the memory it holds while running is another. On an old phone with other things running in the background, the app may well be killed while the model is loading. Accounting for that at design time is cheaper than hunting later for “why does it crash on some devices.”

How I planned to absorb the accuracy loss

The most honest property a small model can have is being able to say it is not sure. In my designs I framed inference not as an answer on its own, but as a suggestion that arrives together with a confidence value.

      // Pseudocode: try on the device first, be honest when you are not sure
      final sonuc = await yerelModel.calistir(goruntu);

      if (sonuc.guven >= esik) {
        return Teshis.yerel(sonuc);          // the network was never touched
      }

      if (!await agVar()) {
        return Teshis.belirsiz(sonuc);       // say “I am not sure”, do not make it up
      }

      return await bulutModel.calistir(goruntu);   // heavy model, only when needed

The threshold here is not a fixed truth; it is a design parameter that has to be calibrated against images coming from the field. But the structure itself matters: the goal is for most requests to be answered without ever touching the network; the cloud steps in only for the cases the model struggles with, and only when there is a connection.

This setup has one more side benefit: the requests that fall through to the cloud are exactly the samples the model struggles with. So every request you pay for is also a data point that will be useful in the next training round.

TarımGöz: the field made the decision

I designed TarımGöz as a tool that diagnoses plant disease with the phone camera. The user is standing in the middle of a field. There is either no connection or a single bar. A flow that waits for a photo to upload to a server means a flow that does not work at exactly the moment it needs to.

That is why I planned the diagnosis model to run on-device from the start: a classifier trained with transfer learning on an open dataset, then quantization and conversion down to LiteRT and Core ML formats.

But the plan does not move the whole app onto the device. The question I asked myself to draw the line was: does this feature already depend on the internet? The irrigation calculation needs weather and satellite data; that data comes over the network anyway, so there is no harm in doing the calculation on the server. Diagnosis, on the other hand, needs no external data at all — only the photo and the model. So diagnosis runs on the device and irrigation in the cloud.

The privacy side gets easier with the same split. A photo a farmer takes in their field is data that carries information about their own land. As long as diagnosis stays on the device you never have to explain where that photo goes; the user shares a correction if they want to, and if they do not, nothing leaves the phone.

Sağlam: there was nothing to debate here

With Sağlam the situation is sharper. After an earthquake, the first thing to go is the connection. If you tie crack detection, the evacuation map and the risk score to a server, the app dies in the middle of the very reason it exists.

So the question was not “on the device or in the cloud.” The question was: how much space on the phone am I entitled to spend? Because the same download holds both the crack detection model and the offline map data. Both take space and neither is negotiable.

The balance I landed on in the design: keep the model as small as possible — it has one job, it does not need to recognize a hundred classes — and download the map region by region. The user only downloads the province they live in instead of carrying the whole country. Because the model stays small, the fixed budget of the download can go to the map side.

LiteRT and Core ML: one model, two targets

The moment you move on-device you take on the responsibility of not one model but two deployment formats. LiteRT (formerly TensorFlow Lite) on Android, Core ML on iOS. Training happens in one place, then the same weights are converted into two separate formats.

In practice this has two consequences. First, the two outputs do not have to give a bit-for-bit identical answer to the same image; differences in quantization and hardware acceleration can produce small deviations. So I made running the same validation images through both platforms separately a part of the process — the assumption “it works on Android, so it works on iOS” does not hold here.

Second, on the Flutter side the model call does not stay in Dart. I planned to hand inference off to the native side over a platform channel and take the result back; the interface stays in Flutter, and the place the model runs is each platform's own runtime. That separation also makes it easier to keep inference off the UI thread — the screen must not freeze while the model is running.

Hardware acceleration is not a single path either. A model that runs accelerated on new devices can fall back to the CPU on an older one. So in the design I treated the absence of acceleration not as a failure but as a normal mode of operation: a slow but working answer is better than no answer at all.

The questions I ask when deciding

So that I do not argue this out from scratch every time, the list I ask myself has gotten short:

• Does the feature already depend on the internet? If it does, the gain from moving it to the device shrinks; if it cannot work without external data, doing the calculation on the server is cleaner.
• Where does the user use this feature? A field, a disaster area, the subway, a basement floor — if these are places without a connection, the answer is already obvious.
• Is the data sensitive? If photos, location or health data never leave the device, there is less consent copy to write and less responsibility to carry.
• Can the model be shrunk? A classifier with one job can shrink; a large general-purpose network cannot, and if you force it, it becomes useless.
• Are mistakes expensive? If they are, on-device inference alone is not enough; a confidence threshold and an “I am not sure” state are mandatory.

Both of these R&D designs are not at the code stage yet; they are architectural decisions I am still working on. But most of these are decisions that have to be made before a single line of code is written. Putting the model in the cloud and then saying “actually this should have worked offline too” means rewriting half the product. The easy path is not always the cheap path.

• Artificial Intelligence
• On-Device Inference
• LiteRT
• Core ML
• Quantization
• Offline