On-Device Inference with Core ML and Vision

Plenty of apps still ship a photo to a backend and wait for a classification response — a network dependency, added latency, and a copy of the user's image leaving the device for something the phone's own silicon can do.

#The old way

import Foundation

func classifyRemote(imageData: Data) async throws -> String {
    var request = URLRequest(url: URL(string: "https://api.example.com/classify")!)
    request.httpMethod = "POST"
    request.httpBody = imageData

    let (data, _) = try await URLSession.shared.data(for: request)
    let result = try JSONDecoder().decode(ClassificationResponse.self, from: data)
    return result.label
}

struct ClassificationResponse: Decodable {
    let label: String
}

#The new way

import Vision
import CoreML
import UIKit

func classify(image: UIImage) throws -> String {
    guard let cgImage = image.cgImage else {
        throw ClassificationError.invalidImage
    }

    let model = try VNCoreMLModel(for: MobileNetV2(configuration: MLModelConfiguration()).model)
    let request = VNCoreMLRequest(model: model)

    let handler = VNImageRequestHandler(cgImage: cgImage)
    try handler.perform([request])

    guard let results = request.results as? [VNClassificationObservation],
          let top = results.first else {
        throw ClassificationError.noResult
    }

    return top.identifier
}

enum ClassificationError: Error {
    case invalidImage
    case noResult
}

#Why it matters

• Inference runs on-device — no network dependency, no round-trip latency, and no image data leaving the device
• Vision handles image preprocessing (cropping, scaling, orientation) that the model expects, so pixel buffer conversion isn't hand-rolled
• Core ML automatically routes execution across the CPU, GPU, and Neural Engine for the best available performance on that device
• Works fully offline, which matters for any camera-driven feature that needs to respond instantly

#Gotcha

VNCoreMLRequest runs synchronously on whatever thread calls handler.perform(_:) — do it off the main thread (a background queue or a detached Task) or the UI stalls on every frame for anything camera-driven. A bundled .mlmodel file also adds real weight to the app's download size, and swapping in a bigger, more accurate model can meaningfully increase both binary size and per-inference latency on older devices — profile on the oldest device supported, not just a current-generation phone. Exact class and initializer names depend on the model compiled in, so verify against the generated interface for the actual .mlmodel in use.