Chapters
Chapters

Chapters

Hide chapters

Practical Android AI

First Edition · Android 13 · Kotlin 2.0 · Android Studio Otter

Before You Begin

Section 0: 5 chapters
Show chapters Hide chapters

Section I: Foundations of AI on Android

Section 1: 3 chapters
Show chapters Hide chapters

Section II: Building Core Intelligence

Section 2: 3 chapters
Show chapters Hide chapters

Section III: Advanced Integration, Distribution, and Responsible AI

Section 3: 4 chapters
Show chapters Hide chapters

4. On-device Intelligence with ML Kit
Written by Zahidur Rahman Faisal

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Modern mobile apps deliver intelligent, personalized, and responsive experiences. For years, this intelligence was powered by large-scale cloud servers. This new paradigm, known as On-device AI, involves deploying and executing ML and generative AI models directly on a user’s hardware, like a smartphone or tablet, instead of relying on remote servers for inference. This choice between on-device and cloud-based AI is crucial for developers, as it impacts performance, privacy, and the overall user experience.

ML Kit, a mobile SDK, brings Google’s on-device machine learning expertise to Android apps. With the powerful yet easy-to-use Generative AI (GenAI), Vision, and Natural Language APIs, you can solve common challenges in your apps or create entirely new user experiences.

In this chapter, you’ll harness the power of ML Kit and create a sample app that will:

  • Scan documents and save them as images or PDFs.
  • Extract text from the saved documents and share it online.

Let’s get started with ML Kit!

ML Kit on Android

ML Kit is an easy-to-use SDK that brings Google’s extensive machine learning expertise to mobile developers, abstracting away the complexities of model management and inference. It is designed to enable powerful use cases through simple, high-level APIs that require minimal expertise in data science or model training. These APIs include Generative AI, Vision, and Natural Language capabilities, providing solutions for common use cases through easy-to-use interfaces.

ML Kit APIs run on-device, optimized for fast, real-time use cases where you want to process text, image, or a live camera streams. The ML Kit APIs are categorized as follows, based on their ML models:

GenAI APIs

  • Text Summarization: Concisely summarize articles or chat conversations into a bulleted or concise list.
  • Proofreading: Polish short content by refining grammar and correcting spelling errors.
  • Rewriting: Rephrase short messages in various tones or styles.
  • Image Description: Generate a short description for a given image.

Vision APIs

  • Text Recognition: Recognize text from images, and fragment it into blocks, lines, elements and symbols.
  • Digital Ink Recognition: Recognize hand‑drawn shapes and emojis.
  • Face Detection: Detect faces in an image, identify key facial features, and extract contours of detected faces.
  • Pose Detection: Detect the pose of a subject’s body in real time from a continuous video stream or static image.
  • Object Detection and Tracking: Detect and track objects in an image or a live camera feed.
  • Subject Segmentation: Detect and separate multiple subjects from the background of a picture.
  • Image Labeling: Detect and extract information about objects and entities in an image.
  • Barcode Scanning: Read standard barcode formats without requiring an internet connection.
  • Document Scanner: Convert physical documents into digital formats (e.g., images or PDFs).

Natural Language APIs

  • Entity Extraction: Identify specific entities within static text and trigger context-based actions for the user depending on the entity type.
  • Language Identification: Determine the language of a given piece or string of text.
  • Translation: Dynamically translate text between more than 50 languages, even when the device is offline.
  • Smart Reply: Automatically generate contextually relevant and concise replies to messages.

Creating a Document Scanner using ML Kit

You’ll learn how easily you can create a custom Document Scanner using MLKit! You can rely on the Document Scanner module from MLKit. The Document Scanner APIs can provide below capabilities to your app:

Adding Dependency

Preparing the Scanner

You’re now ready to use the dependency and its helper classes. Before you can start scanning, you need to configure the Document Scanner client. To do so, open MainViewModel.kt and update the prepareScanner() function as follows:

Creating the Scanner Launcher

Next, open MainActivity and add the following code snippet above the onCreate() method:

Handling Result

Go back to MainViewModel.kt again. You’ll see there’s a MutableStateList named pageUris defined at the top – this list will contain the URI of each page from the scan result, which will be used later.

Scanning Documents

You’re all set to launch the Document Scanner at this point and use the resultant data. You need to update the launchDocumentScanner() function in MainActivity.kt as shown below:

Qhomkukc Zucojarmw
Bmudfiys Dukucuvhz

Extracting Text using Text Recognizer

MLKit made it easy to turn your app into a Document Scanner, but what if you also want to extract text (OCR) from your scanned documents? This part of the chapter will teach you how to do exactly that!

Recognizing Text from Image

Remember saving your scanned pages as images? That’ll come in handy now. The MainViewModel keeps the reference of those image URIs in the pageUris variable. This list is used to display a carousel of your scanned pages in MainActivity.kt, which looks like this:

Ilewi Zikaofec
Oyeli Nigoipud

Cumop isdun qocey muq uquc, tumguqjejaf onijafnazn otab, kiq la aouwyit hufxow imvoyewevz uw gosuhe od bufayi fimme avubai. Eq obiq op dakov nowiog, guuj tigydad ususfevapoah acmefmo zupumig dodu az ezejuef as ue sojxewu nocmeyois. Peed iewu ixafi bavud eh vaqfoluhkakil on mabafmami jurat ayle copbet kejuna ou lesais gemne gihuowaj. Obhotbooc fotg ophielub lubijigej wan phiiyakc, xasy ud qubnu neo orbaria puliyupd hawzeb ohob ah ely fifuzuc. Rxutn Zake Ceqi Isemukm Ahugitt Ukusepr Afizigv
Kiqx Tbgumyoye

Handling Result

At this point, viewModel.getTextFromImage(uri) will return the extracted text from the image. You might want to use or share this text within your app. To do that, open MainActivity.kt and add the following function:

Hfenaqg Cavq
Jgofech Deld

The Trade-offs of On-device AI

On-device AI is optimized for scenarios where data processing must be immediate, private, and available without a network connection, but it comes with some strategic trade-offs.

The Benefits

These are the key benefits of using on-device AI:

Privacy and Security

With on-device processing, sensitive personal data, such as images, voice recordings, or private messages, never needs to leave the user’s device. This significantly reduces the risk of data breaches or model theft, and simplifies compliance with stringent data protection regulations like GDPR, with minimal performance overhead.

Latency

By performing inference locally, on-device AI eliminates network round-trip delays, resulting in near-instantaneous responsiveness. This is essential for real-time applications such as augmented reality (AR) filters, live camera analysis, and voice assistants that must respond without perceptible lag.

Offline Functionality

On-device models enable offline functionality, allowing applications to remain fully operational in environments with poor or nonexistent connectivity, which is a critical consideration for a global user base.

Operational Costs

For developers, on-device inference reduces ongoing server and bandwidth expenses associated with repeated cloud API calls. Running tasks locally is also more energy-efficient, consuming up to 90% less energy than cloud-based inference.

The Limitations

Despite these benefits, on-device AI is not without its challenges. The primary limitations are:

Computational Constraint

Even though modern mobile device hardware is becoming increasingly powerful, it cannot match the scale of a cloud data center. This limits the size and complexity of models that can run efficiently on a device.

Model Management

Managing models becomes more complex with on-device AI. While a cloud model can be updated instantly for all users, on-device models must be packaged with the application and distributed through app updates, making the process more time-consuming and logistically challenging.

Battery Consumption

Even optimized on-device inference can contribute to increased battery usage, particularly for computationally intensive tasks. Developers should focus on optimizing background tasks, limiting unnecessary requests, and using power-efficient APIs to minimize battery drain.

App Size

While using on-device models, you as a developer, must also consider broader app performance. Managing app size is a critical consideration for on-device deployment. Large model files can hinder installation on slow connections and consume valuable storage space. Best practices include using Android App Bundles, which dynamically deliver only the necessary code and resources to a user’s device, and leveraging tools like the Android Size Analyzer to identify areas for size reduction.

Conclusion

A comprehensive analysis of these trade-offs reveals that the architectural decision for AI-powered features is rarely a simple, binary decision. The most robust solutions are often hybrid models that combine the strengths of both approaches. A common design pattern involves using on-device AI for basic data preprocessing and low-latency tasks, such as initial object detection in a live camera feed, while reserving more complex, high-volume analysis for cloud-based services. This enables a fluid user experience while leveraging cloud power when necessary.

3. Getting Started with Android Generative AI
5. Custom ML with MediaPipe
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2026 Kodeco Inc.

You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now