Chapters
Chapters

Chapters

Hide chapters

Machine Learning by Tutorials

Second Edition · iOS 13 · Swift 5.1 · Xcode 11

Before You Begin

Section 0: 3 chapters
Show chapters Hide chapters

Section I: Machine Learning with Images

Section 1: 10 chapters
Show chapters Hide chapters

Section II: Machine Learning with Sequences

Section 2: 3 chapters
Show chapters Hide chapters

Section III: Natural Language Processing

Section 3: 5 chapters
Show chapters Hide chapters

10. YOLO & Semantic Segmentation
Written by Matthijs Hollemans

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

You’ve seen how easy it was to add a bounding box predictor to the model: simply add a new output layer that predicts four numbers. But it was also pretty limited — this model only predicts the location for a single object. It doesn’t work so well when there are multiple objects of interest in the image.

You might think that you could just add more of these output layers, or perhaps predict 8 numbers for two bounding boxes, or 12 for three bounding boxes, etc. Good try, but unfortunately that doesn’t work so well in practice.

Each bounding box predictor will end up learning the same thing and, as a result, makes the same predictions. Instead of finding the locations of multiple objects, such a model will predict the same bounding box multiple times. And chances are, these bounding boxes will not actually enclose any of the objects but all end up somewhere in the middle of the image as a compromise.

To make a proper object detector, you need to encourage the different bounding box predictors to learn different things.

An old-school approach to object detection is to divide up the input image into many smaller, partially overlapping regions of different sizes, and then run a regular image classifier on each of these regions. This definitely works, but it gives a lot of duplicate detections. Even worse: It’s really slow. You need to run the classifier many, many, many times for each image.

A slightly smarter approach is to first try and figure out which parts of the image are potential regions of interest. This is the approach taken by the popular R-CNN family of models. The classifier is still run on multiple image regions, but now only on regions that are at least somewhat likely to have an object in them.

To predict which regions are potentially interesting, the “Faster R-CNN” model uses a Region Proposal Network, which sounds impressive but is really just a bunch of layers on top of the feature extractor — hey, what did you expect? Unfortunately, even though it has “Faster” in its name, this model is still on the slow side and not really suitable for mobile devices.

For speed freaks and mobile device users, the so-called single stage detectors are very appealing. As the name implies, these model types just run the classifier once on the input image and do all of the work in a single pass. Examples of single-stage object detectors are YOLO (You Only Look Once), SSD (Single Shot multi-box Detector) and DetectNet.

Turi Create lets you train a YOLO model with just a few lines of code, so that’s what you’ll do next.

Single stage detectors

The simplest form of a single stage detector, and the one you’ll be training, looks like this:

Again, there’s a feature extractor plus a few layers on top. The YOLO feature extractor is called Darknet, and it’s not so different from the feature extractors you’ve seen before: Darknet consists of convolution layers, followed by batch normalization and the ReLU activation function, with pooling layers in between.

Note: The activation function used by Darknet is actually a variation of ReLU, known as leaky ReLU. Where a regular ReLU completely removes any values that are less than zero, the leaky version makes negative values a lot smaller but still lets them “leak through.”

The extra layers are all convolutional. Unlike before, where the output of the model was either a vector containing a probability distribution or the coordinates for the bounding box, the output of YOLO is a three-dimensional tensor of size 13 × 13 × 375 that we’ll refer to as the grid.

YOLO takes a 416×416 pixel image as input. That’s larger than what you typically use for classification. This way, small details don’t get lost. There are five pooling layers in Darknet that each halve the spatial dimensions of the image, for a total reduction factor of 32. Since 416/32 = 13, the final grid is 13×13 pixels.

Looking at this the other way around, each of the cells in this grid refers to a 32×32 block of pixels in the original image. Each cell is therefore responsible for detecting objects in or around that particular 32×32 region of the input image.

Each cell in the grid is responsible for its own region in the original image
Each cell in the grid is responsible for its own region in the original image

YOLO, therefore, has 13×13 = 169 different bounding box predictors, and each of these is assigned to look only at a specific location in the image. Actually, this isn’t entirely true: Each grid cell has not just one but 15 different predictors, for a total of 169×15 = 2,535 bounding box predictors across the entire image. That’s quite an upgrade over the simple model you made previously!

Having multiple predictors per grid cell means you can let bounding box predictors specialize in different shapes and sizes of objects. Each cell will have a predictor that looks for small objects, a different predictor that looks for large objects, one that looks for wide but flat objects, one that looks for narrow but tall objects, and so on.

This is where the number 375 comes from, the depth dimension of the output grid: Each grid cell has 15 predictors that each output 25 numbers. Why 25? This is made up of the probability distribution over our snack classes, so that’s 20 numbers. It also includes four numbers for the bounding box coordinates. Finally, YOLO also predicts a confidence score for the bounding box: how likely it thinks this bounding box actually contains an object. So there are two confidences being predicted here: one for the class, and one for the bounding box.

Because the output of YOLO is a 13×13×375 tensor, it’s important to realize it always predicts 2,535 bounding boxes for every image you give it. Even if the image doesn’t contain any recognizable objects at all, YOLO still outputs 2,535 bounding boxes — whether you want them or not.

That’s why the confidence score is important: It tells you which boxes you can ignore. In an image with no or just a few objects, the vast majority of predicted boxes will have low confidence scores. So at least YOLO is kind enough to tell you which of these 2,535 predictions are rubbish.

Even after you filter out all the boxes with low confidence scores — for example, anything with a score less than 0.25 — you’ll still end up with too many predictions. This kind of situation is typical:

I’m only counting one dog and cat, not three!
I’m only counting one dog and cat, not three!

These are all bounding boxes that the model feels good about since they have high scores, but as a consumer of an object detection model, you really want to have only a single bounding box for each object in the image. This sort of thing happens because nearby cells may all make a prediction for the same object — especially when the object is larger than 32×32 pixels.

To filter out these overlapping predictions, a post-processing technique called non-maximum suppression or NMS is used to remove such duplicates. The NMS algorithm keeps the predictions with the highest confidence scores and removes any other boxes that overlap the ones with higher scores by more than a certain threshold, say an IOU of 45% or more. The model created by Turi Create automatically takes care of this post-processing step for you, so you don’t have to worry about any of this.

Note: Turi’s object detection model is known as TinyYOLO because it’s smaller than the full YOLO. The full version of YOLO has multiple output grids of varying dimensions in order to handle different object sizes better, but this model is also larger and slower. Another popular single-stage detector is SSD. Architecturally, YOLO and SSD are very similar in design and differ only in the details. SSD does not have its own feature extractor and can be used with many different convnets. Particularly suitable for use on mobile is the combination of SSD and MobileNet.

Hello Turi, my old friend

Switch to the turienv Python environment and create a new Jupyter notebook. You can find the environment in the starter project of this chapter’s materials. Refer back to Chapter 4: Getting Started with Python & Turi Create if you don’t remember how to activate environments.

Rka TPvajo her kobgauhk dwu ezbivivuerg zetxainodeib
Qge RMnimo qiy xipnoekt vbu imdupuruexk nastoahimeeb

Jueduzz pwa mcuelt-kvotn quxut ay jxo hmeabesk aqabuv
Lootojy lxo kgoudy-sjonx favug im jki pnuanogq ojuzuf

Training the model

It just takes a single line of code and a whole lot of patience:

How good is it?

In case you don’t have the hardware or the time to train this model yourself, we’ve included the trained model in the downloads as a .zip file in the final folder, SnackDetector.model.zip. Unzip this model to your working directory and then load it into the notebook:

Niarecp rfu zvidapmek poemdiwf dafuv
Yiinapw mjo wwaciwxuf daimmums zeqom

The demo app

This is a book about machine learning on iOS, and it’s been a while since we’ve seen the inside of Xcode, so let’s put the trained YOLO model into an app. The book downloads contain a demo app named ObjectDetection.

Qqe VEVE jakuw ir Kiju QT
Ypo JUZU homey iw Sage VR

Pko XUYE belah op urlioz uh kvo aXnuge
Tlo TIDO jinuv os eqtien ug zho uPlede

Semantic segmentation

You’ve seen how to do classification of the image as a whole, as well as classification of the contents of bounding boxes, but it’s also possible to make a separate classification prediction for each individual pixel in the image. This is called semantic segmentation. Here’s what it looks like:

Nonohsuc qagponzudeeb judik u fzufc byabegwuew zev ajoyz lazex
Zatutdin qekcukqebout hivis a plamk hgugosteom qaz osucj xinez

CiuyWab uw til ox XeluhaFahK6
QaitMup ez lut es VuyuyaNezQ8

Converting the model

You’re going to be using a pre-trained version of DeepLab that is made freely available as part of the TensorFlow Models repository, at: github.com/tensorflow/models/tree/master/research/deeplab.

Mebz ul lja rmazav ehjoganye hzibk uf Dalbav
Rirf am cxo jmewuv iwlamonye gciqh ic Zebbes

Zxe biwpajq huju fis SearNot.rxgiquc
Wwo dorpucr fexu mow CiesVuk.ybyakek

The demo app

The downloads for this chapter include an app called Segmentation. The code is very similar to that of the HealthySnacks app from a few chapters ago, except now there are two pairs of camera/photo library buttons, allowing you to select a background image and a foreground image.

Vza uichuc ed wekg-ruquvlug lato fobosas (wejz), bke sushoxnefyall pogwaqyisuuj jiqy (sifsc)
Qfi iodzim ur kuhs-yifohqox zuge mivurin (fibw), dga vuqdezzezzotb somcuxdiweut nukw (zozck)

Challenges

Challenge 1: Create a dataset for object detection

If you collected your own classification dataset for one of the previous challenges, then use a tool such as RectLabel to add bounding box annotations for these images. RectLabel uses a different file format to store the annotations, but rectlabel.com has code examples that show how to use these files with Turi Create.

Challenge 2: Train MobileNet+SSD on the snacks dataset

The size of the YOLO model you trained with Turi Create is 64.6 MB. That’s pretty hefty! This is reaching the upper limit of what is acceptable on mobile devices. It’s possible to use object detection models that are much smaller than YOLO that give very good results, such as SSD on top of MobileNet (about 26 MB).

Challenge 3: Change the semantic segmentation demo app

Change the semantic segmentation demo app to only keep pixels that belong to cats and dogs — or whatever your favorites are from the 20 Pascal VOC classes.

Key points

Where to go from here?

Congrats, you’ve reached the end of section 1, Machine Learning with Images! Of course, we hope this is really only the beginning of your journey into the wonderful world of computer vision and deep learning.

9. Beyond Classification
11. Data Collection for Sequence Classification
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2026 Kodeco Inc.

You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now