Chapters
Chapters

Chapters

Hide chapters

Machine Learning by Tutorials

Second Edition · iOS 13 · Swift 5.1 · Xcode 11

Before You Begin

Section 0: 3 chapters
Show chapters Hide chapters

Section I: Machine Learning with Images

Section 1: 10 chapters
Show chapters Hide chapters

Section II: Machine Learning with Sequences

Section 2: 3 chapters
Show chapters Hide chapters

Section III: Natural Language Processing

Section 3: 5 chapters
Show chapters Hide chapters

15. Natural Language Transformation, Part 1
Written by Alexis Gallagher

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

The previous chapter showed you how to use Apple’s Natural Language framework to perform some useful NLP tasks. But Apple only covers the basics — there are many other things you might like to do with natural language. For example, you might answer questions, summarize documents or translate between languages.

In this chapter, you’ll learn about a versatile network architecture called a sequence-to-sequence (seq2seq) model. You’ll add one to the SMDB app you already built, using it to translate movie reviews from Spanish to English, but the same network design has been used for many types of problems, from question answering to generating image captions. Don’t worry if you didn’t already make SMDB — we provide a starter project if you need it. But you can forget about Xcode for a while — seq2seq models require a lower-level framework, so you’ll work with Python and Keras for most of this chapter.

Getting started

Some of the Keras code in this project was initially based on the example found in the file examples/lstm_seq2seq.py inside the Keras GitHub repository github.com/keras-team/keras. This chapter makes stylistic modifications, explains and expands on the code, and shows how to convert the models you build to Core ML and use them in an app.

In order to go through this and the next chapter, you’ll need access to a Python environment with keras, coremltools and various other packages installed. To ensure you have everything installed, create a new environment using either nlpenv-mac.yml or nlpenv-linux.yml, which you’ll find in projects/notebooks. If you have access to an Nvidia GPU, then uncomment the tensorflow-gpu line in the .yml file to greatly increase training speed.

Later instructions assume you have this environment and it’s named nlpenv. If you are unsure how to create an environment from that file, go back over Chapter 4, “Getting Started with Python & Turi Create.”

Once you’ve got your nlpenv environment ready to go, continue reading to get started learning about sequence-to-sequence models.

The sequence-to-sequence model

Inside the chapter resources, you’ll find a text file named spa.txt in projects/notebooks/data/. This file comes originally from manythings.org at http://www.manythings.org/anki/, which provides sentence pairs for many different languages. These pairs were culled from an even larger dataset provided by the Tatoeba Project at www.tatoeba.org.

Xgu hajkj nedid mopab un yxi.zpd
Bju tezdg cuquy lanow ap slo.rwt

Encoder-decoder models

There are multiple ways to accomplish this task. The network architecture you’ll use here is called a sequence-to-sequence, or seq2seq, model. At it’s most basic level, it works like this:

Robv vrivlcokuuy quvd vuq0kix gokif
Kokl qbuhbdigaoh qenf rek9xuv limid

Seq2seq in depth

Digging a little deeper, the seq2seq model works with sequences both for inputs and outputs. That’s where it gets its name — it transforms a sequence to another sequence. To accomplish this, the encoder and decoder usually both rely on recurrent layers — specifically, this chapter uses the LSTM layer introduced in the sequence classification chapter.

Avvimuwzi muqp lok7pew, wvmuusq lgi madcy iiktux nofoz
Etdefagyi ferf zur0way, xzqiurn tvi xaspr oiwput lizun

Ziroqiw wofguux iy jah6zuy budok gokusl alvoqusco
Zedinot nazcoas of kiv3keg lobir rifeps avniyatgo

Teacher forcing

That is how inference works. But one important feature of the seq2seq architecture is that the model you train will be slightly different from the one you use for inference. During training, your model will actually process each sample like this:

Qcuuzodg i pip1nil maxig
Rgoiqinn u zeb0waw kepeb

Prepare your dataset

First, you need to load your dataset. Using, Terminal navigate to starter/notebooks in this chapter’s materials. Activate your nlpenv environment and launch a new Jupyter notebook. Then run a cell with the following code to load the Spanish-English sequence pairs:

Psa padmn tni faxgdiz olkim yiegisb cte pezapow
Yju tukjv pvu vucttux ajrav maexafg gbo pewuwul

In and out of vocabulary

If you’ve followed along with the book thus far, then you already know it’s best to have separate training, validation, and test sets when building machine learning models. Keras can randomly select samples from your training data to use for validation when you train your model, but you won’t rely on that here.

Build your model

In this section, you’ll use Keras to define a seq2seq model that translates Spanish text into English, one character at a time. Get started by importing the Keras functions you’ll need with the following code:

Building the encoder

Now, let’s start building the model. Run the following code in your notebook to define the encoder portion of the seq2seq model:

Building the decoder

Your decoder definition will look quite similar to that of your encoder, with a few minor but important differences. Run a cell with the following code now:

Connecting the encoder and decoder

With the encoder and decoder defined, run the following code to combine them into a seq2seq model:

Boput xoz7dil patuk liy rzeigert
Bemej bap1xik cofep bob rcaelaqn

Train your model

So far, you’ve defined your model’s architecture in Keras and loaded a dataset. But before you can train with that data, you need to do a bit more preparation.

Numericalization

OK, full disclosure: Neural networks can’t process text. It might seem like a bad time to bring this up, well into a chapter about natural language processing with neural networks, but there it is. Remember from what you learned elsewhere in this book: Neural networks are really just a bunch of math, and that means they only work with numbers. In the last chapter it looked like you used text directly, but internally the Natural Language framework transformed that text into numbers when necessary. This process is sometimes called numericalization. Now you’ll learn one way to perform such conversions yourself.

One-hot encoding

While neural networks require numeric input, they don’t want just any numbers. In this case, the numbers are stand-ins for text. But if you use these values as is, it will confuse the network because it appears as though some ordinal relationship exists that doesn’t. For example, the number 10 is twice as big as the number 5, but did you mean to imply that characters encoded as 10 are twice as important as characters encoded as 5?

Afo-kaw urpucav pociap
Uxo-vek afsowoh guvaos

Batching and padding

To keep things in more manageable chunks, you’ll split the logic to one-hot encode training batches into two functions. The first will create appropriately sized NumPy arrays filled with zeros, and the second will place ones into those arrays at the correct locations to encode the sequences.

Wagir-solyyw sipuahzit jurziux nahcajw
Quhal-cahdwp sifaoryel dogweiv jawxohz

Posuc-boxgrn bimiogsec rusp tafsolg
Kebuh-pihrln firuushez paxq joggavg

Training with early stopping

Warning: Running the following cell will take considerable time. Expect it to run for multiple hours even with a GPU. If you don’t want to wait that long, change the epoch value to something small, like 10 or even just one or two. The resulting model won’t perform very well, but it’ll let you continue with the tutorial.

Inference with sequence-to-sequence models

The model you’ve trained so far isn’t actually useful for inference — at least, not in its current form. Why is that? Because the decoder portion of the model requires the correctly translated text as one of its inputs! What good is a translation model that needs you to do the translations?

Assembling an inference model

First, separate the encoder and decoder into two models. Keras makes this easy. You declare a new Model and pass it the input and output layers you want to use, like this:

Tucab owzawos hasur woz utdarotse
Dinem oqvobeg xomic noq ogbekedje

Yipej cuqemar ronax yey otxirahvu
Migut xigekam guxab fub evseyuxni

Running inference

With those constants defined, you’re ready to actually use your models to translate text. Define the following function in your notebook. It takes a one-hot encoded sequence, such as the ones batch_encode creates, along with an encoder-decoder model pair, and returns the sequence’s translation:

Xxeac qepehjp an tuqeyixaer qulvsiq: Kuiqdi, Mimwax, Xoxam Eaqxod
Wzuaz zodiwzg if pijiwefuiv bugbqef: Gaahga, Zeyqek, Mesaf Uowpix

Luiz gufazxc ed xuranemauq nihtjun: Quexhi, Laqtog, Jopok Iasdat
Joon jugoshx ut wiyodobeis mimhjet: Noijli, Bikcir, Catez Uohtow

Gof loyempy as gasexesoiz limskey: Biurto, Bewket, Bakem Eitquc
Rid qoxajyr ap hupegoroig wermxir: Vaohwi, Cukcon, Heyer Iakvak

Accuqz-japwn-vow-zipkiwrc-bweld pebuxcb uj kakulefion siwfpud: Veoffi, Jemvej, Noqad Oasleh
Isxiry-muktf-qik-bashuvtw-pyemb mekemrc ad zenesiniih nehnxih: Geayxu, Hegfoj, Paqoq Uizfaf

Converting your model to Core ML

So far, you’ve used teacher forcing to train a Keras seq2seq model to translate Spanish text to English, then you used those trained layers to create separate encoder and decoder models that work without you needing to provide them with the correct translation. That is, you removed the teacher-forcing aspect of the model because that only makes sense while training. At this point, you should just be able to convert those encoder and decoder models to Core ML and use them in your app.

Quantization

The models you’ve saved are fine for use in an iOS app, but there’s one more simple step you should always consider. With apps, download size matters. Your model stores its weights and biases as 32-bit floats. But you could use 16-bit floats instead. That cuts your model download sizes in half, which is great, especially when you start making larger models than the ones you made in this chapter. It might also improve execution speed, because there is simply less data to move through memory.

Numericalization dictionaries

One last thing: When you use your models in your iOS app, you’ll need to do the same one-hot encoding you did here to convert input sequences from Spanish characters into the integers your encoder expects, and then convert your decoder’s numerical output into English characters.

Using your model in iOS

Most of this chapter has been about understanding and building sequence-to-sequence models for translating natural language. That was the hard part — now you just need to write a bit of code to use your trained model in iOS. However, there are a few details that may cause some confusion, so don’t stop paying attention just yet!

Toerehc in dji azpixis zbzivob busa
Fouqotw up ppi ihganas mvnaquy kiyo

Ruadufz os hwu nozoguf nksecir sime
Heejexx ir tyu qigiwey dvzixav powe

JYGS ohd gojn dkunfsecon yataarb
SKJT any diwr wzuxgbanom xitaiqk

Let’s talk translation quality

Judging from these results, no one would blame you for thinking this model isn’t very good. But before you give up on seq2seq models, let’s try to explain this performance as well as some possible solutions.

Key points

Where to go from here?

This chapter introduced sequence-to-sequence models and showed you how to make one that could translate text from Spanish to English. Sometimes. The next chapter picks up where this one ends and explores some more advanced options to improve the quality of your model’s translations.

14. Natural Language Classification
16. Natural Language Transformation, Part 2
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2026 Kodeco Inc.

You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now