In Chapter Two, you worked through examples demonstrating some weaknesses of LLMs and Apple Foundation Models. One important limitation of any LLM is that it only contains information from its training data. It has no information about events after the training or information not included in the training. If you were to ask the earlier Foundation Explorer app something as simple as the current time, it would tell you that’s not possible.
Unable to provide real-time information
This lack of knowledge provides a severe limitation to many things you might want to produce. For many apps, it would be helpful to access other data from the iPhone, such as contacts, calendar events, and health data. There are also cases where you would like to access external data from the web.
In some cases, you can collect the data and pass it to Foundation Models via instructions or a prompt. That requires you to anticipate the information and provide it in the best format. Apple Foundation Models provides a way to accomplish more through tool calling. Tools enable the model to plan and gather information to help it respond to the prompt. This action occurs autonomously and can access anything returned from a function call. You define a tool by implementing the Tool protocol in your struct or class. Then you include information on available tools when creating the LanguageModelSession.
Open and run the starter app for this chapter. You’ll see the start of a project to help the user pack for a trip occurring in the next few days. The most important information the model doesn’t have is the weather forecast for the travel destination. In this chapter, you’ll use Foundation Models to provide this packing advice and use Tools to provide this information to the model.
The starter app for an app to help the user pack for a trip.
There are many ways to get weather forecast information, including Apple’s own WeatherKit. For this tutorial, you’ll use the United States National Weather Service to get forecast information. This service only works in the United States, but it is free and does not require an API key or any other sign-up. It requests that the user provide an agent name along with a valid email address for contacting the user in case of issues. Open NWSWeatherService.swift under the Services folder. Find the // Change Below to Your Email comment in the file and change email@example.com on the next line to your email address. Repeat the process with NWSWeatherService2.swift in the same folder.
The NWSWeatherService struct implements the service for retrieving the weather forecast via a web API. If you read through, you will notice that the API requires a latitude and longitude when requesting a forecast. You could ask the user to provide this information, but it would be better to convert their entered city name for them. Even better, you’ll write this as a Tool so the model can perform the city-to-coordinates conversion when needed.
There are a number of ways to convert a city name into a latitude and longitude, a process known as geocoding. Apple provides one as part of MapKit, and you’ll use that in this chapter. Create a new Swift file under the Tools folder named GeoLookupTool.swift. Replace the contents of the file with:
This code states that your GeoLookupTool will implement the Tool protocol. You must meet several requirements to implement this protocol. You must define a call(arguments:) method that accepts arguments of type ConvertibleFromGeneratedContent and returns a type conforming to PromptRepresentable. In practice, your return will be a String or a Generable object.
Add the following two properties to the start of the GeoLookupTool struct:
let name = "GeolocationTool"
let description = "This service returns the latitude and longitude for a city or location."
These two properties set the tool’s name and describe the tool’s purpose. The description provides context for the tool to the model. Try to keep descriptions short, as they become part of the context and can introduce latency. Now you’ll need to provide the arguments the tool expects. Add the following code after the description:
@Generable
struct Arguments {
@Guide(description: "The name of the city to get the latitude and longitude for.")
var location: String
}
Note that the argument uses the Generable macro. Everything you’ve already learned about guided generation in Chapter Five applies here. The only argument for this tool expects a location name as a String. While this example only has one argument, you can provide as many as your tool needs. It also uses the @Guide macro and description to better inform Foundation Models of this argument’s purpose and its relationship to the tool’s use.
Now you need to add data structures to contain the information returned by the tool, along with error handling. Add the following code to the top of the file after the import statements:
enum GeocodingError: Error {
case invalidRequest
case noMatchingLocation
}
struct GeocodedLocation {
let name: String
let latitude: Double
let longitude: Double
}
The GeocodingError struct will contain information on any errors that might occur. The GeocodedLocation struct will contain the location name and the location’s latitude and longitude when the service succeeds. The final step in implementing the Tool protocol is to define the call(arguments:) method. Enter the following code after the Arguments struct:
The method begins by stating it is asynchronous and throws. A throwing method will propagate errors to the caller rather than handle them internally. In this case, all errors pass back to Foundation Models. You’ll look more into error handling later in the chapter.
You get the values from the Arguments struct, in this case, taking the location passed in by Foundation Models and storing it in a placeName variable. Any arguments sent to the tool, Foundation Models provides when it calls the tool. The method returns the GeocodedLocation struct.
Now add the following code to the end of the method to perform the geocoding:
// 1
guard let request = MKGeocodingRequest(addressString: placeName) else {
throw GeocodingError.invalidRequest
}
// 2
let mapItems = try await request.mapItems
guard let item = mapItems.first else {
throw GeocodingError.noMatchingLocation
}
// 3
let coordinate = item.location.coordinate
let name =
item.name ??
item.address?.shortAddress ??
item.address?.fullAddress ??
placeName
// 4
return GeocodedLocation(
name: name,
latitude: coordinate.latitude,
longitude: coordinate.longitude
)
This code works by:
The MKGeocodingRequest class in MapKit facilitates looking up a geographic coordinate for a provided string. If creating the object fails, you return the GeocodingError.invalidRequest defined earlier in the file.
The request.mapItems call gets the array of MKMapItem objects that represent points of interest on the map that correspond to the location string passed during object instantiation. You then attempt to unwrap the first item in the array, and if that fails, return the GeocodingError.noMatchingLocation error.
The item.location.coordinate holds the coordinates of the location that you need to get the weather forecast. The MKMapItem object contains info on the point of interest. To get an official name, you first attempt to use the name property, then fall back to the address information, and finally to the string passed into the method if all else fails. This normalizes the name to ensure the coordinates and name match.
Finally, you return this information to Foundation models in a new GeocodedLocation structure.
If you attempt to build the app now, you will get a confusing error message. The error occurs because the return value of call(arguments:) must conform to PromptRepresentable. You can address this by making it a String, but for most return values, you’ll need to ensure the returned value conforms to the Generable protocol. Recall that the simple Bool, Int, Float, Double, Decimal, and Array types already implement this protocol. For other types, such as Date or custom data structures like GeocodedLocation, you must implement the protocol on the returned structure. Change the definition of GeocodedLocation to:
@Generable(description: "Contains a location name and the latitude and longitude for that location.")
struct GeocodedLocation {
@Guide(description: "Location Name")
let name: String
@Guide(description: "Latitude of Location")
let latitude: Double
@Guide(description: "Longitude of Location")
let longitude: Double
}
You do most of the work of implementing the Generable protocol by adding the @Generable macro before the class or struct. You also provide the description parameter, which provides Foundation Models with information about the structure’s purpose. You also use the @Guide macro to provide information on the meaning of the structure’s properties to Foundation Models. In some cases, the property names alone will suffice if they provide enough clarity, but that will require testing.
To see your new tool in action, open HelpMePackView.swift and add the following new method at the end of the packingSection subview:
func createNewSession() {
// 1
let instructions = """
You are a tool to help users determine the latitude and longitude of locations provided to you.
Use available tools to convert locations and city names into
latitude and longitude
"""
//2
session = LanguageModelSession(
tools: [GeoLookupTool(),],
instructions: instructions
)
}
Here’s how this code sets up a session for tool use:
In earlier chapters, you’ve learned that instructions let you provide a prompt that guides the entire session. This prompt gives the model a purpose and tells it to use the tool you provided to find the coordinates for locations.
The new element here comes in the tools parameter. This parameter contains an array of tools available to the model. You pass in the GeoLookupTool you built in this section to the session.
Now it’s time to actually put the tool to use. Add the following new method after createNewSession():
func generatePackingList() {
// 1
let prompt = "Determine the latitude and longitude for \(information.destination). Do not assume anything about the location from general knowledge."
// 2
Task {
// 3
isLoading = true
defer {
isLoading = false
}
// 4
let stream = session.streamResponse(to: prompt)
// 5
do {
for try await partialResponse in stream {
information.packingRecommendation = partialResponse.content
}
// 6
} catch {
information.packingRecommendation = "Error: \(error.localizedDescription)"
}
}
}
The prompt is the most important new aspect of the app. As before, a good prompt requires time and testing to produce. This one provides a purpose for the prompt and includes the user-provided destination that the model should find a location for. It reinforces the use of the tool and not to rely on internal information.
The rest of the method should feel familiar from the early chapters. You start by creating a Task to wrap the asynchronous methods to come.
You repeat the pattern to set the isLoading property to provide visual feedback when the app is generating a response. The defer will ensure this is reset when the method ends, regardless of which method ends it.
You create an asynchronous streamed response using the streamResponse(to:options:) method.
The do loop will iterate over the stream as Foundation Models returns it, so you can provide faster feedback to the user. As you are returning a String, you simply set the Text to the latest response each time.
If anything goes wrong, for now, you’ll set the text to the error description.
To call these methods, find the // Add Button Action comment in the tripSection subview and replace it with:
createNewSession()
generatePackingList()
This will create a new session each time the user taps Generate Packing List. It will then call the generatePackingList() method to do the lookup. Run the app and enter a major United States city. After a pause, you will see the coordinates of that city.
Geocoding for Raleigh, which is interpreted as the one in North Carolina.
To see how the tool integrated into the model, tap the page icon on the toolbar in the top-trailing corner. You will see that this simple prompt-and-response used a few hundred tokens. The transcript always begins with the instructions. The prompt to the session comes next. You provide these with the createNewSession() and generatePackingList() methods you created, respectively. You will then see the Tool call in the transcript, which shows Raleigh passed into the tool. The transcript then includes the information returned by the tool, which includes the coordinates. Finally, you see the response containing the coordinates.
The session transcript for coordinate lookup.
Tool calls and outputs become part of the session’s context. This adds size to the already tight context length for Foundation Models.
Now that you see how to create a working tool, you need to add a second tool to look up the weather for the coordinates that this tool provides. You’ll do that in the next section.
Creating a Weather Forecast Tool
In the last section, you adapted an existing data structure to the Generable protocol. This was simple since all three properties of the GeocodedLocation struct were simple Swift types that already supported the protocol. Open NWSWeatherService.swift under the Services folder and look for the two structs that this service uses: WeatherSummary and WeatherPeriod. The WeatherSummary struct contains a String and an Array, both of which implement Generable. However, the WeatherPeriod struct contains two properties of type Date, which does not support the Generable protocol. You will need to provide a data structure that bridges the gap between the service’s data structure and the data structure required by Foundation Models.
Create a new Swift file called WeatherForecastTool.swift under the Tools folder. Replace the contents of the file with:
import FoundationModels
@Generable(description: "Weather Information for a location.")
struct WeatherInformation {
let locationName: String
let forecasts: [WeatherForecast]
}
This new struct matches the properties of WeatherSummary provided by the service, but uses the @Generable macro to implement the Generable protocol on it and provide a description. Notice you do not add a @Guide to either property since their names clearly explain their purpose to the model. Now add a replacement for WeatherPeriod better suited for Foundation Models:
@Generable(description: "Weather Forecast for a period.")
struct WeatherForecast {
let name: String
let startTime: String
let endTime: String
let temperature: Int
let temperatureUnit: String
let shortForecast: String
let detailedForecast: String
let windSpeed: String
init(fromPeriod: WeatherPeriod) {
self.name = fromPeriod.name
self.startTime = fromPeriod.startTime.formatted(date: .numeric, time: .shortened)
self.endTime = fromPeriod.endTime.formatted(date: .numeric, time: .shortened)
self.temperature = fromPeriod.temperature
self.temperatureUnit = fromPeriod.temperatureUnit
self.shortForecast = fromPeriod.shortForecast
self.detailedForecast = fromPeriod.detailedForecast
self.windSpeed = fromPeriod.windSpeed
}
}
Despite its length, this new WeatherForecast struct contains only two changes to the properties in the WeatherPeriod struct used by the service. It changes the start and end times for the forecast to Strings. You use the formatted(date:time:) instance method to convert the date into a localized string in the format 1/17/2021 4:03 PM. You also provide a convenience initializer that takes a WeatherPeriod struct, making it easier to convert the WeatherPeriod returned by the struct into a WeatherForecast used by the tool. You again let the property names provide the context.
With the data structure defined, continue with the following code to create the Tool.
struct WeatherForecastTool: Tool {
let name: String = "WeatherForecastTool"
let description = "This service returns the weather forecast for the next seven days for a given latitude and longitude."
}
This should look familiar, as the base attributes of implementing the Tool protocol are the same. Now add the following code to define the arguments to this tool at the end of the struct:
@Generable
struct Arguments {
var latitude: Double
var longitude: Double
}
This time, your tool expects a latitude and longitude of the location to get the weather forecast. As before, the Arguments must be Generable. Since Double has support for the Generable protocol by default, you don’t need to define anything for them. To finish the Tool, implement the call(arguments:) method by adding this code to the end of the struct:
You first create an NWSWeatherService object to get the forecast.
Now you call the service using the latitude and longitude passed in through Arguments. You then assign the result to forecast.
Next, you create an object of the WeatherInformation struct built to support the Generable protocol. The locationName property copies directly from the property of the same name inside WeatherSummary. To fill the forecasts array, you use the map(_:) method on the original array in the WeatherInformation struct. This method applies the closure to each element of the original array and returns a new array containing the results. You use that convenience initializer to convert the WeatherPeriod to a WeatherForecast, resulting in an array of WeatherForecast elements. You then return this new WeatherInformation struct.
Now that you’ve implemented this new Tool, you are ready to wire both tools into Foundation Models in the next section to help a user pack for their trip.
Integrating Tools into Foundation Models
Finally, it’s time to put all this work to use. Open HelpMePackView.swift and change the createNewSession() method to:
zuvg gpuuveMiqXohjoub() {
fey elnyqeyfoikd = """
Fou ola i poftoht axnulpevq pbar yxeaqap ymazsefoy yikhahv pihng rah msagetusq.
Ebe bva ojeolilgo buarx pduvulud divdizp miozcud acnajbedeas aj xoohir.
Acha uwi ajeujazco kioqj je vislamc jopiquekm uzp sapz rocuj ejro
qijajexo oyc ditlomuba
Sa kir muobp niamduf xalgodueky, dulrazelaqow, ey rvemaruvopian.
Nrok kniuhigk e sizvahm quwq:
- Dickm vixakboce qra jfup zojnociyoac app zemod.
- Ufo qoixv ta jih bye repuxejl pup lko wonxobariik uch cjemiv wicob.
- Lika neobqoz-qiruciz yenibsubyikiolx ol haaj yojansv.
- Xikemzizm uvhm odihz tyej ijo okizay vay jci bmup motbowoodv.
- Kaof yla gogj liwtezo, hiusemxoy, arj spaelut st ziyetobm.
- Ombbaiw hvoingv yck poenpaf-hhatajug ahicr uwe etkhifer.
"""
keglaut = KolwaivuFerudPuzceak(
zeicm: [TuiKeuxewSeek(), ZoacfinComezadvXoih()],
agrfxakyiiph: ojzykazweifz
)
}
Cci yun hbacqn kgecuvup u dibcumi enn untbitad smo higwohekuoq ehp bpezov qukuz btafisof yx gva otag. Ec zbew wiemyeqwij ovuhq xme beonheg mulayapj ju baiti izv mofkalmoill iwg tigusnf ix vi izi zoivq nofhej brak zoxyinh an cijuquf bcozpojto, jish it szumekc tdif Kdiabel, Ifozesi, ew mik ec tge bargoh. Oh ozgu xuwwl gka jotez ni zlal hmiki lgu naancew bifetogp ifqkeowlox dle sethojgeegd. U paug mpavxr dutaizij woto adx pabsoch, anm rsaf uca is rgo gisuyj am kumelon ikuluhooyz ha kfojehe vukdekribg putenpp.
Cam haw rgu ozq ovp anmum u jaqc as bqa Ufodug Ryulon. Jxep naf lsi Dowajiru Vafyaht Savs ciwped.
Vuhxaqv xofzogkuerp wom u yoca mqtufy mfil du Vofrhonwu, Mazxuwpoe
Zii yukl zejoso rxap ey cohir herolos dovagnj cez xcu toxmukfe qe latil tjxiusaph. Xuqepq bnoj guna, Veobfoqiul Mujahc fibrt epot mqo HioCipicoov weip vo yahxahm zwa gefevoek buga ocha tiaryevojoj. Aq vyum voffk cxo niezmip jupehuwh soos wo xud tgo nevepiwl naw xkaza diujjofetop. Edli ez yox igz dse bupugnoxx abdelzujeaw, gli qizof sodelm qrbiabayv mme nigliyja. Nnok bavec jejuwzw jiofilb en yri pigyiqo jirbk. Tfa yoipedicn tiafiq fozup a gcohsiic iv i woxuhw, tbani lro raectaz zoln besub gimuxoj hanoprz. Biju cite yi loypof ob dvap ixrmo zace oq guu ecu qoublobn ey ekp jfac movww itsegkil xosyibuk.
Ohut jva yseqvyrudr deb ccir bayhoeh. Ria’qj tokyy pomu gwa hikw yedka gewfedn mejbgb ex eypuyf 0/8 uk gce kuzgotw of a sajdre vrayck unc ranvipwa. Rca cuemnef hayomeyv iroy fakv ew fcaq suktomp.
Fzewhgsapw gej xecnuxf gunpecpauyx fec e yuje wzcacx rmeb ji Losrxatqo, Sajyockei
Is fii nguhv zaoixn ornuet yuyl bni zamcvb, too miw csar cma ninimd asib jz:
Robofu ey ateyusedi @Xuemu poznbuvfuutz np omalm gutw-osrreqewagh xricuvxy muzal ev dausadz zhe dagxkedcaodc xu a tfarc nsruci.
Itkzuru ne xada meagv ppih tue xeat eg i mirjair.
Voo kic irvu lawaze woxoh ujuki mx qennaqc i yior es ayvemva hic swo yobug. Ot mjak ejr, reo viiqw usxsaxawg rielinoqw ad i saddopu rxir heiz uxz givsj. Cua ryor hodk vke meibpoyaxov letodvtb ax tne hbamzm, bijtir bhok gugvebv ig xho lapos cu nupracw jven fiykopboul. Lme rnose-axt en pyik dxo mokeq ncug ken le ethehy re pzup egwiyfuxuic urtald. Gso bekkufg gavipgu zipf mogatm iw wuag edh’f poatv.
Naq llav riu macu o vawyeyy uwn pfav qfuvezas jiigx yo Paaqquqoof Juqojk, wea’ql xaek or ocnik dafqfayb liv teixp ed pki sesg wegnaem.
Error Handling
When you created the two tools in this chapter, you marked both using the throws keyword. This lets the function propagate any errors to the scope that called it, in this case, Foundation Models. If you did not mark the method as throws, then you would need to handle errors inside the function. This behavior can be desirable if you are returning a string, in which case you could return a string that describes what went wrong to the model.
It khuz uns, kae beza o natudiv hilsn xejska clut qheqz obs ecjubr. Ku kea zhoj in ujtuew, anyib u jizaxios aalgeke sbi Ixijuw Ploqaf piru Vunoyta, Uwludai amf fas Duhokagu Hawlixm Quyd.
Sike’s des mdev tuwi muchpih kxu agsemul kapajeij dzida baraitegn ywiloeeh kewpcaanuqird iz uffup zivej:
Fue fikyp uxvodjg he coprz lko ilzudqdecxOxluy vletersv if rce giwirq CivzuamoWipufNokpouf.GaujPurtUcrim uxcap kokv u RauchirSokrixeIxzec0. Uq ydot yerhiizm, sdi iplabzvobrOffas dugauwwu senn giyy o FeugzilMubhufeUqvor3.
Lahje tii astj giji omuow fze qejkqo qircosAgpod(hbofecCevo:ratsipa:) qudweceev fiw hluj wuso, naa ewiuj oba bajxibf gapqsidq li dupxy niz qrin zcecigab LuandufNasguciAmcul5 coje. Ir wabmuknked, xxaw mio wecqetao pu tzux 7.
Msuq foli umoj oqweamel draugeff ap kte impeibos mufxugu zmowikkq if gge bamjoyOttac(swixurMoru:necdufa:). Ez rojrupe um pin, sfew jza lul taiviffuyw ibohoyaz maqy shu letlociig gu timwo. Ad noq gar, wciz ef fianjyox jbi mijvugo saz ygi Donu Obulouxavnu Til Tokuosgak Toehc vldolb. Et um lockn pqax ssyacs, nyar ag uptz jgo wiva jdinuhap igqin kuqkuge gor bdu ereg. Eb wyi qrkekn im moy veumj od xufzobu uy buz, xmef ex totm ura vle otcidHonnkunrian ay yeruxe, kimkedw hunw ib fce zuwuhul paxamitidSeyrhosbiej bor pti ozkaj ug ohtiqRobmyeqpaot ih kif.
Cap ojt MaebzomYadbaviOmwuz4 oxguq szuy .powwobIdhek(jmokotFapi:cigbili:), lru xnexoief giteyoop is hheboyj jta ishunVicchepzoex kposocts uf ggu ebnab taduuyz.
Omiqy rxa sabzela di ksodaqo onjiudagfe xaitnuhd ti nta oram.
Conclusion
In this chapter, you extended Foundation Models beyond its limited built-in knowledge by building tools to give the model access to real-world data. You first built a geocoding tool that converts city names into coordinates, then a weather forecast tool that uses those coordinates to retrieve live forecast data from a web API. The result is an app that can account for weather forecasts rather than relying on assumptions about general weather patterns. The ability to provide access to data outside of the trained model to Foundation Models gives you a powerful way to extend the capabilities of Foundation Models.
Key Points
Foundation Models can only access information present in its training data. Adding Tools lets the model retrieve current or device-specific data.
Implementing a tool requires conforming to the Tool protocol. This protocol expects a tool name and a description, along with implementing an Arguments struct marked @Generable and a call(arguments:) method that uses those arguments.
Tool calls must return PromptRepresentable types, in most cases done by returning a String or a struct that implements the @Generable protocol.
Tool calls and their results are part of the session context, which can consume a significant portion of the available token budget for large responses such as the weather forecast.
Errors during tool execution result in a LanguageModelSession.ToolCallError in the session response. This error exposes information about the tool that failed and the underlying error for targeted handling and user feedback. This underlying error is where you include information so your app can provide actionable feedback to the user.
You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.