Soto and Swift Build Plugins

Swift 5.6 introduced a new Swift Package Manager feature: build tool plugins. These allows custom tools to be invoked during the build process. They are primarily used for generating Swift source code from another source.

AWS define the APIs for all their services using the Smithy interface design language (IDL). The majority of Soto is generated code built from these model files. The SotoCodeGenerator project is used to build the Soto source files from the AWS model files. Currently with each new release of Soto, the latest model files are copied from the aws-sdk-go-v2 repository and new Soto source code is generated. This creates a lot of code, 1430181 lines at the last count. Along with the model files Soto is a very large repository to download.

Given Soto source code is generated, could we use a SwiftPM build plugin as a means to avoid downloading all this extra source code? The answer to that is yes, with some gotchas. There is a pull request (at the time of writing this it is still unmerged) that adds a build plugin to the SotoCodeGenerator project. The build plugin generates the Soto source code locally from a Smithy model file.

Using plugin

It is possible to use this build plugin, alongside SotoCore (Repository containing the underlying code that runs Soto) to give a project access to an AWS service without including the whole of Soto as a dependency. You can set this up as follows.

SwiftPM build plugins are a Swift 5.6 feature so your Package.swift should indicate it needs Swift 5.6 or later

// swift-tools-version: 5.6

Where previously you would include Soto as a dependency instead include the following

.package(url: "https://github.com/soto-project/soto-codegenerator", from: "0.6.0"),
.package(url: "https://github.com/soto-project/soto-core.git", from: "6.4.0")

And the target you want to add the generated Soto code to should be setup as follows

.target(
    name: "MySotoServices",
    dependencies: [.product(name: "SotoCore", package: "soto-core")],
    plugins: [.plugin(name: "SotoCodeGeneratorPlugin", package: "soto-codegenerator")]
)

You then need a couple of files from the Soto repository.

• The region endpoint definition file endpoints.json
• The model file for the service you want to use. You can find these here.

Copy both of these into the Source folder of the target where you want to generate the Soto Swift code. If you have multiple targets you are generating code in, you can copy endpoints.json to the root of your project instead. This will avoid having multiple copies of the file across multiple targets. Unless you are using a new region endpoint or new functionality from a service you shouldn't need to update these files again.

Finally if your target only includes AWS Smithy model files in it, you need to add a dummy empty Swift file to the folder. Otherwise SwiftPM will warn you have no source files for your target and not build the target. You could add a comment to this file to document where the generated Swift source files are.

Now build your target

swift build

The build plugin will put the generated Swift code in .build/plugins/outputs/<package name>/<target name>/SotoCodeGeneratorPlugin/GeneratedSources and will include it in the list of Swift files for that target.

A sample project using the SotoCodeGenerator build plugin can be found here.

Gotchas

Earlier I mentioned there were some gotchas using the build plugin.

• If you are dependent on extension code in the Soto repository eg S3 multipart upload, STS/Cognito credential providers or DynamoDB Codable support they will not be available. You need to include Soto as a dependency to have access to these.
• VS Code has some issues when working with build plugins. It looks like SourceKit-LSP is interfering with the build process and it fails to link symbols on the first attempt to build a project. Subsequent builds generally work, but not always. Also once you have generated your source code you need to restart the SourceKit-LSP server (either by restarting VS Code, or invoking the Developer: reload window command) to get code completion, go to definition etc to work. Hopefully these issues can be resolved in the near future.

Removing source code from Soto

In theory I could replace all the generated source code in Soto with the build plugin. This isn't something I am ready to do yet. This feature is still experimental. Also build plugins require Swift 5.6 and Soto supports earlier versions of Swift so this is not possible until we drop support for those earlier versions.

Maybe at some point in the future though you might see a version of Soto with almost all of the Swift source code removed from it.