Streaming Payloads

When uploading or downloading large blocks of data it is preferable if you can stream this data instead of holding it all in memory. Soto supplies methods for streaming raw data payloads of both requests to AWS and responses from AWS. The most common use of this would be when uploading or downloading large objects to S3.

Payload object

All raw data payloads in Soto are represented by an AWSHTTPBody object. This can be initialized with Data, String, ByteBuffer or an AsyncSequence whose Element is ByteBuffer.

Request streaming

let stream = AsyncStream<ByteBuffer> {
    return try await giveMeAChunkFromMyPayload()
}
let body = AWSHTTPBody(asyncSequence: stream, length: 2*1024*1024)
let request = S3.PutObjectRequest(body: body, bucket: "my-bucket", key: "my-file")
let response = try await s3.putObject(request)

If the AsyncSequence provides more or less data than you specify an error will be thrown. In some cases you might not need to provide a length.

File uploading

It is easy to implement streaming of a file using NIOFileSystem.

import _NIOFileSystem

try await FileSystem.shared.withFileHandle(forReadingAt: .init(filename)) { handle in
    let fileSize = try await handle.info().size
    let fileChunks = handle.readChunks(in: 0..<fileSize, chunkLength: .kilobytes(32))
    let body = AWSHTTPBody(asyncSequence: fileChunks, length: fileSize)
    let request = S3.PutObjectRequest(body: body, bucket: "my-bucket", key: "my-file")
    let response = try await s3.putObject(request)
}

Response streaming

When a response has a streamable payload that payload will be part of the response type returned by the operation. It will be returned as a AWSHTTPBody which conforms to AsyncSequence.

let getRequest = S3.GetObjectRequest(bucket: "my-bucket", key: "my-file")
let response = try await s3.getObject(getRequest)
for try await buffer in response.body {
    processByteBuffer(buffer)
}