One of the challenges of building a serverless API is handling binary data. The main serverless API platforms, like AWS API Gateway, only have ancillary support for binary data. Further, binary data handling is inefficient as the data is encoded using base64 format. Taking into account the message payload limits of AWS Lambda, the maximum size for binary request and response payloads is 4.5 KB. This may be enough for an icon, but not enough for even a small image.

Luckily, there's another mechanism for handling binary data: redirects using AWS S3 signed URLs.

What Are AWS S3 Signed URLs?

AWS Simple Storage Service (S3) provides storage and access to arbitrary files. Usually, you use an SDK to upload and download files from an S3 bucket. However, it is possible to generate temporary signed URLs to upload and download files using simple HTTP methods (GET, PUT, DELETE).

These signed URLs enable you to share links to content within a private bucket. The signatures are specific to each file in the bucket, so you can have one bucket for many users of a product. By sharing signed URLs of specific files with individual users you can leverage having a private, multi-tenant store with extremely fine-grained access controls.

OK, But How Do I Use Signed URLs?

I'm glad you asked! Let's say you have an API endpoint to retrieve an image. It's not possible to respond from a Lambda function through API Gateway with the image data. But we can send an HTTP response to the client that redirects them to the image in an S3 bucket using a signed URL:

HTTP/1.1 307 Temporary Redirect Location: https://s3.amazonaws.com/mybucket/myimage.png?AWSAccessKeyId=AKISKSD87A3C4&Expires=109838429&Signature=s98df7s8df12f2jo4lfjfs9d0fu0sd9f

The beauty of using this approach is that redirection for signed URLs works not only for getting files, but also for uploading and deleting files. Simply change the request method from GET to PUT or DELETE. The client, whether it's your browser or a command line tool like cURL, should respect the redirect

Do You Have an Example?

Of course I do! Check out the Stackery S3 Signed URLs Example repo!

This little-known technique of using signed S3 URLs can unlock huge potential for serverless APIs. Keep it in mind the next time you need to host large and/or binary data through your services!