How the inventor of AWS Lambda built their latest stack 60x faster

ServerlessConf NY this past October was an important milestone for those of us tracking how software is built on cloud services. . We’ve seen the serverless talks evolve from “what is serverless” to “I built a weird thing” to “We built a new business” to “We refactored a legacy app and kickstarted our feature velocity.” We’ll come back to those last two soon, , but I want to highlight some points from one in particular by Tim Wagner. Tim is generally credited as the “inventor of serverless” and his presentation showed how serverless now supports a thriving ecosystem and where the boundaries of the technologies can be pushed.

Tim’s goal was to try to build something with serverless that was widely regarded as the wrong application for it. He decided to focus on laying the groundwork for a serverless supercomputer. Sound nuts? Definitely. But Tim himself said in 2017, that there were some things serverless just wasn’t usable for, like video encoding. That same year, presenters from Stanford and UC San Diego demonstrated that it was not only possible to video encode with serverless, but actually superior to traditional compute methods in many ways.

Within a few months AWS would have a product performing serverless video encoding!

If live video encoding works well with serverless, what else could be possible?

When Tim started to work on his supercomputer idea, he knew it involved coordinating a lot of Lambdas and performing WebSocket communication. Since this meant deploying and configuring multiple Lambdas, the bulk of his effort was spent creating CloudFormation templates. By the time he had any appreciable progress, he’d burned over 15 hours!

(A caveat — what Tim was trying to do with WebSockets isn’t well documented. If this was a well-known use case with better documentation, it might have only taken 10 hours instead of the 15 just to write the CloudFormation template!)

This is the story of how 15 hours of hand-written CloudFormation became 15 minutes of using Stackery.

Let’s think about this situation for a minute. You have someone who essentially has a Ph.D. in AWS who wasn’t able to move fluidly in these tools. That almost sounds like serverless really wasn’t “ready for prime time”, but we must consider the alternatives. With handwritten CloudFormation, you’re getting high-performance infrastructure that can easily scale after a few day’s of work. This is way faster than waiting for an order from Cray Computers (or Dell, HP, Cisco, etc) and even faster than standing up databases, streams, auto-scaling, and whatever else you might run in a fleet of auto-scaling virtual machines in the cloud.

It’s really surprising how much this new model has accelerated platform development; serverless can be extraordinarily fast to those who are used to waiting. But for those of us who have shifted to a serverless mindset, more than an hour from idea to execution feels slow. In this case, Tim did a cool thing and documented each step in his 15-hour process.

If you’re spending days or weeks configuring a stack, obviously it’s not something you’re going to be eager to repeat. One of the things we see at Stackery is that infrastructure templates and architectures are not static environments. Architecture is not a one-time or even an annual process. We see Stackery customers iterating on their architectures more than once a week, it’s not as often as application code, but shifting the architecture with the code has obvious advantages

After his hard work, Tim started over again. This time, he used Stackery to configure and deploy his stack. Along with generating YAML for you, meaning you’re not having to manually edit a whitespace-dependant config file, Stackery can manage dependencies, permissions, and environment variables to make a complex process a lot simpler.

The result was a much MUCH faster deployment for this first step to a serverless supercomputer. I’m not sure serverless technology will be winning any Jeopardy tournaments this year, but the future of serverless continues to offer developers a path toward delivering business value in use cases we couldn’t even imagine two years ago.

This talk was near and dear to our hearts for another reason: Tim Wagner, the inventor of serverless who had his process so accelerated by Stackery’s tools, joined the board of Stackery this month.