Exploring Serialization and Deserialization (SerDes) in AWS Lambda Durable Functions

The working of AWS Lambda Durable Functions relies heavily on checkpointing. Information such as the durable step type, name, input, result, and timestamps is checkpointed. And this checkpointed information is the key ingredient behind the flawless replay and retry mechanism of durable functions 💪🏼

Now information can’t be stored in the checkpoint as it is. It first needs to be converted into a format that can be persisted. And SerDes is all about dumping and loading data to and from the checkpoint.

What is Serialization and Deserialization (SerDes)?

Serialization is the process of converting data into a format that can be conveniently stored in the checkpoint. In comparison, deserialization is the process of reconstructing the data from the checkpoint.

By default, durable execution sdk uses JSON.stringify (official docs) to serialize and JSON.parse to deserialize the data.

But in some cases, this default behaviour may not satisfy our needs. And for that, we can use our own SerDes strategy.

What is the Problem with the default SerDes strategy?

Consider a scenario where a durable step returns an instance (object) of a class. And that class has a method in it.

import { withDurableExecution } from "@aws/durable-execution-sdk-js";

class Order {
    id = '';
    createdAt = '';

    label() {
        return `Order ${this.id} placed at ${this.createdAt}`;
    }
}

export const handler = withDurableExecution(async (event, context) => {
    const order = await context.step('fetch-order', createOrder);
    
    return { label: order.label() };
});

function createOrder() {
    const order = new Order();
    order.id = 'order-123';
    order.createdAt = new Date().toISOString();
    return order;
}

In the above example, order.label() will throw an error because that method is lost during serialization and deserialization. When you do JSON.parse(JSON.stringify (order)), the label() method is lost. That’s because JSON.stringify serializes only data, not the class prototype where methods like label() live.

You might be wondering what need for serialization is here?

The answer is that before data is returned by the durable step, it is serialized and deserialized as follows:

Solving the problem caused by the default SerDes using a custom strategy

Durable execution allows attaching a custom SerDes strategy to a durable step. And in that strategy, we can define how serialization and deserialization should behave.

import { withDurableExecution } from "@aws/durable-execution-sdk-js";

class Order {
  id = '';
  createdAt = '';

  label() {
    return `Order ${this.id} placed at ${this.createdAt}`;
  }
}

const orderSerdes = {
  serialize(order) {
    return JSON.stringify({ id: order.id, createdAt: order.createdAt });
  },
  deserialize(data) {
    const parsed = JSON.parse(data);
    const order = new Order();
    order.id = parsed.id;
    order.createdAt = parsed.createdAt;
    return order;
  }
};

export const handler = withDurableExecution(async (event, context) => {
  const order = await context.step('fetch-order', createOrder,
    { serdes: orderSerdes }
  );

  return { label: order.label() };
});

function createOrder() {
  const order = new Order();
  order.id = 'order-123';
  order.createdAt = new Date().toISOString();
  return order;
}

Notice, we are still using JSON.stringify for serialization. But during deserialization, we are constructing a new instance of the class to recover the lost method.

And durable SDK provides an even simpler way to achieve this same thing in fewer lines of code using the createClassSerdes helper function:

import { withDurableExecution, createClassSerdes } from '@aws/durable-execution-sdk-js';

class Order {
    id = '';
    createdAt = '';

    label() {
        return `Order ${this.id} placed at ${this.createdAt}`;
    }
}

const orderSerdes = createClassSerdes(Order);

export const handler = withDurableExecution(async (event, context) => {
    const order = await context.step('fetch-order', createOrder,
        { serdes: orderSerdes }
    );


    return { label: order.label() };
});

function createOrder() {
    const order = new Order();
    order.id = 'order-123';
    order.createdAt = new Date().toISOString();
    return order;
}

Cost savings using custom SerDes (createFileSystemSerdes) and S3 Files

Checkpoint payloads contribute to Durable Execution costs, so reducing the amount of data checkpointed can reduce costs.

To save cost, we can utilize the createFileSystemSerdes function that creates a SerDes strategy to write actual data in the filesystem and store a pointer to that data in the checkpoint.

Thanks to the newly launched S3 file feature, we can use an S3 bucket as a data source. So create a durable Lambda function with S3 files integration and use the code:

import {
  withDurableExecution,
  createFileSystemSerdes,
} from "@aws/durable-execution-sdk-js";

export const handler = withDurableExecution(async (event, context) => {

  const fsSerdes = createFileSystemSerdes("/mnt/s3");

  const profile = await context.step(
    "load-profile",
    async () => loadLargeProfile(event.customerId),
    { serdes: fsSerdes }
  );

  return { name: profile.name };
});

async function loadLargeProfile(customerId) {
  // Example implementation
  return {
    name: "John Doe",
  };
}

Conclusion

In this blog post, we understood how serialization and deserialization work under the hood in a durable Lambda function. And how we can override the default behaviour according to our needs. We also explored how we can use the SerDes strategy to save costs by reducing data stored in the checkpoint.

Hope you enjoyed the blog. Thanks 😊