Using CRaC with the Fn Project

5 min readFeb 22, 2023

CRaC

In November 2022 AWS launched a new service called Lambda Snapstart ( https://docs.aws.amazon.com/lambda/latest/dg/snapstart-runtime-hooks.html) that improves JVM startup time. It is a new solution to create checkpoints and restore through Firecracker microvm AWS implementation.

CRaC is a new technique to take a running JVM and save all its state to a directory. This save includes all JVM data, like compiled methods by hotspot, thread states, and all the Linux Process information.

The CRaC implementation is not available in the official JDK yet, however there are some custom JDK builds that enable this feature.

However, CRaC has some restrictions. It’s not possible to dump open files or open sockets. The restore process can happen on a different machine, and these resources maybe are not available in this new environment.

This strategy can work very well in a function environment. Every function stays in an idle method waiting for managed server communication and when a new request arrives from this server a small java framework invokes your function and sends the response using this socket:

FnProject

FnProject is an opensource project to implement Function as a Service (FAAS) and Oracle Cloud Functions is developed using this framework. This project will be used to the PoC of a FAAS with CRaC support.

There are several repositories in the project but the most important ones are:

FNServer — The server component can run a docker container with the function and orchestrate the communication with the function using a socket. This repo is strange because the last commits are about 4 years ago but it is working yet! (https://github.com/fnproject/fn) (https://github.com/tanquetav/fn/tree/crac)
FNCli - This is the CLI used for the project, to invoke, build and deploy the function. This tool must be installed to use this PoC (https://github.com/fnproject/cli)
FDK-Java — this is the java framework to receive the socket communication and invoke the function. (https://github.com/fnproject/fdk-java) (https://github.com/tanquetav/fdk-java/tree/crac)
Runtime environment. I cannot find where and how the oracle FN images are built.

It is necessary to change some of these components to implement a PoC to use CRaC with fnproject. My repos are listed above after the official repo.

FNServer

FNServer is the component that starts a docker container to run the function, maps some sockets shared between fnserver and function, and shut down the container if no request is received for a while.

To support CRaC some small changes were necessary for this component. It must start the docker container in privileged mode and a Volume must be shared with host to save CRaC data in checkpoint process if this saved state is present, restart from the savepoint. ( https://github.com/tanquetav/fn/commit/0c472432a3869e350d95774d3e391976f0bdc55e)

For this PoC a hardcoded directory is used: the host path /tmp/cracdata is mapped to /cracdata inside the container.

The new entrypoint of runtime docker image will check this path to restore or save the data.

FDK-java

This component must be changed a bit more. There is a class that handles the socket communication and I decided to delegate the socket to a new class, and implement a logic that after the fifth invocation, trigger the checkpoint, and dump VM State. Before it dumps, the socket is closed, and after the dump the socket is reopened. Any stacktrace is recorded in some files (/tmp/cracdata/aaa , /tmp/cracdata/bbb) to investigate any problem. Current CRaC implementation exits the process when the dump is finished.

The diff is bigger than the previous one ( https://github.com/tanquetav/fdk-java/commit/6edf3b07454891915556d23d44e0501b26d0c8a4) but all this logic is implemented on the new class SocketDelegate.java. The org.crac.Core.checkpointRestore() is triggered after some function invocations and the checkpoint process starts at line 98. After everything is saved, the process exits. When JVM is restarted the execution flow is restarted after line 98. The socket is reopened and the function is ready to handle new socket messages.

There are some callbacks available to monitor the checkpoint and restore at lines 30 and 35 just to log the checkpoint and restore process.

Image

After the build of the runtime some jar are generated and they are used to build the new runtime image. It is created using github actions and a new Dockerfile ( https://github.com/tanquetav/fdk-java/commit/195887a58a89b6341208fd0ad6eab18b19df4da6)

The process is simple:

Build the maven project
Copy runtime jar and the dependencies to docker/runtime folder
Build docker runtime image

The docker build process accomplish this:

Download some linux dependencies
Copy libfnunixsocket.so to image (library to socket communication)
Copy runtime jars
Download the custom JDK with crac support (https://github.com/CRaC/openjdk-builds/releases)
Install the entrypoint

The entrypoint is available on github. It checks if the /cracdata/file.img exists. If it is found the JVM is started with -XX:CRaCRestoreFrom=/cracdata flag to restore process from this directory. If the file does not exists, -XX:CRaCCheckpointTo=/cracdata flag instructs JVM to dump the checkpoint to directory when the code invoke the checkpoint method. After that, a restore process is invoked on JVM. All execution and some ps are redirected to /tmp/cracdata/log to monitoring. There are some bugs in this process to restart but after a few errors the restore works fine.

FNDemo

This small demo implements a demo function to do a hello world ( https://github.com/tanquetav/fndemo). A static block with a sleep of 5 seconds is implemented to simulate a slow framework to be started and we expect to not have this delay on function restore later (src/main/java/com/example/fn/HelloFunction.java).

A func.yml file must be created to describe the function to be compiled and deployed with fn cli:

schema_version: 20180708 
name: fndemo 
version: 0.0.1 
runtime: java 
build_image: fnproject/fn-java-fdk-build:jdk17-1.0.165 
run_image: tanquetav/fdk-java-crac 
cmd: com.example.fn.HelloFunction::handleRequest

The run_image is our custom image created before and the cmd specifies the class and method to invoke.

Let’s run this function:

First start in a new terminal the server environment:

docker run --rm -i --name fnserver -v $HOME/.fn/iofs:/iofs -e FN_IOFS_DOCKER_PATH=$HOME/.fn/iofs -e FN_IOFS_PATH=/iofs -v $HOME/.fn/data:/app/data -v /var/run/docker.sock:/var/run/docker.sock --privileged -p 8080:8080 --entrypoint ./fnserver tanquetav/fn-crac:latest

Next we need to register a new application to this server:

fn create app server

And then deploy this function. This process will build a docker image using the new runtime:

fn deploy --app server --local

Then we can invoke the function :

fn invoke server fndemo

This first run takes 5 seconds at least (the static block) and the /tmp/cracdata directory is created. We can check in the log ( /tmp/cracdata/log ) this init delay and we can see a container paused using docker ps.

Doing more invocations on we can check in the directory the dump is generated. It restarts the container restoring from the previous state.

The restore does not run the static block and is immediately available

FN Function CRaC Demo