Containers and Kubernetes as Container Orchestrator make it much more comfortable and more uncomplicated to run, operate and scale complex applications at ease. Long running tasks should be executed in worker services connected loosely to the app via reliable message transfer and queues. There are several technologies on the market but in the end the patterns a very similar.

Architecture: A process registers on a message queue via polling, web sockets or some other technology. This pattern blocks to process even for the time it’s waiting and doing nothing (idle mode) but also when a message is processed.

Continuous Deployment, Continuous Integration or the Immutable Infrastructure approach increases the number of deployments in a production environment dramatically. This means it is most likely that Kubernetes as the container orchestrator would shut down the container in the middle of a long-running task by sending a SIGTERM signal. This would end up in a hard shutdown which just interrupts the current work and leads to a lot of problems. It could also happen that your managed Kubernetes Cluster Service Provider just reorganizes the infrastructure without knowing the usage pattern of your application. It ends up in the same hard shutdown.

What means a graceful shutdown?
A graceful shutdown means that the hosting infrastructure announces a stop request and gives the container the chance to interrupt or finish pending work. As long the container does not commit the shutdown the infrastructure will wait, wait for updating the container image or moving the container to another host.

How to implement a graceful shutdown?
Kubernetes does not know a lot about the usage pattern of the containers, but it supports a preStop hook which can be used for a graceful shutdown. The preStop hook is executed inside the container and blocks the termination process in Kubernetes until the grace period is over. Implementing a graceful shutdown is now easy by configuring a preStop hook which blocks as long as the container is doing his work, the hook should also notify the container about this job because Kubernetes just waits the grace period. The grace period should be configured on the deployment level to a useful size which fits the usage pattern of your application.

Sample Based on the Azure Worker CoreHelpers
Based on the Azure Worker CoreHelpers a simple worker can be implemented which is establishing a preStop Hook based on files. Every other IPC technology can be used if required!

Visit the project

Establish Soft and Graceful Shutdowns where ever you can to add more reliability to the part of the service your customers will never see. Any other ideas regarding graceful shutdowns or workers in containers, just drop a comment below…