Kubernetes: Implement Graceful Docker Container Shutdown

May 4, 2019 de Reviews azure, cloud, k8s, microsoft

Containers and Kubernetes as Container Orchestrator make it much more comfortable and more uncomplicated to run, operate and scale complex applications at ease. Long running tasks should be executed in worker services connected loosely to the app via reliable message transfer and queues. There are several technologies on the market but in the end the patterns a very similar.

Architecture: A process registers on a message queue via polling, web sockets or some other technology. This pattern blocks to process even for the time it’s waiting and doing nothing (idle mode) but also when a message is processed.

Continuous Deployment, Continuous Integration or the Immutable Infrastructure approach increases the number of deployments in a production environment dramatically. This means it is most likely that Kubernetes as the container orchestrator would shut down the container in the middle of a long-running task by sending a SIGTERM signal. This would end up in a hard shutdown which just interrupts the current work and leads to a lot of problems. It could also happen that your managed Kubernetes Cluster Service Provider just reorganizes the infrastructure without knowing the usage pattern of your application. It ends up in the same hard shutdown.

What means a graceful shutdown?
A graceful shutdown means that the hosting infrastructure announces a stop request and gives the container the chance to interrupt or finish pending work. As long the container does not commit the shutdown the infrastructure will wait, wait for updating the container image or moving the container to another host.

How to implement a graceful shutdown?
Kubernetes does not know a lot about the usage pattern of the containers, but it supports a preStop hook which can be used for a graceful shutdown. The preStop hook is executed inside the container and blocks the termination process in Kubernetes until the grace period is over. Implementing a graceful shutdown is now easy by configuring a preStop hook which blocks as long as the container is doing his work, the hook should also notify the container about this job because Kubernetes just waits the grace period. The grace period should be configured on the deployment level to a useful size which fits the usage pattern of your application.

Sample Based on the Azure Worker CoreHelpers
Based on the Azure Worker CoreHelpers a simple worker can be implemented which is establishing a preStop Hook based on files. Every other IPC technology can be used if required!

Visit the project

Establish Soft and Graceful Shutdowns where ever you can to add more reliability to the part of the service your customers will never see. Any other ideas regarding graceful shutdowns or workers in containers, just drop a comment below…

Kubernetes vs. Azure App Services WebJobs – Why I switched

July 17, 2018 de Reviews azure, cloud, microsoft

When Microsoft was launching Azure during the last years, the team invented a service called Azure WebSites in the early days. This service was an amazing step in the direction of making a very complex topic as easy as possible. Just today we are looking for a similar thing in Amazon Web Services without luck.

Nowadays the service is called Azure App Services and can be called the backbone of Microsofts strategy to make the life of Software as a Service developers easier. Definitely Azure App Services is the solution for headache free operations of Web Applications because you get so many important features for free e.g.:

Load Balancing
SSL Certificate Management
Patch Management and Operating System Updates
Deployment Slots

Besides that, an Azure App Service is fault tolerant even if you operate just a single instance and allows you to scale out based on many indicators like CPU, RAM or Queue Length. Adding WebJobs to App Services was the missing piece to bring the workload of a Web Application into the background, e.g. the processing of uploaded images or data reports which needs to be prepared.

As soon as Software as a Service offerings are growing (e.g. Azure Costs) the background work is something you could start worrying when using Azure App Services. The scale out features look great for small apps but have their pitfalls e.g. you can only scale out on instance level and not on job level so when you want to increase the amount of workers in App Services you need to increase the instance count. The other challenge we had is that the WebJob shares CPU and RAM with the Website process. This could influence the perceived performance of your application. This challenge could be covered by operating several App Service Plans in parallel and let the WebSite run in a different plan to the background worker.

If you are at this point the costs perspective comes into the game. Operating a App Services Plan which just runs WebJobs is a great thing but the reality is that you don’t need all the nice WebSite features and normally you want to have more granular ways for scale out. We at Azure Costs started searching for an alternative which runs as well in Azure because we would like to keep the traffic in one data center to keep an eye on these costs as well. During this research we identified that container technology could be very helpful because it allows us to spin another 20 containers just for scale out in a specific area.

Microsoft offers AKS (Azure Kubernetes Services) which gives you a fully managed Kubernetes Cluster running on Azure without having the pain to operate control plans. So it becomes very attractive also for smaller clusters consistent out of 2 up to 5 servers. Let’s review if and when yes, how AKS can cover the new requirements.

Requirement 1: Background Workers can’t influence the Web Site Host directly
We decided to let the WebSite be on an Azure App Service plan and use all the features like SSL certificates, Instance based Scale out based on CPU and RAM or Load Balancing. So by design the Kubernetes Cluster just runs on different hosts as the WebSite and they never influence each other on a direct way.

Requirement 2: Scale-Out is possible on a per job level
Kubernetes relies on docker containers which means what was a WebJob before, now becomes a container and it’s possible to scale out the containers as part of a deployment definition easily. Kubernetes has also native support for Azure to scale out the underlying servers called node pools in Azure. So the system becomes fully flexible and can breath as you need.

Requirement 3: Dedicated Resource Allocation for Job-Worker are possible
Azure App Services is totally shared which means the jobs are fighting with the Webserver for resources like RAM and CPU. It’s even not possible to allocate a minimum amount of CPU tickets to a dedicated worker. It could happen very easily that you overload the CPU when a lot of background work happens. Kubernetes has a different concept and strict Quality of Service classes (QOS) to avoid this topic and make it manageable.

Requirement 4: Patch-Management comes for free
Patch-Management is somehow the downside of AKS because you are operating virtual machines in the cloud and it’s up to you to trigger updates and so on. We decided to follow the idea of immutable infrastructure which means when we need an update we just re-deploy the whole cluster and remove the old one. It gives use all the capabilities Microsoft invests in his virtual machine image gallery for free.

Finally we defined for us that a good structure is to bail out all background work into a growing Kubernetes Cluster and dockerize our whole background logic. Since we did the change we have full control of resource allocation and can easily scale out on a per job level which means more people need on demand reports which should not block the IIS threads the system scales up this group of workers. Over the night when we import tons of data, the system scales up these kind of workers very easily.

Reviewing the costs speaks also a positive language. As Microsoft is not charging you for cluster management you just pay the virtual machines, and this means normally machines on premium storage for approx. 90% of the price of the corresponding App Service plan. Normally the machines can have double of the RAM the App Service Plan offers.

I can recommend the combination of Azure App Service Plans for your WebSite and WebServices and a Managed Kubernetes Cluster know as AKS for all the work behind the scene. How do you think about this architecture, did I miss something? Do you follow a different approach?

Hey node developers, switch to .NET Core – now!

August 7, 2017July 26, 2017 de Reviews c#, coding, javascript, microsoft, node

Several years ago I started building a bigger project as a Software as a Service application. Beside all the different technical requirements, being able to work directly on my MacBook Pro without starting virtual machines, was a big wish. At this time a tool chain based on Node.js, Express, NPM and WebStorm was available. Over the years building backend services with Node.js, and this means with JavaScript, felt like rapid prototyping. Getting started is very fast and lightweight but when the project grows compile time features like a strong type system are missing. Year over year I did reviews how to get esp. the beauty of C# and the powerful compiler infrastructure of .NET back.

In the last month Microsoft released Visual Studio for Mac and with it the investments into .NET Core increased. Building backend services in Visual Studio for Mac based on .NET Core and ASP.NET Core including the out of the box support in Azure App Services is definitely what I was looking for.

Because of that I decided to give it a spin. After a couple weeks working with the framework I can say it was the right decision because of the following key reasons:

Full managed and type safe environment based on the powerful C# compiler
Broad ecosystem for components similar to NPM via NuGet
Ability to convert runtime errors in compile time errors
Hosting the results on Linux, Docker Containers or just Azure App Services
ASP.NET Core comes with out of the box easy to use Dependency Injection system
ASP.NET Core learned and steeled the best things from the node + express chain
C# Attribute and Extension-Classes are unbeatable to beatify your code

And last but not least everything works well on my MacBook Pro without the need of a single virtual machines – Thanks Microsoft for letting me keep the platform I love!

Git rid of your XAML builds – Migrate now

April 23, 2016April 24, 2016 de Reviews azure, cloud, coding, git, hockeyapp, tfs

Have you recently migrated from TFS to Visual Studio Team Services?
Are you happy with that decision but the existing XAML based build definitions sucks? Then this article might help, as it explains the structure of the agent based build system, which is very powerful especially for companies with cross platform projects.

The Architecture

In the former days the XAML based build system used a dedicated build controller which was able to trigger a build via push. This meant that the build agent needed to be available from the build controller. Hosting build agents in the cloud or better build agents dynamically, was always a pain.

With the new step-based build system of Visual Studio Team Services, Microsoft simplified the architecture and removed the build controller out of the game. The only component which is required is a simple agent that needs to be installed. Based to Web-Sockets this agent does not need to be available from outside because the agent dials into the VSTS services and offers triggering build runs. This makes adding and removing build agents dynamically a breeze. Just prepare your AMI, VHD or other image and spin off new virtual machines if needed.

The Step System

The described new architecture also delivers a brand-new step-based build system. Editing complex XML based workflows in Visual Studio is not required any more. The author of a build-process can pick and choose pre-defined activities to execute a very specific build-step, e.g. compile source code, generate a web deploy package or sign files.

At the end every build step is nothing more then a command line tool, that is executed in the context of the current build. Because of that, also the Command-Line-Activity is one of the most important actions in the step-based build system when it comes to more complex workflows.

Cross-Product-Groups/Departments will love the possibility to create new custom activities, which can be used from other teams and groups within the company. This makes complex build tasks really easy to use for other teams. There is an important idea behind the new build system: “Let developers create the build, get rid of your dedicated build team.” In an Agile world we are talking so much about cross functional teams, that build management should become part of it.

Advanced features you should know about

Microsoft also delivers a couple of features in the new step-based build system to target more complex scenarios.

Multiple Build Agents on the same machine

There are no reasons for using just a single machine per registered build agent. If you would like to get the most out of your existing infrastructure, just install the Microsoft build agent multiple times on your build servers.

Tip: We are using 3 build agents on the same virtual machine which gives us a good utilisation regarding CPU and RAM. On Linux we switched to machines backed via SSD because of I/O intensive docker and Java builds

Release Processes vs. Build Processes

The new release process feature in VSTS is a great option to differentiate between building a pice of software and delivering a piece of software. Especially when the vendor works with a test chain it’s possible to move SaaS software through different states. Especially codeship becomes popular with this kind of feature set.

Push to HockeyApp

Microsoft acquired HockeyApp, a leading vendor in crash dump collection and beta app deployments for Android, iOS and Windows.

Related to this acquisition the Visual Studio Team Services team offers a seamless integration, which allows to upload your build directly to HockeyApp. This works on all required platforms, also on XCode builds for iOS or OS X. At Microsoft Build 2016 the HockeyApp team announces that Xamarin Insights becomes part of HockeyApp. Related to this check our the technical presentation of the HockeyApp team here.

Summary

With the new step-based build system Microsoft introduced in Visual Studio Team Services, a more flexible and simple-to-use option. Especially when you start with new projects, just use the step-based build system and do not invest in the old XAML builds. Software vendors who are writing software for different platforms can now use VSTS as a cloud based management tool, which means to get the most out of your MSDN benefits. Also when you are owning XAML based builds, invest in the migration. Microsoft recently announced that the XAML builds will be discontinued after 2016, so it’s time act.

Review: GitHub vs. Visual Studio Team Services (VSTS) – Should you switch?

January 23, 2016January 23, 2016 de Reviews azure, cloud, coding, git, microsoft, SaaS

Disclaimer: This article is about Visual Studio Team Services (VSTS) and GitHub. The author has a very positive opinion about both services and does not get any offers from Microsoft or GitHub writing this article. The whole article is written IMHO.

Over the last years GitHub has become a success factor for projects related to git repositories for me personally. I also tested several other solutions but mainly I struggeled with the performance or the usability. About 2 years ago I started using Visual Studio Team Services (VSTS) with a work related project. The service looked promising but had a lot of early release issues, so in the last 24 months it was interesting to see how a promising solution became adult. Last week I decided to move away from GitHub for all my closed source projects and rely on Visual Studio Team Service (VSTS). This article gives a deeper look on the main reasons of why I did the move and may help deciders to get detailed information before implementing:

Git-Repository sprawl
Nowadays thanks to bower, npm, bundler or NuGet the amount of Git repositories are growing steadily. When you are focused on component based software development Git is a great helper but the amount of repositories is sprawling because very often every component has its own live in a separated Git repository, which means you have a separate release cycle and a different versioning compared to your main project.

I guess this is the secret source of component managers which are working closely together with git repositories. Of course VSTS and GitHub are supporting multiple git repositories but GitHub lets you pay on a package on git repositories, VSTS lets you pay for users. Especially for small teams paying for users is the better deal, compared to paying for repositories. At the end Microsoft offers a smaller groups of 5 everything for free. This fact stopped my permanent GitHub problem: Having too less private repository space.

GitHub:
O – Allows to have as many GitHub repositories as needed
+ – Generates releases out of tags automatically (good semver integration)
– – Charging base is the count of private git repositories

VSTS:
O – Allows to have as many GitHub repositories as needed
+ – Comes with unlimited git repositories, plans are user based
– – Repository management is not that intuitive as it is in other solutions

Pull Requests and Forks
Forks and pull requests are the most important features GitHub introduced very early to support community driven development. I would say in the open source world Github is the platform when it comes to forks and pull requests. Currently I would never think about moving open source repositories away from GitHub because of this great feature.

When it comes to closed source projects, forking and pull requests are becoming only important in bigger teams with different products or product lines. We are using this feature strongly in our teams at my company. Smaller startups or teams will not use these workflows often but nevertheless it is a road blocker for me to do the 100% switch to VSTS. I guess it took Microsoft around 12 month to deliver it in a more simple way GitHub is doing this.

GitHub
+ – Cross repository forking
+ – Pull requests incl. discussion thread and a lot of community features

VSTS:
O – Supports pull request on repository level

Agile Process support with EPiCs, features and backlogs
When it comes to bigger teams or more structure, people are having the option that to implement a process is the key. This brings me to the biggest enterprise blocker I see in GitHub, the Issue-Tracking system. Companies who migrated to an agile framework like SCRUM or KANBAN need to have the option to work with features, backlogs or bugs. Microsoft delivers with VSTS a highly customizable and adaptable work item management system. The SCRUM and KANBAN template makes perfect sense for agile teams but also the traditional waterfall model can be implemented (even if I don’t understand why someone should do this.)

GitHub
– – It’s just an Issue Tracker
+ – Has good integration into many cloud services

VSTS:
+ – Offers customisable work item management
+ – Comes with templates for agile team structures

Handle your Test-Cases
Even if your projects comes with a great code coverage and good unit tests, the requirements to execute manual tests or just to orchestrate automated integration tests exists. VSTS implements based on the work item management a test case management which has the option to integrate it with automation bots via WinRM and other protocols.

The ability to document test cases and write specific step by step guides how to verify a feature is a big plus esp. in small teams where no dedicated QA resources are hired.

GitHub
O – Ability to integrate with external QA services
– – No integrated test case management

VSTS:
O – Ability to integrate with external QA services
+ – Test Case management is part of the work item management

Centralised Source Code management as migration path
For a couple month Microsoft offers virtual TFS collection, which allows companies that want to go pave the road for git based repositories to stay with the existing centralised source code management besides new git repositories. In the early day VSTS just supported a TFS collection per project space but now maintaining TFS collection is that easy as it is creating new git repositories. This will become very important features for existing TFS customers.

Revised Build-System incl. Linux Support
I guess Microsoft learned very fast that the XAML file based build system was very inflexible and complicated for a SaaS service like VSTS. Because of that a couple month ago a new step based build system was introduced which will orchestrate the build agent out of VSTS

Since Microsoft supports Windows, Linux and Mac build agents there are no road blockers anymore, besides that the hosted build services for VSTS are very rare. There is a build server Microsoft offers out of the box but from my experience you will get more problems customizing that. When you are focused on Azure check also what the Azure App Services can do for you because KuduSync offers out of the box build for your .NET application during deployment.

GitHub
+ – Many different build services available with GitHub integration (CodeShip, T..)
– – Build definitions are not part of the code project
– – No release management to aggregate several projects

VSTS:
– – Hosted build services for VSTS rare
+ – Build definitions are part of the code project
+ – Release Management allows to aggregate several sub projects

Other services and options:
There are also other options and products on the market and I guess one of the most popular one is Assembla which is pretty comparable to Visual Studio Team Services. Also the products from Atlassian (Jira, BitBucket) are great options supporting your development cycle. This article had the intention to support companies who are dealing with GitHub and / or On Premise TFS and now are thinking about combining the positive of both.

I personally think GitHub could become a great option for enterprises as well, when the Issue-Tracker problem is solved which was mainly the reason why I searched for an alternative!

Thoughts about making software

a blog of Dirk Eisenberg

Reviews

Hey node developers, switch to .NET Core – now!