Hey node developers, switch to .NET Core – now!

August 7, 2017July 26, 2017 de Reviews c#, coding, javascript, microsoft, node

Several years ago I started building a bigger project as a Software as a Service application. Beside all the different technical requirements, being able to work directly on my MacBook Pro without starting virtual machines, was a big wish. At this time a tool chain based on Node.js, Express, NPM and WebStorm was available. Over the years building backend services with Node.js, and this means with JavaScript, felt like rapid prototyping. Getting started is very fast and lightweight but when the project grows compile time features like a strong type system are missing. Year over year I did reviews how to get esp. the beauty of C# and the powerful compiler infrastructure of .NET back.

In the last month Microsoft released Visual Studio for Mac and with it the investments into .NET Core increased. Building backend services in Visual Studio for Mac based on .NET Core and ASP.NET Core including the out of the box support in Azure App Services is definitely what I was looking for.

Because of that I decided to give it a spin. After a couple weeks working with the framework I can say it was the right decision because of the following key reasons:

Full managed and type safe environment based on the powerful C# compiler
Broad ecosystem for components similar to NPM via NuGet
Ability to convert runtime errors in compile time errors
Hosting the results on Linux, Docker Containers or just Azure App Services
ASP.NET Core comes with out of the box easy to use Dependency Injection system
ASP.NET Core learned and steeled the best things from the node + express chain
C# Attribute and Extension-Classes are unbeatable to beatify your code

And last but not least everything works well on my MacBook Pro without the need of a single virtual machines – Thanks Microsoft for letting me keep the platform I love!

Git Deployment – Shallow Clone Support in Azure App Services – The missing piece

July 24, 2017July 18, 2017 de Misc codeship, coding, javascript, microsoft, node

Azure App Services and the open source project KuduSync behind this great Azure Service is a huge time saver for agile teams. Especially DevOps teams will like the continuous deployment features. Personally I focus a lot on the Git based deployment which enables you to roll back and forward in seconds whenever it is required. Beside that, it is possible to work with standard tools available on market to implement continuous deployment or integration.

When I started working with Azure App Services building Node.js apps, I wrote a little node package called Azure Deploy. It allowed me to push changes as part of a build process directly into the Azure App Service. Originally, CodeShip was the service of choice for the build process but since I need to support Git Repositories beside GitHub, BitBucket and GitLabs, I migrated to Visual Studio Team Services (VSTS) and the integrated build platform.

After several months and hundreds of deploys, which means hundreds of commits to the local git repository, it became a fairly complex and fat thing. This is normally not a problem but my Azure Deploy package clones the local git repository from Azure App Service to a temp directory and copies the build output over it. Last but not least it commits and pushes the changes back to Azure. The big repository took more than 4 minutes to clone so I was wondering if I can use Shallow Clone to get only the latest state of the repository.

This idea works well on Unix based git servers, on GitHub or even in Visual Studio Team Services as well. But when you try to clone a local Git Repository of Azure App Services via Shallow Clone option

git clone --depth 1 https://github.com/jquery/jquery.git jquery

it ends up with an error. The error and its background is also documented in the GitHub project of KuduSync here. So what to do now?

Another nice option of Azure App Services is the option to pull changes from a Git Repository instantly after a commit. This works well in VSTS, based on GitHooks but also with GitHub and a couple other platforms. It’s also possible to clone via shallow clone flag from these repositories which closes the loop. The final solution is to commit into a VSTS or GitHub hosted publishing repository which triggers a pull deployment in Azure App Services.

At the end this change reduced the whole deployment time from 5 up to 9 minutes, down to approx. 90 seconds. You can find the updated Azure Deploy component in the NPM registry here.

Streamlining API Testing with Postman

September 10, 2016 de MyWorkspace api, coding

David Creedy a colleague and teammate published a great series of blog articles to explain how to handle API workflows because we are heavily involved in API related projects in our day by day business. Postman is definitely the tool of choice to supercharge all your API workflows and the following blog series will give everybody an important overview:

Streamlining API Testing with Postman – Part 1
Part 1 gives an overview about the different features of Postman and will guide you through a couple use cases to consume APIs as fast as possible.
Streamlining API Testing with Postman – Part 2
Part 2 enables you to save and organize API queries in libraries to share them fast and easy with your teammates.
Streamlining API Testing with Postman – Part 3
Part 3 explains how to leverage Postman for API testing. Especially teams working in an agile process needs to ensure that nothing is broken to hit their release goals.

So I hope David will move forward with additional blog posts in this series. All what I hear the next post, we will look at how you can use different APIs that are available. I guess addressing the various options how to deal with authentication is worth an extra article.

Git rid of your XAML builds – Migrate now

April 23, 2016April 24, 2016 de Reviews azure, cloud, coding, git, hockeyapp, tfs

Have you recently migrated from TFS to Visual Studio Team Services?
Are you happy with that decision but the existing XAML based build definitions sucks? Then this article might help, as it explains the structure of the agent based build system, which is very powerful especially for companies with cross platform projects.

The Architecture

In the former days the XAML based build system used a dedicated build controller which was able to trigger a build via push. This meant that the build agent needed to be available from the build controller. Hosting build agents in the cloud or better build agents dynamically, was always a pain.

With the new step-based build system of Visual Studio Team Services, Microsoft simplified the architecture and removed the build controller out of the game. The only component which is required is a simple agent that needs to be installed. Based to Web-Sockets this agent does not need to be available from outside because the agent dials into the VSTS services and offers triggering build runs. This makes adding and removing build agents dynamically a breeze. Just prepare your AMI, VHD or other image and spin off new virtual machines if needed.

The Step System

The described new architecture also delivers a brand-new step-based build system. Editing complex XML based workflows in Visual Studio is not required any more. The author of a build-process can pick and choose pre-defined activities to execute a very specific build-step, e.g. compile source code, generate a web deploy package or sign files.

At the end every build step is nothing more then a command line tool, that is executed in the context of the current build. Because of that, also the Command-Line-Activity is one of the most important actions in the step-based build system when it comes to more complex workflows.

Cross-Product-Groups/Departments will love the possibility to create new custom activities, which can be used from other teams and groups within the company. This makes complex build tasks really easy to use for other teams. There is an important idea behind the new build system: “Let developers create the build, get rid of your dedicated build team.” In an Agile world we are talking so much about cross functional teams, that build management should become part of it.

Advanced features you should know about

Microsoft also delivers a couple of features in the new step-based build system to target more complex scenarios.

Multiple Build Agents on the same machine

There are no reasons for using just a single machine per registered build agent. If you would like to get the most out of your existing infrastructure, just install the Microsoft build agent multiple times on your build servers.

Tip: We are using 3 build agents on the same virtual machine which gives us a good utilisation regarding CPU and RAM. On Linux we switched to machines backed via SSD because of I/O intensive docker and Java builds

Release Processes vs. Build Processes

The new release process feature in VSTS is a great option to differentiate between building a pice of software and delivering a piece of software. Especially when the vendor works with a test chain it’s possible to move SaaS software through different states. Especially codeship becomes popular with this kind of feature set.

Push to HockeyApp

Microsoft acquired HockeyApp, a leading vendor in crash dump collection and beta app deployments for Android, iOS and Windows.

Related to this acquisition the Visual Studio Team Services team offers a seamless integration, which allows to upload your build directly to HockeyApp. This works on all required platforms, also on XCode builds for iOS or OS X. At Microsoft Build 2016 the HockeyApp team announces that Xamarin Insights becomes part of HockeyApp. Related to this check our the technical presentation of the HockeyApp team here.

Summary

With the new step-based build system Microsoft introduced in Visual Studio Team Services, a more flexible and simple-to-use option. Especially when you start with new projects, just use the step-based build system and do not invest in the old XAML builds. Software vendors who are writing software for different platforms can now use VSTS as a cloud based management tool, which means to get the most out of your MSDN benefits. Also when you are owning XAML based builds, invest in the migration. Microsoft recently announced that the XAML builds will be discontinued after 2016, so it’s time act.

Review: GitHub vs. Visual Studio Team Services (VSTS) – Should you switch?

January 23, 2016January 23, 2016 de Reviews azure, cloud, coding, git, microsoft, SaaS

Disclaimer: This article is about Visual Studio Team Services (VSTS) and GitHub. The author has a very positive opinion about both services and does not get any offers from Microsoft or GitHub writing this article. The whole article is written IMHO.

Over the last years GitHub has become a success factor for projects related to git repositories for me personally. I also tested several other solutions but mainly I struggeled with the performance or the usability. About 2 years ago I started using Visual Studio Team Services (VSTS) with a work related project. The service looked promising but had a lot of early release issues, so in the last 24 months it was interesting to see how a promising solution became adult. Last week I decided to move away from GitHub for all my closed source projects and rely on Visual Studio Team Service (VSTS). This article gives a deeper look on the main reasons of why I did the move and may help deciders to get detailed information before implementing:

Git-Repository sprawl
Nowadays thanks to bower, npm, bundler or NuGet the amount of Git repositories are growing steadily. When you are focused on component based software development Git is a great helper but the amount of repositories is sprawling because very often every component has its own live in a separated Git repository, which means you have a separate release cycle and a different versioning compared to your main project.

I guess this is the secret source of component managers which are working closely together with git repositories. Of course VSTS and GitHub are supporting multiple git repositories but GitHub lets you pay on a package on git repositories, VSTS lets you pay for users. Especially for small teams paying for users is the better deal, compared to paying for repositories. At the end Microsoft offers a smaller groups of 5 everything for free. This fact stopped my permanent GitHub problem: Having too less private repository space.

GitHub:
O – Allows to have as many GitHub repositories as needed
+ – Generates releases out of tags automatically (good semver integration)
– – Charging base is the count of private git repositories

VSTS:
O – Allows to have as many GitHub repositories as needed
+ – Comes with unlimited git repositories, plans are user based
– – Repository management is not that intuitive as it is in other solutions

Pull Requests and Forks
Forks and pull requests are the most important features GitHub introduced very early to support community driven development. I would say in the open source world Github is the platform when it comes to forks and pull requests. Currently I would never think about moving open source repositories away from GitHub because of this great feature.

When it comes to closed source projects, forking and pull requests are becoming only important in bigger teams with different products or product lines. We are using this feature strongly in our teams at my company. Smaller startups or teams will not use these workflows often but nevertheless it is a road blocker for me to do the 100% switch to VSTS. I guess it took Microsoft around 12 month to deliver it in a more simple way GitHub is doing this.

GitHub
+ – Cross repository forking
+ – Pull requests incl. discussion thread and a lot of community features

VSTS:
O – Supports pull request on repository level

Agile Process support with EPiCs, features and backlogs
When it comes to bigger teams or more structure, people are having the option that to implement a process is the key. This brings me to the biggest enterprise blocker I see in GitHub, the Issue-Tracking system. Companies who migrated to an agile framework like SCRUM or KANBAN need to have the option to work with features, backlogs or bugs. Microsoft delivers with VSTS a highly customizable and adaptable work item management system. The SCRUM and KANBAN template makes perfect sense for agile teams but also the traditional waterfall model can be implemented (even if I don’t understand why someone should do this.)

GitHub
– – It’s just an Issue Tracker
+ – Has good integration into many cloud services

VSTS:
+ – Offers customisable work item management
+ – Comes with templates for agile team structures

Handle your Test-Cases
Even if your projects comes with a great code coverage and good unit tests, the requirements to execute manual tests or just to orchestrate automated integration tests exists. VSTS implements based on the work item management a test case management which has the option to integrate it with automation bots via WinRM and other protocols.

The ability to document test cases and write specific step by step guides how to verify a feature is a big plus esp. in small teams where no dedicated QA resources are hired.

GitHub
O – Ability to integrate with external QA services
– – No integrated test case management

VSTS:
O – Ability to integrate with external QA services
+ – Test Case management is part of the work item management

Centralised Source Code management as migration path
For a couple month Microsoft offers virtual TFS collection, which allows companies that want to go pave the road for git based repositories to stay with the existing centralised source code management besides new git repositories. In the early day VSTS just supported a TFS collection per project space but now maintaining TFS collection is that easy as it is creating new git repositories. This will become very important features for existing TFS customers.

Revised Build-System incl. Linux Support
I guess Microsoft learned very fast that the XAML file based build system was very inflexible and complicated for a SaaS service like VSTS. Because of that a couple month ago a new step based build system was introduced which will orchestrate the build agent out of VSTS

Since Microsoft supports Windows, Linux and Mac build agents there are no road blockers anymore, besides that the hosted build services for VSTS are very rare. There is a build server Microsoft offers out of the box but from my experience you will get more problems customizing that. When you are focused on Azure check also what the Azure App Services can do for you because KuduSync offers out of the box build for your .NET application during deployment.

GitHub
+ – Many different build services available with GitHub integration (CodeShip, T..)
– – Build definitions are not part of the code project
– – No release management to aggregate several projects

VSTS:
– – Hosted build services for VSTS rare
+ – Build definitions are part of the code project
+ – Release Management allows to aggregate several sub projects

Other services and options:
There are also other options and products on the market and I guess one of the most popular one is Assembla which is pretty comparable to Visual Studio Team Services. Also the products from Atlassian (Jira, BitBucket) are great options supporting your development cycle. This article had the intention to support companies who are dealing with GitHub and / or On Premise TFS and now are thinking about combining the positive of both.

I personally think GitHub could become a great option for enterprises as well, when the Issue-Tracker problem is solved which was mainly the reason why I searched for an alternative!

azure-queue-client: delaying jobs made easy

January 6, 2016 de Misc coding, javascript, node

Microsoft Azure offers a very powerful and cheap queueing system, based on Azure Storage. The node module azure-queue-client is a powerful component for node developers in order to interact with the Azure queues easily.

The updated version of the azure-queue-client now supports delayed jobs. This makes it possible to easily delay a running job in the queue worker for a specific time, .e.g. 5 minutes, 1 hour or any other time less than 7 days in the future.

// config with your settings
var qName = '<<YOURQUEUENAME>>';
var qStorageAccount = '<<YOURACCOUNTNAME>>';
var qStorageSecret = '<<YOURACCOUNTSECRET>>';
var qPolling = 2;

// load the module
var azureQueueClient = new require('../lib/azure-queue-client.js');

// create the listener
var queueListener = new azureQueueClient.AzureQueueListener();

// establish a message handler
queueListener.onMessage(function(message) {

// just logging
 console.log('Message received: ' + JSON.stringify(message));
 console.log('Message Date: ' + new Date());

// generate the delay policy
 var exponentialRetryPolicy = new azureQueueClient.AzureQueueDelayedJobPolicies.ExponentialDelayPolicy(1, 5);

// delay the job
 console.log("Job was delayed " + exponentialRetryPolicy.count(message) + " times");
 console.log("Delaying the job by " + exponentialRetryPolicy.nextTimeout(message) + " seconds");
 return queueListener.delay(message, exponentialRetryPolicy);
});

// start the listening
queueListener.listen(qName, qStorageAccount, qStorageSecret, qPolling, null);

As the code sample shows, the module relies on the concept of delay policies. Implementing custom policies is allowed and supported. Built-in policies are the exponential delay policy and the static delay policy.

The module is actively used and maintained in the azure costs service, so it can be used in production. If you would like to contribute or get more detailed information, please visit the github project page.

Big Data in your browser: Parallel.js

September 7, 2015September 7, 2015 de Misc cloud, coding, javascript, SaaS

Big Data often has something todo with analysing a big amount of data. The nature of this data makes it possible to split it up into smaller parts and let them be processed from many distributed nodes. Inspired from the team of CrowdProcess we like the idea to use the computing power of a growing web browser grid to solve data analytic problems.

The Azure Cost Monitor does not have the requirement to solve big data problems of user A in the browser of user B, we would never do this because of data privacy but we have a lot of statistic jobs which need to be processed. From an architecture perspective the question comes up why not to use a growing amount of browser based compute nodes connected with our system instead? Starting with this idea we identified that WebWorkers in modern browsers are acting like small and primitive compute nodes in big data networks. The team from the SETI@Home project also gave us the hint that this option works very well to solve big data challenges.

A very simple picture was painted very fast on the board to illustrate our requirements. The user should not be disturbed from the pre-calculation of statistic data in his browser and the whole solution should prevent battery drain and unwanted fan activities:

It’s also important to understand that some smaller devices like a RaspberryPI which is used for internet browsing or an older smartphone is not able to process the job in time to generate a great user experience. Because of this, the picture changed a bit and we invented a principal we call “Preemptive Task Offloading”.

“Preemptive Task Offloading” lives from the idea that the server and the browser are using the same programming language and the same threading subsystem to manage tasks. Because of that the service itself can decide whether it moves tasks in the browser on the end user or pre-calculates them on the server to ensure great user experience.

The illustrated solution is able to improve the user experience for your end users dramatically and lowers the hosting costs for SaaS applications in the same time.

How it works

The first step is to find the lowest common denominator, in our case it’s called JavaScript. Javascript can be executed in all modern browsers and in the server via node.js. Besides this node and web browser has concepts, e.g. WebWorkers to handle multi threading and multi tasking. The second important ingredient is a framework which abstracts the technical handling of threads or tasks because they are working different in the backend or frontend. We identified parallel.js as a great solution for this because it gives us a common interface to the world of parallel tasks in frontend and backend technologies. Last but not least a system needs to identify the capabilities of the browser. For this we are using two main approaches. The first one tries to identify the capability to spin of web workers and identifies the amount of CPUs. For this we are using the CPU Core Estimator to also support older browsers. The second step of capability negotiation is a small fibonacci calculation to identify how fast the browser really is. If we come to a positive result our system starts the task offloading into the web browser, a negative result leads to a small call against our API to get the preprocessed information from our servers.

Conclusion

After testing this idea several weeks, I can say that this approach helps a lot to build high performance applications, with acceptable costs on the server side. Personally I don’t like the approach to give customer sensitive data into the browser of other customers to much, but I think this approach works great in scientific projects. What do you think about big data approaches in the browser? What are your pitfall or challenges in this area? Just leave a comment bellow or push a message on Twitter.

Azure Active Directory: Verify issued JWT in node.js

June 13, 2015June 13, 2015 de Tutorials aad, azure, coding, node

Microsoft Azure Active Directory is a steady growing identity- and access-management platform which can be used from developers to swap out user management, authentication and authorisation. Azure Active Directory offers several end points and authentication protocols e.g. SAML2, WS-FED or oAuth2. A widely adopted protocol is oAuth2 which ends up with an issued JWT token. This article describes how the JWT token issued by Azure Active Directory can be verified in a node.js application.

Anatomy of a JWT
A JWT token is a non-encrypted digitally signed JSON payload which contains different attributes (claims) to identify the user.

The header is very static and should be used to identify which algorithm was used for the digital signing. This signing algorithm needs to be used to verify the digital signature in the node.js application later on. The payload contains the JSON object with all the claims and information which can be used to verify the user. Trusting this content is only possible when the digital signature of the token is valid and some standard claims, e.g. the issuer or the audience are verified. Otherwise it could be that someone else generated a JWT (man in the middle attack) to get unauthorised access to your application. The signature is the last part of the JWT and needs to be used for verification of the payload. This signature was generated with the algorithm described in the header to prevent unauthorised access.

How AAD issues a token
Azure Active Directory offers every developer the possibility to create applications. If this application is a multi tenant application, other active directory administrators are able to install this application into their directory. At the end of the day an Azure Active Directory application can live in many tenants. Every tenant in the AAD ecosystem has an own set of keys and certificates which are used to sign cryptographic messages. This means that when a directory with the Id “DIRAAA” issues a token for an application the issuer would be

https://sts.windows.net/DIRAAA/

If a directory with the Id “DIRBBBB” issues a token for the same application the issuer would be

https://sts.windows.net/DIRBBB/

So the node.js application needs to verify if the token was issued from the directory we expect. Another side effect of this is, that Azure Active Directory uses different keys for every tenant to issue tokens. This means that the validation code needs to get the right verification key for the token. Microsoft uses RS256 for JWTs issued via oAuth2, so the right certificate needs to be downloaded from somewhere.

Download the right certificates
Microsoft publishes the certificates (public portion of the signing keys) as part of the well known OpenId configuration. It can be downloaded here:

https://login.windows.net/<<tenantid>>/.well-known/openid-configuration

The result is a JSON payload which contains the jwks_uri that should be used to download the certificates. Behind the URI several certificates are available and we currently don’t know which the right one is. The simplest way would be to do a little brute force and verify the JWT against every certificate.

Verification Strategy
Inventing code which is able to verify any AAD issued JWT, without knowing if the application is a multi-tenant or single tenant application is the goal. The following process describes a possible algorithm which can be implemented with existing JWT libraries very easily:

Decode the token to extract the tenant-id because the tenant-id is part of the payload, stored as tid-claim. (!!! Currently we don’t know if we can trust this information !!!)
Download the signing certificates from the well known openid configuration endpoint Microsoft provides. The end point url can be generated with the help of the tenant-id.
Verify the JWT with RS256 against the downloaded certificates. For this, every existing JWT module can be used.
After the token is validated check if the iss-claim contains the same value we expect from the tenantid.

After this process the system verifies the token and we know that this token was issued by Azure Active Directory for the described tenant. This means we are now able to rely on this information.

Node.js integration
All described steps are implemented in a small node package which allows to verify a given token as long as the node application has internet access and can download the certificates. The component can be installed via:

npm install azure-ad-jwt –save

A basic example to verify a given token could look like this:

	var aad = require('azure-ad-jwt');

	var jwtToken = '<<yourtoken>>';

	aad.verify(jwtToken, null, function(err, result) {
	if (result) {
	console.log("JWT is valid");
	} else {
	console.log("JWT is invalid: " + err);
	}
	});

view raw

verify-aad-token.js

hosted with ❤ by GitHub

The component is currently not intended to be an express middleware but it’s easy to extend it that way. A good starting point is the express-jwt middleware which should be used as starting point. The current implementation does not work with certificate caching, so when your system has a huge amount of verification requests it makes no sense to download the certificates during every request. This can be done once when the application starts or in a small cache implementation which invalidates the certificate when it was expired as well.

I hope this helps everybody in the node.js space to integrate Azure Active Directory very fast and easily. The described component is used from the Azure Cost Monitor in the production environment so feel free to integrate the package also in your real world applications.

So when you have any questions, feel free and leave a message on this blog.

Azure App Services: Restart your node-WebJobs during GitDeploy

May 20, 2015 de Tutorials azure, coding, node, SaaS

With Azure App Services (aka. Azure WebSites), the Microsoft Azure cloud offers a great, highly scalable and simple way to host cloud and SaaS services. Besides ASP.NET, several other platforms and languages are supported, e.g. node.js, Python or Java. I personally prefer hosting services written in node.js on this nice managed service of Microsoft.

A common problem for web-services are background jobs like e.g. sending out e-mails or calculating some sales numbers once a day. This use-case can be addressed with Azure WebJobs which are running on the same instance as the web service itself. Jamie Espinosa described the behaviour of WebJobs on an Azure Friday very well. Azure Friday is BTW hosting a whole series about Azure WebJobs, so check it out to get more information.

Normally when deploying a web service into the Azure WebSite the associated WebJobs will be restarted out of the box. A special thing of node.js based Azure WebJobs is that only when the run.js file is changed the WebJob will be restarted. This means when the system just changes an other module or updates the npm dependencies no restart will be enforced.

The whole deployment is based on the Kudu-Project and this project offers so called Post-Deployment-Action-Hooks to trigger a simple script right after the successful deployment of the sources. When ever the run.js file becomes touched the system just restarts the web service, so the solution for this deployment issue was to write a short batch which touches all run.js files:

@echo off

echo Restarting all WebJobs
for /R ..\wwwroot\App_Data\jobs %%G IN (*run.js) DO echo Touching %%G
for /R ..\wwwroot\App_Data\jobs %%G IN (*run.js) DO touch %%G

exit 0

This script can be registered as Post-Deployment-Action-Hook via FTP at every Azure WebSite. Just copy the file to the following location:

This works fine but after all there is still one piece missing: How to get the deployment hooks deployed with git themselves? There are several options to reconfigure the deployment hook directory but I was not able to figure this out. So when you have an idea, feel free and leave a message to discuss any options.

Thoughts about making software

a blog of Dirk Eisenberg

coding