Hey node developers, switch to .NET Core – now!

August 7, 2017July 26, 2017 de Reviews c#, coding, javascript, microsoft, node

Several years ago I started building a bigger project as a Software as a Service application. Beside all the different technical requirements, being able to work directly on my MacBook Pro without starting virtual machines, was a big wish. At this time a tool chain based on Node.js, Express, NPM and WebStorm was available. Over the years building backend services with Node.js, and this means with JavaScript, felt like rapid prototyping. Getting started is very fast and lightweight but when the project grows compile time features like a strong type system are missing. Year over year I did reviews how to get esp. the beauty of C# and the powerful compiler infrastructure of .NET back.

In the last month Microsoft released Visual Studio for Mac and with it the investments into .NET Core increased. Building backend services in Visual Studio for Mac based on .NET Core and ASP.NET Core including the out of the box support in Azure App Services is definitely what I was looking for.

Because of that I decided to give it a spin. After a couple weeks working with the framework I can say it was the right decision because of the following key reasons:

Full managed and type safe environment based on the powerful C# compiler
Broad ecosystem for components similar to NPM via NuGet
Ability to convert runtime errors in compile time errors
Hosting the results on Linux, Docker Containers or just Azure App Services
ASP.NET Core comes with out of the box easy to use Dependency Injection system
ASP.NET Core learned and steeled the best things from the node + express chain
C# Attribute and Extension-Classes are unbeatable to beatify your code

And last but not least everything works well on my MacBook Pro without the need of a single virtual machines – Thanks Microsoft for letting me keep the platform I love!

Git Deployment – Shallow Clone Support in Azure App Services – The missing piece

July 24, 2017July 18, 2017 de Misc codeship, coding, javascript, microsoft, node

Azure App Services and the open source project KuduSync behind this great Azure Service is a huge time saver for agile teams. Especially DevOps teams will like the continuous deployment features. Personally I focus a lot on the Git based deployment which enables you to roll back and forward in seconds whenever it is required. Beside that, it is possible to work with standard tools available on market to implement continuous deployment or integration.

When I started working with Azure App Services building Node.js apps, I wrote a little node package called Azure Deploy. It allowed me to push changes as part of a build process directly into the Azure App Service. Originally, CodeShip was the service of choice for the build process but since I need to support Git Repositories beside GitHub, BitBucket and GitLabs, I migrated to Visual Studio Team Services (VSTS) and the integrated build platform.

After several months and hundreds of deploys, which means hundreds of commits to the local git repository, it became a fairly complex and fat thing. This is normally not a problem but my Azure Deploy package clones the local git repository from Azure App Service to a temp directory and copies the build output over it. Last but not least it commits and pushes the changes back to Azure. The big repository took more than 4 minutes to clone so I was wondering if I can use Shallow Clone to get only the latest state of the repository.

This idea works well on Unix based git servers, on GitHub or even in Visual Studio Team Services as well. But when you try to clone a local Git Repository of Azure App Services via Shallow Clone option

git clone --depth 1 https://github.com/jquery/jquery.git jquery

it ends up with an error. The error and its background is also documented in the GitHub project of KuduSync here. So what to do now?

Another nice option of Azure App Services is the option to pull changes from a Git Repository instantly after a commit. This works well in VSTS, based on GitHooks but also with GitHub and a couple other platforms. It’s also possible to clone via shallow clone flag from these repositories which closes the loop. The final solution is to commit into a VSTS or GitHub hosted publishing repository which triggers a pull deployment in Azure App Services.

At the end this change reduced the whole deployment time from 5 up to 9 minutes, down to approx. 90 seconds. You can find the updated Azure Deploy component in the NPM registry here.

azure-queue-client: delaying jobs made easy

January 6, 2016 de Misc coding, javascript, node

Microsoft Azure offers a very powerful and cheap queueing system, based on Azure Storage. The node module azure-queue-client is a powerful component for node developers in order to interact with the Azure queues easily.

The updated version of the azure-queue-client now supports delayed jobs. This makes it possible to easily delay a running job in the queue worker for a specific time, .e.g. 5 minutes, 1 hour or any other time less than 7 days in the future.

// config with your settings
var qName = '<<YOURQUEUENAME>>';
var qStorageAccount = '<<YOURACCOUNTNAME>>';
var qStorageSecret = '<<YOURACCOUNTSECRET>>';
var qPolling = 2;

// load the module
var azureQueueClient = new require('../lib/azure-queue-client.js');

// create the listener
var queueListener = new azureQueueClient.AzureQueueListener();

// establish a message handler
queueListener.onMessage(function(message) {

// just logging
 console.log('Message received: ' + JSON.stringify(message));
 console.log('Message Date: ' + new Date());

// generate the delay policy
 var exponentialRetryPolicy = new azureQueueClient.AzureQueueDelayedJobPolicies.ExponentialDelayPolicy(1, 5);

// delay the job
 console.log("Job was delayed " + exponentialRetryPolicy.count(message) + " times");
 console.log("Delaying the job by " + exponentialRetryPolicy.nextTimeout(message) + " seconds");
 return queueListener.delay(message, exponentialRetryPolicy);
});

// start the listening
queueListener.listen(qName, qStorageAccount, qStorageSecret, qPolling, null);

As the code sample shows, the module relies on the concept of delay policies. Implementing custom policies is allowed and supported. Built-in policies are the exponential delay policy and the static delay policy.

The module is actively used and maintained in the azure costs service, so it can be used in production. If you would like to contribute or get more detailed information, please visit the github project page.

Big Data in your browser: Parallel.js

September 7, 2015September 7, 2015 de Misc cloud, coding, javascript, SaaS

Big Data often has something todo with analysing a big amount of data. The nature of this data makes it possible to split it up into smaller parts and let them be processed from many distributed nodes. Inspired from the team of CrowdProcess we like the idea to use the computing power of a growing web browser grid to solve data analytic problems.

The Azure Cost Monitor does not have the requirement to solve big data problems of user A in the browser of user B, we would never do this because of data privacy but we have a lot of statistic jobs which need to be processed. From an architecture perspective the question comes up why not to use a growing amount of browser based compute nodes connected with our system instead? Starting with this idea we identified that WebWorkers in modern browsers are acting like small and primitive compute nodes in big data networks. The team from the SETI@Home project also gave us the hint that this option works very well to solve big data challenges.

A very simple picture was painted very fast on the board to illustrate our requirements. The user should not be disturbed from the pre-calculation of statistic data in his browser and the whole solution should prevent battery drain and unwanted fan activities:

It’s also important to understand that some smaller devices like a RaspberryPI which is used for internet browsing or an older smartphone is not able to process the job in time to generate a great user experience. Because of this, the picture changed a bit and we invented a principal we call “Preemptive Task Offloading”.

“Preemptive Task Offloading” lives from the idea that the server and the browser are using the same programming language and the same threading subsystem to manage tasks. Because of that the service itself can decide whether it moves tasks in the browser on the end user or pre-calculates them on the server to ensure great user experience.

The illustrated solution is able to improve the user experience for your end users dramatically and lowers the hosting costs for SaaS applications in the same time.

How it works

The first step is to find the lowest common denominator, in our case it’s called JavaScript. Javascript can be executed in all modern browsers and in the server via node.js. Besides this node and web browser has concepts, e.g. WebWorkers to handle multi threading and multi tasking. The second important ingredient is a framework which abstracts the technical handling of threads or tasks because they are working different in the backend or frontend. We identified parallel.js as a great solution for this because it gives us a common interface to the world of parallel tasks in frontend and backend technologies. Last but not least a system needs to identify the capabilities of the browser. For this we are using two main approaches. The first one tries to identify the capability to spin of web workers and identifies the amount of CPUs. For this we are using the CPU Core Estimator to also support older browsers. The second step of capability negotiation is a small fibonacci calculation to identify how fast the browser really is. If we come to a positive result our system starts the task offloading into the web browser, a negative result leads to a small call against our API to get the preprocessed information from our servers.

Conclusion

After testing this idea several weeks, I can say that this approach helps a lot to build high performance applications, with acceptable costs on the server side. Personally I don’t like the approach to give customer sensitive data into the browser of other customers to much, but I think this approach works great in scientific projects. What do you think about big data approaches in the browser? What are your pitfall or challenges in this area? Just leave a comment bellow or push a message on Twitter.

ngHelper-Toolbar: Now supports secondary actions & dividers

April 9, 2015 de ngHelper angular, angularjs, coding, javascript, nghelper

The $toolbar service is a great helper when it comes to building toolbars in AngularJS applications. The new version 0.0.3 allows you to handle new secondary actions, as shown here in the Azure Cost Monitor application:

The secondary action can be defined in the addItem function similar to all other options the API supports:

$toolbar.addItem(‘childContract, contract, null, null, true, ‘/report/1234’, null, ‘activeContract’, ‘fa-trash’, function () { $scope.removeContract(contract);
});

Making the menu more user-friendly can be achieved by adding dividers in the structure. When using the special menu title “DIVIDER” the system will use this in the menu structure as divider:

$toolbar.addItem(‘user.divider’, ‘DIVIDER’, null, null, true, null, null, ‘user’);

The new navigation infrastructure of the Azure Cost Monitor is using the $toolbar service from the ngHelper-Toolbar project. We hope this feature makes it simple to maintain your toolbars. Any questions, wishes or ideas? Try the issue button on the GitHub page or contact the author via this blog.

ngHelperAirbrake: Airbrake for AngularJS

March 12, 2015March 11, 2015 de ngHelper angular, angularjs, coding, javascript, nghelper

Airbrake is a well known exception tracker which is used from thousands of users. A cool thing is that the Airbrake team also supports browser based javascript exception. Integrating these kind of javascript code gives AngularJS developers sometime a headache. The newest member of the ngHelper collection, the ngHelperAirbrake component makes it super simple and easy to integrate Airbrake in an existing AngularJS application.

It’s a bower component and works well with scaffolding tools like Yeoman. Installing the component is possible with the following command line:

bower install ng-helper-airbrake –save

After that the component is registered in the bower.json of the project. Moving up the dependency entry to the position right after the inclusion of angular ensures that the Airbrake-Shim is loaded as early as possible when doing a full page reload.

“dependencies”: {
“angular”: “~1.3.8”,
“ng-helper-airbrake”: “~0.1.0”,

ngHelperAirbrake offers the $airbrake angular service which allows to configure the different Airbrake settings. The documentation at our project page describes how to set the right configuration: https://github.com/ngHelper/ngHelperAirbrake

After configuring the project everything works as expected and Airbrake receives exception from the AngularJS application.

azure-queue-client: build azure queue based workers in node.js

January 20, 2015January 18, 2015 de Tutorials coding, javascript, node

Microsoft Azure offers a very powerful and cheap queueing system based on Azure Storage. As a node developer the challenge is to build a simple to use system which is able to consume messages from the azure queue. The Azure Cost Monitor is for instance using this module to process all costs analytic tasks in the backend.

The module azure-queue-client is able to implement this in a couple of simple steps. It supports multiple workers in different processes and on different machines. The following example illustrates the usage:

// config with your settings
var qName = ‘<<YOURQUEUENAME>>’;
var qStorageAccount = ‘<<YOURACCOUNTNAME>>’;
var qStorageSecret = ‘<<YOURACCOUNTSECRET>>’;
var qPolling = 2;

// load the modules
var queueListener = require(‘azure-queue-client’).AzureQueueListener;
var q = require(“Q”);

// establish a message handler
queueListener.onMessage(function(message) {
var defer = q.defer();

console.log(‘Message received: ‘ + JSON.stringify(message));
defer.resolve();

return defer.promise;
});

// start the listening
queueListener.listen(qName, qStorageAccount, qStorageSecret, qPolling, null);

Developers who are using the Azure Scheduler might recognize that the payload of the scheduler is encapsulated in a XML wrapper. This XML wrapper can be handled by the module as well so that it doesn’t matter if the message comes from an other queue client or the Azure Scheduler.

This module makes writing job workers in node, hosted on Azure or any other cloud provider a breeze.

Thoughts about making software

a blog of Dirk Eisenberg

javascript