Git Deployment – Shallow Clone Support in Azure App Services – The missing piece

Azure App Services and the open source project KuduSync behind this great Azure Service is a huge time saver for agile teams. Especially DevOps teams will like the continuous deployment features.  Personally I focus a lot on the Git based deployment which enables you to roll back and forward in seconds whenever it is required. Beside that, it is possible to work with standard tools available on market to implement continuous deployment or integration.

Deployments - Microsoft Azure 2017-07-18 06-48-11

When I started working with Azure App Services building Node.js apps, I wrote a little node package called Azure Deploy. It allowed me to push changes as part of a build process directly into the Azure App Service. Originally, CodeShip was the service of choice for the build process but since I need to support Git Repositories beside GitHub, BitBucket and GitLabs, I migrated to Visual Studio Team Services (VSTS) and the integrated build platform.

vso-build-tasks

After several months and hundreds of deploys, which means hundreds of commits to the local git repository, it became a fairly complex and fat thing. This is normally not a problem but my Azure Deploy package clones the local git repository from Azure App Service to a temp directory and copies the build output over it. Last but not least it commits and pushes the changes back to Azure. The big repository took more than 4 minutes to clone so I was wondering if I can use Shallow Clone to get only the latest state of the repository.

This idea works well on Unix based git servers, on GitHub or even in Visual Studio Team Services as well. But when you try to clone a local Git Repository of Azure App Services via Shallow Clone option

git clone --depth 1 https://github.com/jquery/jquery.git jquery

it ends up with an error. The error and its background is also documented in the GitHub project of KuduSync here. So what to do now?

Another nice option of Azure App Services is the option to pull changes from a Git Repository instantly after a commit. This works well in VSTS, based on GitHooks but also with GitHub and a couple other platforms. It’s also possible to clone via shallow clone flag from these repositories which closes the loop. The final solution is to commit into a VSTS or GitHub hosted publishing repository which triggers a pull deployment in Azure App Services.

At the end this change reduced the whole deployment time from 5 up to 9 minutes, down to approx. 90 seconds. You can find the updated Azure Deploy component in the NPM registry here.

Advertisements

Notomia for software licenses

Check out the future of Software Asset Management for SaaS, Web and Legacy applications…

MyWorkspace by Matrix42

Notomia – that was Leonardo da Vinci’s synonym for anatomy. In 1515, when a Papal decree prohibited the dissection of humans, he ignored this decree and dissected 30 men and women who had been executed, in order to study the structure and functioning of the human body. Neither the decree nor the significant risk of infection through dead bodies kept him from doing what he wanted to do, and his success justified the risks he had taken: he discovered arteriosclerosis and was probably the first among mankind to create drawings of a child in the mother’s womb. Even today, his 200 or so anatomic drawings are precious works of arts, combined with scientific precision and detail. And what applied to Leonardo da Vinci, is well worth to be applied to software license management: the more precise and accurate knowledge is supplied to license managers on the company’s license landscape, the…

View original post 702 more words

Tips on How to Set up the Cloud Workspace

Start right now with Hybrid & Cloud Workspace Management with MyWorkspace.

MyWorkspace by Matrix42

Sir Winston Churchill once said: “We shape our buildings; thereafter they shape us.” This is also true or even especially true for the workspace; in the age of the mobile workforce the focus is more on the design of the personal cloud-based workspace, rather than on spatial concepts. After all, high workforce productivity and satisfaction are directly related to the functionality, efficiency and security of the work environment. Similar to the ancient Greek temple builders, MyWorkspace relies on the tested and proven pillar principle.

churchill

No company that moves into the cloud will just throw away their equipment to set up completely new cloud workspaces. Most companies take their existing “furniture”, i.e., existing applications along and add some cloud “furnishings” to set up hybrid clouds where multiple applications are connected and data are exchanged automatically. Independent from the details of the workspace set-up, it is the following three pillars that form the foundation of the cloud workspace:

1) Identity & Access…

View original post 352 more words

Smooth Workspace for Everybody: Our Recipe…

Enjoy the upcoming summer with our recipe for a smooth workspace!

MyWorkspace by Matrix42

For their fans, smoothies are their daily portion of health and fitness and even make them happy, because they can be made without specific cooking skills and combine the properties and benefits of their ingredients into a balanced and effective mixture. What are the ingredients of a workspace solution that leads to happy end users, a happy IT department and a happy company alike?

smoothie

In many organizations, the happiness and fitness or productivity of end users, the IT department and the entire company are rather clouded, since work styles have undergone radical changes. While in the past administrators provisioned the required applications for their users on an in-house server within the company’s own data center, today’s provisioning and sourcing models are increasingly based on web or SaaS applications. Companies use at least ten and very often many more external SaaS applications, which results in a high degree of complexity for both…

View original post 310 more words

Crankdesk on vcnrw

Great event, besides all other sessions check out session 7 if you would like to learn more about identity and access management 😉

Crankdesk...

Today i enjoy a day on the vcnrw community Event, managed by Timm and Helge. I am proud that i can say, that Matrix42 is sponsoring this great community with round about 100 people. This community is  about virtualization, datacenter and data Management.

Within the following blog entry, i will document the day and the session of room 1.

WP_20160420_08_32_39_Rich

Have fun…

View original post 2,716 more words

azure-queue-client: delaying jobs made easy

Microsoft Azure offers a very powerful and cheap queueing system, based on Azure Storage. The node module azure-queue-client is a powerful component for node developers in order to interact with the Azure queues easily.    

The updated version of the azure-queue-client now supports delayed jobs. This makes it possible to easily delay a running job in the queue worker for a specific time, .e.g. 5 minutes, 1 hour or any other time less than 7 days in the future.

// config with your settings
var qName = '<<YOURQUEUENAME>>';
var qStorageAccount = '<<YOURACCOUNTNAME>>';
var qStorageSecret = '<<YOURACCOUNTSECRET>>';
var qPolling = 2;
// load the module
var azureQueueClient = new require('../lib/azure-queue-client.js');
// create the listener
var queueListener = new azureQueueClient.AzureQueueListener();
// establish a message handler
queueListener.onMessage(function(message) {
// just logging
 console.log('Message received: ' + JSON.stringify(message));
 console.log('Message Date: ' + new Date());
// generate the delay policy
 var exponentialRetryPolicy = new azureQueueClient.AzureQueueDelayedJobPolicies.ExponentialDelayPolicy(1, 5);
// delay the job
 console.log("Job was delayed " + exponentialRetryPolicy.count(message) + " times");
 console.log("Delaying the job by " + exponentialRetryPolicy.nextTimeout(message) + " seconds");
 return queueListener.delay(message, exponentialRetryPolicy);
});
// start the listening
queueListener.listen(qName, qStorageAccount, qStorageSecret, qPolling, null);

As the code sample shows, the module relies on the concept of delay policies. Implementing custom policies is allowed and supported. Built-in policies are the exponential delay policy and the static delay policy.

The module is actively used and maintained in the azure costs service, so it can be used in production. If you would like to contribute or get more detailed information, please visit the github project page.

Big Data in your browser: Parallel.js

Big Data often has something todo with analysing a big amount of data. The nature of this data makes it possible to split it up into smaller parts and let them be processed from many distributed nodes. Inspired from the team of CrowdProcess we like the idea to use the computing power of a growing web browser grid to solve data analytic problems.

The Azure Cost Monitor does not have the requirement to solve big data problems of user A in the browser of user B, we would never do this because of data privacy but we have a lot of statistic jobs which need to be processed. From an architecture perspective the question comes up why not to use a growing amount of browser based compute nodes connected with our system instead? Starting with this idea we identified that WebWorkers in modern browsers are acting like small and primitive compute nodes in big data networks. The team from the SETI@Home project also gave us the hint that this option works very well to solve big data challenges.

A very simple picture was painted very fast on the board to illustrate our requirements. The user should not be disturbed from the pre-calculation of statistic data in his browser and the whole solution should prevent battery drain and unwanted fan activities:

ParallelJS-Pic01

It’s also important to understand that some smaller devices like a RaspberryPI which is used for internet browsing or an older smartphone is not able to process the job in time to generate a great user experience. Because of this, the picture changed a bit and we invented a principal we call “Preemptive Task Offloading”.

ParallelJS-Pic02

“Preemptive Task Offloading” lives from the idea that the server and the browser are using the same programming language and the same threading subsystem to manage tasks. Because of that the service itself can decide whether it moves tasks in the browser on the end user or pre-calculates them on the server to ensure great user experience.

ParallelJS-Pic03

The illustrated solution is able to improve the user experience for your end users dramatically and lowers the hosting costs for SaaS applications in the same time.


How it works

The first step is to find the lowest common denominator, in our case it’s called JavaScript. Javascript can be executed in all modern browsers and in the server via node.js. Besides this node and web browser has concepts, e.g. WebWorkers to handle multi threading and multi tasking. The second important ingredient is a framework which abstracts the technical handling of  threads or tasks because they are working different in the backend or frontend. We identified parallel.js as a great solution for this because it gives us a common interface to the world of parallel tasks in frontend and backend technologies. Last but not least a system needs to identify the capabilities of the browser. For this we are using two main approaches. The first one tries to identify the capability to spin of web workers and identifies the amount of CPUs. For this we are using the CPU Core Estimator to also support older browsers. The second step of capability negotiation is a small fibonacci calculation to identify how fast the browser really is. If we come to a positive result our system starts the task offloading into the web browser, a negative result leads to a small call against our API to get the preprocessed information from our servers.


Conclusion

After testing this idea several weeks, I can say that this approach helps a lot to build high performance applications, with acceptable costs on the server side. Personally I don’t like the approach to give customer sensitive data into the browser of other customers to much, but I think this approach works great in scientific projects. What do you think about big data approaches in the browser? What are your pitfall or challenges in this area? Just leave a comment bellow or push a message on Twitter.