Build your own Twitter – Part 1 – A timeline service with Azure Table Store

de Tutorials , ,

The Azure Cost Monitor and many other cloud services I’m working on, needed a timeline service to present aggregated events and actions similar to an audit trail. When I was thinking about this, it came to my mind that the requirements I had were very similar to a Twitter feed but with less features.

Let’s recap what is needed to build a timeline service in general:

  1. The Timeline
    The Timeline is associated with a subject and contains tons of events happened and triggered by other subjects the timeline owner follows.
  2. The Event
    The Event is the incident that happens in the real world and is stored in different timelines. This means every follower of a subject will get the event in his own timeline. So a single event can be stored in many different timelines.
  3. The Subject
    The Subject is someone or something which triggers an event. Normally it is a natural person with an own timeline but it could also be a piece of hard/software.
  4. The Target(s)
    The Targets are subjects with a timeline who are following other subjects. This means every subject becomes a follower or target as soon as it is following an other subject. Posting an event then means sending the message to the timeline of every subject following the sender.

The following picture should give a good overview about the entities in the timeline service:

Screen Shot 2015-03-21 at 10.40.15

Starting with this definitions in mind it’s possible to identify components needed for building a timeline service:

  • Storage for the timeline 
    The storage for the timeline needs to be able to offer a very fast read performance and an OK write performance. The performance should especially not be dependent from the amount of data in the system. I chose Azure Table Store for this because I can store the information in different partitions and get a clear read performance SLA for every partition. In addition to this it’s totally cheap and payable – also for startups.
  • Access broker to the storage 
    Normally NoSQL storage which can be used for timeline access needs a little helper to manage access. In general a RESTful web-service acts as an access broker and hands out pre-signed links to the timeline content. This ensures that no timeline data needs to go through the timeline service at all. Only the pre-authorised access links will be generated for the system. This also means that at the end the client SDK needs to handle the raw data for the timeline. This makes it a bit more complicated but it lets the performance stay in a good range. An other important operation needs to be implemented in the Access broker as well: Posting events to the subjects followers is a slow and complex operation the system needs to implement with a direct server call normally executed asynchronous with the help of a worker job.
  • Metadata storage for subject and targets
    Last but not least the combination of subject & targets needs to be stored somewhere in the backend. Azure Table Store can be used for this as well. All the operations to create a follow-ship can be implemented in the timeline service as well but not the rights & permission checks to post or create relations because the timeline service should be used machine-2-machine with API tokens.

The following graphic shows the technical architecture of a good scalable timeline service based on Azure services:

arch-timeline

The next part of this tutorial will focus on the correct and scalable table structure based on Microsoft Azure Table Storage.

Azure Cost Monitor sends notifications when Azure EA token has expired

de Azure Cost Monitor , , , ,

Microsoft offers all EA customers the Azure EA portal to manage different accounts and subscriptions. In addition to this portal, every customer can issue up to two security tokens to 3rd party applications. This allows the applications to interact and work with the customers EA data like the Azure Cost Monitor is doing.

Starting this week the Azure Cost Monitor analyses the existing Azure EA security tokens and is able to detect when a token has expired. As soon an expired token is detected the owner of the contract will get an information via mail:

blog-azure-cost-ea-token-expired

In addition, the Azure Cost Monitor is now able to warn about tokens that will expire very soon. The owner of the affected contract will get an information via mail as well:

blog-azure-cost-ea-token-will-expire

This new feature should help you as a contract owner to stay in sync every day. It also helps us to have our system stay healthy and performant every day. Please update the tokens directly when you get this kind of notifications. Additional information on how to renew a token can be found in our knowledge base here.

We hope this feature makes it simple to maintain your Azure Cost Monitor account and makes it much easier for you to manage and control all costs. Any questions, wishes or ideas? Try our feedback portal or drop a mail to tickets@azurecostmonitor.uservoice.com.

Azure Table Store: How to backup safely

de Misc

Microsoft Azure Table store is an amazing, simple, cheap and powerful service of the Microsoft Azure cloud. The service is something between a real NoSQL database and a simple KVP-store. I did many projects in the last time where the Azure Table Store was just a fast read cache or also the whole persistency backend.

As soon as the tables in Azure contain important data for the application provided to customers it’s necessary to think about backup. Microsoft guarantees that the data can not be corrupted on the storage (check out my article about the different storage account options) but accidental deletion, data corruption during automated processes or just by mistake can still happen. Where people are working shit sometimes happens, nobody can change this.

Backing up table stores is not that easy as backing up blob storage based on the idea of stamp copies. Every table needs to be replicated into another table or exported to the blob account. In my current project we searched for the perfect solution and finally came up with the following stack of services: We are using the Azure Cloud Backup from RedGate to export all tables in a GEO redundant storage account. With this solution we get a daily backup of our tables in parallel to Azure SQL backups based on the Microsoft built in features. The backups are stored into a GEO redundant storage account which helps to ensure that we have access to this backups even when one datacenter of Microsoft burns 🙂 or just looses power.

This setup in combination with the Microsoft Backup support for Azure SQL is very powerful and gives everybody the good feeling of being able to recover when chaos happens.

ngHelperAirbrake: Airbrake for AngularJS

de ngHelper , , , ,

Airbrake is a well known exception tracker which is used from thousands of users. A cool thing is that the Airbrake team also supports browser based javascript exception. Integrating these kind of javascript code gives AngularJS developers sometime a headache. The newest member of the ngHelper collection, the ngHelperAirbrake component makes it super simple and easy to integrate Airbrake in an existing AngularJS application.

It’s a bower component and works well with scaffolding tools like Yeoman. Installing the component is possible with the following command line:

bower install ng-helper-airbrake –save

After that the component is registered in the bower.json of the project. Moving up the dependency entry to the position right after the inclusion of angular ensures that the Airbrake-Shim is loaded as early as possible when doing a full page reload.

“dependencies”: {
“angular”: “~1.3.8”,
“ng-helper-airbrake”: “~0.1.0”,

ngHelperAirbrake offers the $airbrake angular service which allows to configure the different Airbrake settings. The documentation at our project page describes how to set the right configuration: https://github.com/ngHelper/ngHelperAirbrake

After configuring the project everything works as expected and Airbrake receives exception from the AngularJS application.

Pit-Falls when you integrate Twitter, Facebook or other social networks

de Misc , ,

Today I received a request from a platform to vote for a specific product. I was willing todo this, because I’m using it every day and I’m really happy with that.

With this kind of sentences discussions about failed social media integration start. In a world where many people are very concerned about their own data privacy a “but” follows very fast:

Currently I can’t do that because the website enforces me to authorise via Twitter or Facebook. This would be doable if they only request the rights they really need. I will not allow every system to post in my name because that’s very personal.

At the end of the story you as a service provider loose a customer on the click path for only one reason: You tried to get more rights as you typically need to operate your application. And from my own experience the right to post in the name of someone requires a strong trustworthy relationship.

Screen Shot 2015-03-06 at 07.32.16

Neither Twitter nor Facebook enforces any developer to request more rights than they really need. I did a lot of Twitter integrations and whenever you are doing this please check the FAQ of the identity provider of your choice clearly and follow the principal of minimal privileges.

Following this ideas you will loose less people on click path to your voting system!

Update expired Azure EA tokens easily

de Azure Cost Monitor , , ,

The Microsoft Azure EA portal issues an Azure EA security token for API access to the cost & usage data. This token expires every 6 month and because of that needs to be renewed in the Azure Cost Monitor on a regular basis. The Azure Cost Monitor team started the project about 5 months ago what means that the first Azure EA security tokens will expire soon. Because of that, we are pleased to announce the launch of the token update wizard which makes it as easy as possible to renew expired Azure EA tokens:

azure-ea-token-settings

From today, the dashboard of the Azure Cost Monitor warns every user when the token is expired to ensure that every administrator always stays informed:

azure-ea-token-warning

If you don’t update an expired token, the system stops syncing data from the Azure EA portal. As soon as the token is valid again the system is able to sync data from the Azure EA portal and is doing that automatically within the next sync cycle over night. We will also get the missed data from the days the token was expired. So no worries when you were in vacation or absent for a couple days.

We hope this feature makes it simple to maintain your Azure Cost Monitor account and makes it much easier for you to manage and control all costs. Any questions, wishes or ideas? Try our feedback portal or drop a mail to tickets@azurecostmonitor.uservoice.com.

Azure: Do never use EF automigrations when working in teams

de Misc , , ,

Todays number one approach in building database access libraries on ASP.NET based applications is the Entity Framework. Since Microsoft supports migration based database updates, which mostly look like copied from the approach Ruby on Rails offers since ages, it’s possible to use them in scenarios where continuous deployment is key.

The core model which fits perfectly into a development process around continuous delivery & deployment is a code-first approach. This means that the developer creates so called POCOs (Plain Old C# Object) and the system is able to generate the needed SQL changes, based on the current state of the database.

The Entity Framework has two options to generate the migration:

  1. Auto Migrations
    The Entity Framework tries to generate the database changes on runtime without any specific migrations implemented as code. Everything happens magically 😦
  2. Explicit Migrations
    The Entity Framework just runs the migration scripts written in a .NET based special DSL for database operations. Nothing happens magically 🙂 This needs the developers brain but is controllable even in large teams.

When you work in a team Auto Migrations are really a bad idea and should be disabled from the beginning. Assume developers do changes manually in the database which means the magic around AutoMigrations will generate a different set on instructions against this database as the next developer will get. The results of auto migrations in team environments are not reliable and repeatable.

So it’s a really good idea to disable automatic migrations from the start:

public Configuration()  {
AutomaticMigrationsEnabled = false;
}

Auto Migrations do not have a win at all when it comes to projects with more than one developer or projects which are deployed automatically. Continuous deployment relies on a strict unit of work and as less magic as possible. This will also help to troubleshoot when the continuous deployment breaks the production and a rollback is required.

Microsoft also published a nice article with more information at the MSDN: http://msdn.microsoft.com/en-US/data/jj554735.aspx

Update: Deploy AngularJS-Apps to Azure WebSites with Codeship

de Tutorials , , , ,

A couple of weeks ago I wrote a tutorial on how to deploy an Angular.js application into Azure WebSites. I explained why the simple GitHub deployment that comes with Azure is not usable and how Codeship as shipping service can do the magic for every team.

Since this article I started using Codeship more extensive and since it’s a part of my daily business I don’t want to miss it at all. Here @Matrix42 HackWeek, Codeship was the most chosen solution to implement AdHoc continuous deployment. I would not be suprised when we replace our traditional TFS build agents in the next months 🙂

To make it more intuitive and easy for everybody who works with node and a javascript task runner, I decided to transform the illustrated deployment script into a node module. The Azure-Deploy module is super simple to integrate in existing NPM driven projects and can be added to existing javascript tasks as well. At the end the system offers a simplified usage like this:

grunt deploy:production

This result is more simple to integrate into the deployment scripts of Codeship than everything else. The shell script of course works but this component gives you the freedom to stay with the current task runner of your choice. To get more information visit the github page for this project: https://github.com/dei79/node-azure-deploy

Manage costs in your local currency – Azure Cost Monitor supports multiple currencies

de Azure Cost Monitor , , ,

The azure cost monitor team is pleased to announce the launch of the multi currency support. This  feature has been built to reflect what our users told us they need:

acm_currency

Currently the Microsoft Azure EA portal does not give any information about the currency of the financial data. Because of that the azure cost monitor displayed all costs in EUR in the past. Now the new multi currency support allows to switch between different currencies.

We hope this feature brings more transparency in your Azure Cloud Spendings and makes it much easier for you to manage and control all costs. Any questions, wishes or ideas? Try our feedback portal or drop a mail to tickets@azurecostmonitor.uservoice.com.

Azure Storage: Is the Geo Redundant Mode really required?

de Misc , ,

Microsoft Azure offers different replication modes for Azure Storage. Every mode approximately doubles the costs for a TB of data. During a great workshop with Patrick Heyde we talked about stamp copies and I asked myself which mode I really needed.

First I took one step back, to identify all requirements I typically have in my projects for a highly scalable, fault safe, redundant storage :

  • When a hard-drive in a storage server brakes, my data still needs to be usable.
  • When Microsoft has huge power outage on a whole datacenter, I want to bring my app up and running again in another datacenter.
  • When I (or my customers) are removing data by accident, there needs to be an option to revert to a former snapshot.

So a review of the different replication modes compared to these requirements leads me to the following results:

When a hard-drive in a storage server brakes, my data still needs to be usable:
The Local Redundant Storage fulfils this requirement perfectly. Microsoft writes 3 different copies of every bit within one single data center. When a hard drive on a stamp or the whole stamp goes down, another one can take over and all data is available without any interruption.

When Microsoft has huge power outage on a whole datacenter, I want to bring my app up and running again in another datacenter:
Microsoft offers a geographical redundant storage mode that stores another 3 copies in another datacenter a hundred miles away. This helps a lot, because every application can use the secondary location to access to the data – but is it worth the price? The price for GRS is three times higher then for LRS.
An automated replication between two different LRS storages, hosted in a Azure WebJob might be a good solution the fulfill the requirement as well.

When I (or my customers) are removing data by accident, there needs to be an option to revert to a former snapshot:
Globally Redundant Storage is not helpful when it comes to removing data by accident. As soon as the data is removed from the primary storage, the system removes the data in the backup location as well, often within seconds.
But this requirement can also be fulfilled with a replication between two different LRS storages, as already described above. The whole application needs to be designed for this use cases.

So this review brings me to the conclusion that in my personal opinion GRS storage is not needed in most of the use cases. Normally several LRS storages and an application logic optimised on the specific data security requirements works well and preserves the budget.

What’s your opinion? Do you have use cases where GRS and Read-GRS are hard requirements? If you like, leave a short comment …