A F# web API in Azure Container

Containerization is one of the optional tenant for micro services.  The .NET community did not get the opportunity for their solution to be on containers apart from the preview version Windows Nano server.  The mono and F# community unveil the container world to .NET community as https://hub.docker.com/r/fsharp/fsharp/.

My intent of this post is to evaluate the feasibility of deploying a F# solution into a production ready cloud based docker containers.  I take the following action items:

  1. Approachable F# framework for exposing an obligatory Web API
  2. Deploy the solution into my local docker host
  3. Deploy the solution into Azure Container Service.

An approachable F# Web API framework

I decide to develop the obligatory web api on non-windows platform.  The intent is to showcase the wide spread  availability of .NET based solution using F# on Windows, Linux and Mac OS X.  The Yeoman generator and WebSharper framework that enable to develop descent web api.  However, Suave.io is an evolving and promising web framework for F#.  The community claimed that as a functional web framework.

I use Xamarin Studio on OS X.  A console app should be created followed by adding Suave library via Packet or Nuget packages.  I add this via nuget package.


I add App.fs file with following content.

open Suave
open Suave.WebPart
open Suave.Successful
open Suave.RequestErrors
open Suave.Operators
open Suave.Filters // for path

let cfg =
{ defaultConfig with
bindings = [ HttpBinding.mk HTTP (System.Net.IPAddress.Parse "") 8083us ] }

// default port: 8803
let catalogSearch =
  request (fun r ->
  match r.queryParam "q" with
  | Choice1Of2 q -> OK (sprintf "You are searching %s" q)
  | Choice2Of2 msg -> BAD_REQUEST msg)

let webPart =
  choose [
    path "/" >=> (OK "Welcome to our online catalog")
    path "/catalog" >=> catalogSearch

startWebServer cfg webPart

startWebServer takes the given configuration and web part, and starts the web server at port 8083.  Web part is the way to map REST resources to appropriate business services.  The web api define a resource “catalog” with following actions:

/ –> for base.  It returns obligator welcome message

/catalog?q –> search term that will be returned.

When you run this app, the web server listens at 8083 and start accepting the request.

If you run

curl http://localhost:8083/catalog?q=fsharp // returns You are searching fsharp

curl http://localhost:8083 // returns Welcome to our online catalog

Solution in my local docker host

I install docker tools (kitematic) for Mac OS X and add official F# docker container fsharp/fsharp.

I create Dockerfile to create custom image that derived from fsharp/fsharp as mentioned below.

FROM fsharp/fsharp:latest
MAINTAINER Sheik Uduman Ali M


CMD ["mono", "CatalogAPI.exe"]

To simplify the build process, I use FAKE tool to build the application into a folder.  This will generate following assets:

  • CatalogAPI.exe
  • FSharp.core.dll
  • Suave.dll

These assets should be deployed into app folder in the container.  I run this container it works as expected.

Deploying the solution into Azure Container

Refer https://azure.microsoft.com/en-in/documentation/articles/container-service-deployment/ for how to create a Docker Swarm cluster and connect to its master node.  I create a standard D2 Azure VM size with 1 master and 1 agent.


The custom docker image should be in accessible location so that I committed the above mentioned docker custom image as udooz/mashup_catalog_api with tag 1 into my docker hub private repository.

Once the Swarm cluster deployed following resources will be created under the specified resource group in Azure.
SSH to the master node (swarm-master-A745DC2-0) and run the following docker command.
docker run \ 
-d -p 8083:8083 
--name mashopretail 

Now either use curl or browser to specify master node IP address (here with appropriate resource pattern to access Web API.  To access the IP address publicly, you should change the settings in master node as shown below.



Note that the MSDN document specifies to use Swarm agent node DNS address to access the application.  However, it is not working and received time out error.

IoT – Intel Galileo Gen 2 Hello World

Intel Galileo Gen 2 is the Arduino based IoT board that enables to interact with sensors and actutators.  The advantage with Galileo is to make ease of writing Arduino’s sketches for reading values from sensors and sending information to LCD display et all as well as you can write typical program using Node.js,  Python and even Mono to make this as IoT agent/propagator/integrator.

SPI flash is the default storage for firmware and it has embedded Linux 3.8 and Arduino libraries.   The extensive part of Linux can be installed on SD card with capacity up to 32 GB.  This post I share my experience on using SPI part of Galileo.  I use OS X as host computer.

Setting up the stage

Following things are needed to run the Hello World sketches I used in this post using SPI part of Galileo:

  • Micro-USB to USB cable
  • 5V Power Card

First of all connect 5V power card followed by connect the Galileo with your computer using Micro-USB to USB cable.  The USB client side should be connected on Micro-USB as shown below.  Once you connected with your host computer, you can check whether it is connected successfully on /dev folder.  In OS X, it will be like cu.usbmodemXXXX where XXXX is the port number as highlighted below.

You should upgrade the firmware using Intel Galileo Firmware Updater application if you are just unboxing Galileo.

Download OS independant version of Intel Galileo Firware Updater instead of OS X specific.  It requires older version of Java.

The firmware update will take few minutes.  Once this is done you can use Arduino IDE to run sketches.

Setting up Arduino IDE is straightforward in OS X.  Arduino 1.6.5 is compatible on Mac OS X 10.7 Lion or newer.  You should check the availability of Intel Galileo board in the IDE by Tools > Board and Tools > Port.  Very first time you might need to install the board using Boards Manager (Tools > Board > Boards Manager).  You should search Arduino Certified type of board where you can see Intel i586 Boards by Intel that includes Galileo as shown below.

 Now you can select Intel Galileo Gen2 as board and /dev/cu.usbmodemXXXX as port.

In the Arduino IDE, choose File > Examples > 01.Basics > Blink in Arduino IDE and upload it into your board.

The LED in the board will be blinked as highlighted above after uploaded the sketch.

Obligatory Hello World

The real journey of Galileo starts if you play around with sensors.  I bought SeedStudio Grove Start Plus kit that comes with necessary sensors, base sheild, RGB backlight LCD and necessary cables.  I use  the following items for this excercise:

  • base shield
  • sound sensor
  • RGB backlighted LCD.

The base shield is used as an expansion board that is compatible with the Arduino interface on Galileo.  The below figure shows the Ardunio interface.

The base shield will be like below

 You should connect the base shield on this interface as shown below.

The sensor and LCD can be connected with Galileo through this shield. Sound sensor to be connected to A0 and LED to be connected to D7 of the base shield. 

// this is for RGB backlighted LCD
#include <Wire.h>
#include "rgb_lcd.h"

const int pinSound = A0; // listen on A0
rgb_lcd lcd;

// threshold value for the sound level
int thresholdValue = 600;

void setup()
// print init message on LCD
lcd.begin(16, 2);
lcd.print("Sound Alert!");

void loop()
// read the value of the sound sensor
int sensorValue = analogRead(pinSound);


// If the measured sound level is above the threshold, display
// the value with red background
if(sensorValue > thresholdValue)
lcd.setRGB(255, 120, 120);
// otherwise green background
lcd.setRGB(120, 255, 120);

// delay time

The final outcome as shown below:

Don’t use elephant for your garden work

While learning the new Tez engine and query vectorization concepts in Hadoop 2.0, I came to know that the query vectorization is claimed as 3x powerful and consume less CPU time in actual Hadoop cluster. Hortonworks tutorial uses a sample sensor data in a CSV that is imported into Hive. Then a sample has been used to explain the performance.

The intention of this post is neither explaining Tez engine and query vectorization nor Hive query.  Let us familiarize the problem I have worked before get to know the purpose of this post. 🙂

One sample CSV file called ‘HVAC.csv’ contains 8000 records that contain temperature information on different building during different days. Part of the file content:


In the Hive, following configurations are specified to enable Tez engine and query vectorization.

hive> set hive.execution.engine=mr;
hive> set hive.execution.engine=tez;
hive> set hive.vectorized.execution.enabled;

I execute the following query in my sandbox  that surprisingly took 48 seconds for a ‘group by’ and ‘count’ on 8000 records as shown below:

select date, count(buildingid) from hvac_orc group by date;

This query groups the sensor data by date and count the number of building for that date.  It produces 30 results as shown below:

Status: Finished successfully
6/1/13  267
6/10/13 267
6/11/13 267
Time taken: 48.261 seconds, Fetched: 30 row(s)

Then I plan to write simple program without MapReduce castle, since it is just 8000 records. I created a F# script that read the CSV (note that I did not use any CSV type provider) and using Deedle exploratory library (again, LINQ can also help). I achieved the same result as shown below.

module ft

#I "..\packages\Deedle.1.0.0"
#load "Deedle.fsx"
open System
open System.IO
open System.Globalization
open System.Diagnostics
open Deedle

type hvac = { Date : DateTime; BuildingID : int}

let execute =
	let stopwatch = Stopwatch.StartNew()

	let enus = new CultureInfo("en-US")
	let fs = new StreamReader("..\ml\SensorFiles\HVAC.csv")
	let lines = fs.ReadToEnd() |> (fun s -> s.Split("\r\n".ToCharArray()))

	let ohvac = lines.[1..(Array.length lines) - 1]
				|> Array.map (fun s -> s.Split(",".ToCharArray()))
				|> Array.map (fun s -> {Date = DateTime.Parse(s.[0], enus); BuildingID = int(s.[6])})
				|> Frame.ofRecords

	let result = ohvac.GroupRowsBy("Date")
				|> Frame.getNumericCols
				|> Series.mapValues (Stats.levelCount fst)
				|> Frame.ofColumns

	(stopwatch.ElapsedMilliseconds, result)

In the FSI,

> #load "finalTouch.fsx";;
> open ft;;
> ft.execute;;
val it : int64 * Deedle.Frame =
01-06-2013 12:00:00 AM -> 267
02-06-2013 12:00:00 AM -> 267
03-06-2013 12:00:00 AM -> 267
04-06-2013 12:00:00 AM -> 267

The is completed within 83 milliseconds. You may argue that I comparing apple with orange. No!.  My intention is to understand when MapReduce is the savior.  The parable of the above exercise is that be cautious and analyze well before moving your data processing mechanisms into MapReduce clusters.

Elephants are very effective in labor requiring hard slogging and heavy lifting. Not for your gardens!! 🙂

Note that the sample CSV files from HortonWorks is clearly for training purpose. This blog post just take that as an example to project the maximum data-generation capability of a small or medium size application for a period. The above script may not scale and will not perform well with more than the above numbers. Hence, this is not anti-MapReduce proposal.

Unpacking Apache Storm in developer box

It is a long time pending task to evaluate and learn Apache Storm. Storm infrastructure needs Nimbus and Zookeeper.

My intention is to install Storm in my regular Ubuntu single box instead of any cluster environment/VMs, the reason is Apache Storm is just a jar file.

Zookeeper was already installed in my machine as single-cluster mode. I already have other prerequisites Java 6 or greater and Python 2.6.6 or greater.

I simply extract apache-storm-0.9.1-incubating.tar.gz into my app directory. We need to play around with two directories as like
|______ bin
|______ conf

Update the following settings in conf/storm.yaml.

    - ""
storm.zookeeper.port: 9191
storm.local.dir: "/mnt/storm"
nimbus.host: ""
   - 6700
   - 6701
   - 6702
   - 6703

Open three terminal windows, switch to super user and type the following commands respectively from the bin folder.

./storm dev-zookeeper
./storm nimbus
./storm supervisor
./storm ui

The last command opens Storm UI portal at localhost:8080 in browser.