HPC + Big Data = Opportunity

Software-defined infrastructure (SDI) as an innovation center

Using AWS CloudFormation - In Search of Software Defined Infrastructure

I have a simple DevOps setup I want to put into the cloud. I was looking at the fastest way I could repeatability set up an infrastructure. I found AWS CloudFormation to be pretty straight forward. And I found that I can use my same CloudFormation Templates in my OpenStack cloud as well as AWS.

Deployment Diagram

In this configuration I am setting up a BuildServer(TeamCity), BuildAgents, and a Defect/Work Management Server (YouTrack). Since these tools come from the same vendor they have nice integrations that I want to take advantage.

As you can see in the Deployment diagram I put the ports that I want exposed in the deployment.

CloudFormation Templates

CloudFormation templates are XML or JSON files that are used to describe resources, their configuration and applications running on those resources. When I first started looking at CloudFormation I put all three nodes into the same Template file. This did not work since I can only define one EC2 instance per Template file. You can describe multiple resources like Databases, Storage, Load Balanacers, etc...

The Template has two major parts: the Description and the Resources.

• Description
• Resources
• Auto Scaling Group – Ensure that at least one is running
• Launch Group – Defines the EC2 Image, Init Scripts and where to get binary.
• EC2 image – ami file, Instance Type, and Security key to use for SSH
• Init Scripts- Installation scripts, start server scripts, stop server scripts.
• Sources for installation – Stored in an S3 location. Automatically unzips in specified directory.
• Security Group
• Defines the ports to open in the firewall.

Installations of Applications

Inside the CloudFormation Template you can specify the installation zip or tar file that can be used for the installation of the application. It will automatically unzip/untar the file and put it in the directory you specify. Make sure the installation zip file is in a location that your instances can use. I use S3 containers to do this. It helps decrease cost from loading over the internet everytime it starts up. It also gives me a way to do "poor-mans" configuration management. I couple of things to remember when you are doing this.

• Make sure you set permission so your instances can access them.
• Get the install zip file URL and put it in your Template file.
• If you are using public installations. You can make use the Public URL and set the Permission to public (readonly).

Init Scripts

These are used to setup and tear-down your applications on the Instances. I use the cfn-init helper script that gives my instance access to the template file.

The cfn-init helper script reads template metadata from the AWS::CloudFormation::Init key and acts accordingly to:

• Fetch and parse metadata from CloudFormation
• Install packages
• Write files to disk
• Enable/disable and start/stop services

Store the stop, start, install scripts in the metadata in the template file. It allows for multiple configsets, which gives me the ability to use the same template file for different configurations of the same application or different platform configurations.

Kicking off your instance

There are two ways to kick off your instances.

• AWS Console UI
• https://console.aws.amazon.com/cloudformation

• AWS Command Line
• aws cloudformation …
• aws cloudformation create-stack –-stack-name “MyStack”  -–template_body file://home/mystack.template ...

CloudFormation Learnings

Here are some key learnings

• One Node - One Template File
• Only one type of EC2 Image can be used in a CloudFormation File. (No-heterogeneous environments). You can have more than one instance in your autoscaling group but they must all be the same instance type.
• Most AWS services can be used. ElasticBeanStalk, OpsWare, EC2, S3, SecurityGroups, etc..
• Use a Common SSH key for everything in the same infrastructure deployment. It makes it easier to check on everything.
• Create an Custom Image for instance that you want quick spin up times. No additional installation of software or packages. (BuildAgents are images).

TeamCity Template File

• For the TeamCity Template file I set up the following for AWS:
• EC2 Image – ami-e7527ed7 – AWS Linx image
• Instance Type – t2.micro
• Ports: 3389, 80, 8111, and 22
• Init Scripts
• 01-stop - teamcity-server.sh stop
• 02-start - teamcity-server.sh start

TeamCity.template file

 

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "TeamCity Server for Linux",
  "Resources": {
    "TeamCityAS": {
      "Type": "AWS::AutoScaling::AutoScalingGroup",
      "Properties": {
        "AvailabilityZones": {
          "Fn::GetAZs": ""
        },
        "LaunchConfigurationName": {
          "Ref": "TeamCityLC"
        },
        "MaxSize": "1",
        "MinSize": "0",
        "DesiredCapacity": "1"
      }
    },
    "TeamCityLC": {
      "Type": "AWS::AutoScaling::LaunchConfiguration",
      "Properties": {
        "ImageId": "ami-e7527ed7",
        "InstanceType": "t2.micro",
        "KeyName": "teamcity",
        "SecurityGroups": [
          {
            "Ref": "SecurityGroup"
          }
        ],
        "UserData": {
          "Fn::Base64": {
            "Fn::Join": [
              "",
              [
                "#!/bin/bash\n",
                "yum update -y aws-cfn-bootstrap\n",
                "# Install the files and packages from the metadata\n",
                "/opt/aws/bin/cfn-init -v ",
                "         --stack ",
                { "Ref": "AWS::StackName" },
                "         --resource TeamCityLC",
                "         --configsets all ",
                "         --region ",
                { "Ref": "AWS::Region" },
                "\n"
              ]
            ]
          }
        }
      },
      "Metadata": {
        "AWS::CloudFormation::Init": {
          "configSets": {
            "all": [
              "server"
            ]
          },
          "server": {
            "commands": {
              "01-stop": {
                "command": "/opt/TeamCity/bin/teamcity-server.sh stop",
                "waitAfterCompletion": 5,
                "ignoreErrors": "true"
              },
              "02-start": {
                "command": "/opt/TeamCity/bin/teamcity-server.sh start",
                "waitAfterCompletion": 120
              }
            },
            "sources": {
              "/opt/": "https://s3-us-west-1.amazonaws.com/com.yoly.teamcity.installation/installations/TeamCity-9.1.tar.gz"
            }
          }
        }
      }
    },
    "SecurityGroup": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupDescription": "TeamCity Server Security Group",
        "SecurityGroupIngress": [
          { "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "3389", "ToPort": "3389" },
          { "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "80", "ToPort": "80" },
          { "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "8111", "ToPort": "8111" },
          {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0"}
        ]
      }
    }
  }
}

YouTrack Template File

• EC2 Image – ami-e7527ed7 – AWS Linx image
• Instance Type – t2.micro
• Ports: 3389, 8080, 80, and 22
• Init Scripts
• 01-permissions – chmod –R 755 /opt/YouTrack
• 02-stop – youtrack.sh stop
• 03-start – youtrack.sh start

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "YouTrack Server for Linux",
  "Resources": {
    "YouTrackAS": {
      "Type": "AWS::AutoScaling::AutoScalingGroup",
      "Properties": {
        "AvailabilityZones": {
          "Fn::GetAZs": ""
        },
        "LaunchConfigurationName": {
          "Ref": "YouTrackLC"
        },
        "MaxSize": "1",
        "MinSize": "0",
        "DesiredCapacity": "1"
      }
    },
    "YouTrackLC": {
      "Type": "AWS::AutoScaling::LaunchConfiguration",
      "Properties": {
        "ImageId": "ami-e7527ed7",
        "InstanceType": "t2.micro",
        "KeyName": "teamcity",
        "SecurityGroups": [
          {
            "Ref": "SecurityGroup"
          }
        ],
        "UserData": {
          "Fn::Base64": {
            "Fn::Join": [
              "",
              [
                "#!/bin/bash\n",
                "yum update -y aws-cfn-bootstrap\n",
                "# Install the files and packages from the metadata\n",
                "/opt/aws/bin/cfn-init -v ",
                "         --stack ",
                { "Ref": "AWS::StackName" },
                "         --resource YouTrackLC ",
                "         --configsets youtrack ",
                "         --region ",
                { "Ref": "AWS::Region" },
                "\n"
              ]
            ]
          }
        }
      },
      "Metadata": {
        "AWS::CloudFormation::Init": {
          "configSets": {
            "youtrack": [
              "youtrack"
            ]
          },
          "youtrack": {
            "commands": {
              "01-permissions": {
                "command": "chmod -R 777 /opt/YouTrack",
                "waitAfterCompletion": 1,
                "ignoreErrors": "true"
              },
              "02-stop": {
                "command": "/opt/YouTrack/bin/youtrack.sh stop",
                "waitAfterCompletion": 5,
                "ignoreErrors": "true"
              },
              "03-start": {
                "command": "/opt/YouTrack/bin/youtrack.sh start",
                "waitAfterCompletion": 120
              }
            },
            "sources": {
              "/opt/YouTrack": "https://s3.....installation/installations/youtrack-6.0.12634.zip"
            }
          }
        }
      }
    },


    "SecurityGroup": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupDescription": "TeamCity Server Security Group",
        "SecurityGroupIngress": [
          { "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "3389", "ToPort": "3389" },
          { "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "80", "ToPort": "80" },
          { "IpProtocol": "tcp", "CidrIp": "0.0.0.0/0", "FromPort": "8080", "ToPort": "8080" },
          {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0"}
        ]
      }
    }
  }
}

Toolbox Full of Container Technology

This last weekend I started cleaning out my garage. I said I started, because my garage will never completely be cleaned out. But I did manage to clean up my tool box. I found lots of interesting things in my tool box. Some broken tools, some tools that I have no idea how they got there. I am sure my neighbors and friends probably are missing them.  I also found half started projects that just got thrown in their that don't matter anymore. Probably in a hurry to clean up the mess in the garage. Anyway. I started thinking about all of these tools in my tool box, and of course comparing them to the tools I use at work. They are not things I can touch like the hardware in my tool box, but they are invaluable tools that I can use to get jobs done. I quickly broke down the tools into categories and found a group of tools that I have downloaded, played with and am using everyday that fit in the "Container Technology" category. So I gathered those tools together and decided to write this blog to help me keep track of what container tool to use for what job. This is what I have so far. Please note this is not a complete list, but a good start I think.

Container Tools

• Docker - Container Definition and nice CLI to control Containers
• Docker Compose - Define applications that contain more than one container.
• Mesos - Meta-scheduler, Distribute Kernel
• Docker Swarm - Container Scheduler that can talk to MesosS
• Kubernetes - Container Scheduler that can schedule and control Containers.
• Marathon - Service based scheduler for containers. Plugin to Mesos
• Chronos - Temporal Based Scheduler for containers. Plugin to Mesos

This is how I see everything fitting together.

Docker (Production - 1.7.0)

This helps developers define how to configure a container. Starts with a base image and then allows you to run commands, expose ports, and copy data into the container through a simple configuration file. Docker also gives CLI and REST API to allow users to control the individual containers. It requires a Docker controller running on a physical or virtual machine.

Docker Compose (Beta - 1.7.0)

Gives developers the ability to define an application as a set of micro-services. It shows the dependencies between the micro-services (containers). Which ports they expose to each other, their startup order, and any shared resources(data) between the containers that make up the application.

Mesos (Production - 0.23.0)

"A distributed systems kernel" from the mesos web site.

Mesos describes itself as a distributed system kernel. It is responsible for picking physical compute nodes to run containers, jobs, micro-services, etc... I gets telemetry from the physical hardware and can determine which machines can best handle incoming jobs. It then provisions machines and executes job on those machines. Schedulers like Kubernetes, Marathon, Chronos, and Docker Swarm sit ontop of Mesos and act as the orchestrator for the containers running in the cloud.

Schedulers

Docker Swarm (Still in Alpha - 0.3.0 )

If you want to use a familiar Docker API you can use Docker Swarm with Mesos to control multiple containers on mutilple hosts. There is rumor it may also allow you to talk to Kubernetes in the future. Check out http://www.techrepublic.com/article/docker-and-mesos-like-peanut-butter-and-jelly/ for more information.

Kubernetes ( pre-production Beta - 0.19.0 )

• If you want to launch groups of containers (K8 Pods) co-scheduled and co-located together, sharing resources on the same machine.
• If you want to launch a service alongside one or more sidekick containers (e.g. log archiver, metrics monitor) that live next to the parent container.
• if you want to use the K8s label-based service-discovery, load-balancing, and replication control. Very cool stuff for managing large number of containers.
• If you want to manage groups of containers(K8 Pods).
• If you want to load balance in isolation (K8 Pods).

Marathon (Production - 0.8.2)

• If you want to launch applications that contain long running heterogeneous apps/services (Docker and Non-Docker).
• If you want to use Mesos attributes for constraint-based scheduling.
• If you want to use application groups and dependencies to launch, scale, or upgrade related services.
• If you want to use event driven health checks to automatically restart unhealthy services.
• If you want to integrate HAProxy or Consul for service discovery.
• If you want a nice web UI or REST API to launch and monitor apps.

Chronos (Production - 2.3.4)

• If you want to launch applications that contain short running heterogeneous apps/services (Docker and Non-Docker).
• If you want to schedule a remporal task to run at a specific time/schedule, just like cron.
• If you want to schedule a DAG workflow of dependent tasks.
• If you want a nice web UI or REST API to launch and monitor apps.
• If  you want to use a scheduler that was built from the start with Mesos in mind.

Ok. This is the start. I am sure many of you have more to add to the list. Please let me know what I am missing.

DWP.

Automating SailsJS Testing with a CI tool

I recently started working with SailsJS. I am enjoying some of the libraries and how easy and fast it is to get things up and running. But I do miss some of the built in tools that come with Grails. A good example of this is the "grails test" tool. This made it easy to run unit level tests, integration or functional tests from the command line with the same command line for all of my projects.

grails test-app 

This made it very easy to integrate into a CI tool like TeamCity. I didn't need any special scripts, relative paths or some dark art of getting environment variables right. Things just worked out of the box.

Since SailsJS lets you plug in any testing you want. AKA it does not come built into the framework. I had to figure out how to run the tests in my IDE (Intellij in this case), and in my CI system (TeamCity). As you can see I am trying to keep the tool set in the same family.

I decided to use Mocha as the test framework. It seemed to have everything that I wanted and was the closest to what I was familiar with in the grails framework. But the commands to get it running were a little more than just typing "sailjs test" in my application directory. But I found a cool little trick to make it easy. Using npm and my package.json file. I can use the same command for both CI and IDE.

By simply adding a line to the package.json file in the root directory of the application source I can then call

npm test

Here is an example of my package.json file.

...

  "sails-mongo": "^0.11.1"},
"scripts": {
  "start": "node app.js",
  "test": "node ./node_modules/mocha/bin/_mocha --recursive --ui bdd ./test",
  "debug": "node debug app.js"},
"main": "app.js",

...

The test script calls the node command with the proper mocha script to run my tests. Notice that the path to the mocha script is relative to the application root directory. It is important in the IDE and CI that you set the working directory to the application root. Which should be the default for both. Now I can simply call:

npm test

from the command line, in the IDE, or in the CI system.

DWP