Stackato Essentials

notes, tips, and useful information

View the Project on GitHub ttilley/StackatoEssentials

My personal goal with this document is to provide a somewhat more gentle introduction to Cloud Foundry and Stackato than you might otherwise receive, as well as to shine light on components and functionality that might not be documented in sufficient detail elsewhere.

If you find any of the information here useful, you can thank me by providing feedback. Please feel free to fork, improve, and submit pull requests.

Core Components

General Architecture

Doozerd

Doozer is a highly-available, completely consistent store for small amounts of extremely important data. When the data changes, it can notify connected clients immediately (no polling), making it ideal for infrequently-updated data for which clients want real-time updates.
https://github.com/ActiveState/doozerd

Doozer is a recent addition to the stack that is unique to Stackato, replacing most configuration sources used by other implementations of Cloud Foundry. In standard Cloud Foundry one might SSH into the machine running a service to configure it (via YAML files), and restart said service to pick to those configuration changes. In Stackato you are able to configure the entire cluster from a single location and services react to configuration changes themselves. You are also able to query doozer for information on currently connected nodes.

Prealloc

Stager

DEA

Cloud Controller

The Cloud Controller handles all state transitions, manages users/apps/services, and provides the external REST API used by the stackato client (or VMC, should you so desire).

Health Manager

Router

The router handles all HTTP traffic into the cluster and maintains distributed routing state. The router responds to realtime updates from DEA nodes. Load balancing is performed when an app has multiple instances.

There are currently two implementations of the router component in Stackato (as of 2.2). In order to make use of websockets, one must use router2g rather than the default router.

NATS

NATS is a lightweight cloud messaging system by Derek Collison, who was previously Chief Architect for TIBCO's messaging products.

Message passing, via NATS, is the foundation of the Cloud Foundry architecture. It is used for addressing and component discovery, command and control, heartbeats, and various other tasks.

Cloud Controller Messages

DEA Messages

Health Manager Messages

Router Messages

Stager Messages

Data Services

All data services have three subcomponents: Node, Provisioner, and Gateway. The Node implements the actual service backend. The provisioner handles various logic around provisioning and unprovisioning a specific instance of the service. The gateway provides a REST interface to the provisioner. In practice the provisioner and gateway are usually implemented as a singular daemon.

Prior to Stackato 2.2, the filesystem service was a special exception to this rule. It did not have a gateway, and was assumed to exist on a singular node. This limitation has since been lifted.

Application Deployment Flow

This is a shortened version of what happens when you deploy an application:

  1. stackato push
  2. framework detection
  3. create app in CloudController
  4. framework staging plugin
  5. droplet creation
  6. request DEA for app
  7. an available DEA node responds
  8. droplet is deployed to the DEA
  9. DEA starts the app
  10. upon successful start, the router creates a route

Cluster Setup

Roles

Fog

require 'rubygems'
require 'fog'

STACKATO_V207_AMI       = "ami-595bf530"

AWS_ACCESS_KEY_ID       = ENV["AWS_ACCESS_KEY_ID"]
AWS_SECRET_ACCESS_KEY   = ENV["AWS_SECRET_ACCESS_KEY"]
EC2_REGION              = ENV["EC2_REGION"]             || "us-east-1"
EC2_AVAILABILITY_ZONE   = ENV["EC2_AVAILABILITY_ZONE"]  || "us-east-1d"

STACKATO_EMAIL          = ENV["STACKATO_EMAIL"]         || "stackato@stackato.local"
STACKATO_PASSWORD       = ENV["STACKATO_PASSWORD"]      || "stackato"

connection = Fog::Compute.new({
  :provider               => :AWS,
  :aws_access_key_id      => AWS_ACCESS_KEY_ID,
  :aws_secret_access_key  => AWS_SECRET_ACCESS_KEY,
  :region                 => EC2_REGION
})

server = connection.servers.bootstrap({
  :private_key_path   => "~/.ec2/default.pem", 
  :public_key_path    => "~/.ec2/default.pem.pub", 
  :availability_zone  => EC2_AVAILABILITY_ZONE, 
  :username           => "ubuntu", 
  :image_id           => STACKATO_V207_AMI,
  :flavor_id          => "m1.small",
  :bits               => 64
})

server.ssh(<<EOF
curl -k "https://api.$(hostname).local/stackato/license" \
     -d "email=#{STACKATO_EMAIL}&password=#{STACKATO_PASSWORD}&unix_password=#{STACKATO_PASSWORD}"
EOF
)

Chef

CloudInit

cloud-init is a highly configurable system for handling early initialization of a cloud instance. More information on cloud-init can be found at the project's launchpad. There is also a screencast covering the cloud-config portion of cloud-init on the ubuntucloud youtube channel.

Data Sources

The cloud-init package supports searching multiple "Data Sources" for information that can be used to configure a particular cloud instance. The list of potential sources is configured in the datasource_list key, with the first entry found to be valid becoming the data source for that run. This value, as well as any detailed configuration in datasource, should be set locally in cloud.cfg or injected early via some other means (like the kernel cmdline). Any Ubuntu based EC2 AMIs will contain EC2 as a data source by default, otherwise the list will contain ["NoCloud", "ConfigDrive", "OVF", "MAAS"].

Note: If the datasources_list value from /etc/cloud/cloud.cfg is not being used, check for an /etc/cloud/cloud.cfg.d/90_dpkg.cfg and run dpkg-reconfigure cloud-init if found.

Note: Selecting the "Ec2" data source may result in lengthy boot times if the EC2 metadata API is unavailable. If you firewall off the metadata API for security purposes and do not wish to use it with cloud-init, make sure to edit datasource_list. For a convenient method of removing access to the metadata API after cloud-init has retrieved any necessary data, enable the disable-ec2-metadata module.

If you are using a deployment platform that provides some level of EC2 compatibility then you may want to ensure that the metadata IP (169.254.169.254) is reachable and that the HTTP endpoints (ex: /2009-04-04/meta-data/instance-id) are available ahead of time. For additional configurability, you also have the option of manually setting metadata_urls on the data source:

datasource:
  Ec2:
    metadata_urls:
      - http://169.254.169.254:80
      - http://instance-data:8773

While CloudStack provides a metadata API with similarities to the one provided by EC2, there are notable differences that make it incompatible. In particular, the metadata API will be available via the default gateway and thus cannot be sanely configured ahead of time. There is a seperate "CloudStack" datasource that handles these differences.

Valid data sources:

Frequency

Scripts and configuration directives may be acted upon with the following frequencies:

  1. once ever
  2. once per instance
  3. once per boot (always)

You can usually assume that the default frequency for an action is once per instance. On every run, cloud-init will check what the current "instance id" is and compare this value with the previously recorded id. This makes it easy to do things like clone an existing system and ensure that a base set of actions are performed when the clone comes online. You may, for example, delete any pre-existing chef configuration and re-register as a new client.

Warning: On EC2, a full shutdown will result in your VM receiving a new instance-id.

The following modules, however, are loaded with a default frequency of "always":

Note that without accompanying configuration, a module that is loaded on each boot will still have no effect on the system.

It is possible to change the run frequency of a module. Under the module section where it is defined, replace the string name with an array of [name, frequency, arguments]. Within this context, frequency must be named once-per-instance, always, or once.

Order of Execution

The cloud-init package is run in several stages during bootstrap:

risks

Please be aware that the metadata service has no concept of access control. An application running on a DEA node has the same level of access to your metadata as anything else running on that machine. There are multiple methods of hiding sensitive information from prying eyes, such as include-once directives that point to single-use URLs. The easiest method by far, however, is enabling the disable_ec2_metadata option in your cloud-config. The disable_ec2_metadata module will run the following: route add -host 169.254.169.254 reject.

Monitoring

Ganglia

Auto-Scaling

Troubleshooting

Application Crash

If an application crashes, the linux container for that app sticks around for about an hour by default before getting cleaned up. This allows you to ssh into the container and perform any necessary debugging. Logs will also be available to you this way, even if they are not directly available via the dashboard any longer.

Note: you can access an app as it's owner, or as an admin with the --group='owner@email' option to the stackato CLI. The way option ordering works, the group option must come before the ssh command:

stackato --group='owner@email.com' ssh appname

In the scenario of an application crash:

  1. The DEA node detects the unexpected exit and broadcasts a message to NATS
  2. Routers remove the crashed app from the routing table
  3. The Health Manager notifies the Cloud Controller
  4. The Cloud Controller re-starts the instance

DEA Crash

In the scenario of a DEA node crash:

  1. Applications handled by the DEA become unavailable
  2. The Health Manager notices the missing instances and notifies the Cloud Controller
  3. The Cloud Controller requests DEA nodes for the apps
  4. As DEA nodes reply, application instances will be started on these new DEA nodes

Note that as part of the Prealloc/DEA init that occurs when starting the DEA service, previously existing linux containers are cleaned up and removed. If you need any of that data for debugging or analysis then you will want to copy it to another location than /lxc/containers or /mnt/lxc/containers.

Inotify

fs.inotify.max_user_instances = 4096
fs.inotify.max_user_watches = 32768
fs.inotify.max_queued_events = 65536