Mesos Services Discovery and Load Balancing with Bamboo
For larger enterprise, running and managing workload in multi-cloud environment is an extremely challenging task. While public cloud vender started offering custom PaaS services to reduce operational overheads, those enterprise need to have a long term view on the strategy before they can adopt in-flight technologies to avoid long term risks. Mesos is able to offer the scalability, improve efficiency and portability of the workload across multiple cloud environments.
“Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared resource and Marathon is a Mesos framework for long-running tasks.”
Although many systems are running “just fine” in traditional virtual machines or managed PaaS, I’m always seeking how to make things better. Sometimes these are small incremental changes, sometimes they are large projects.
Before Kubernetes was available, we were working on self-managed data services running under mixed of bare-metals and public cloud to optimize costs and operability. The data services was handing billions of transactions and petabyte of data. The system was largely engineering on top of Apache Mesos for the management of heterogeneous services in an environment neutral way (i.e. equally happy in both cloud and co-location environments), all scaling dynamically with traffic. As we continued to roll out more services based on Mesos, however, we found that load balancing across various instances of a running service across the Mesos cluster to be a common problem.
Based on those challenges, I initiated on developing a project, Bamboo, that seeks to simplify running a DNS based HAProxy auto configuration and auto service discovery web daemon for Apache Mesos and Marathon. It was initially developed for internal projects, then later on open sourced for the community.
During the development phase, we set some basic goals:
- Access canary services via a human-friendly DNS name
- Review products per development branch feature in an automated fashion
- Quickly promote or rollback a service or its dependent web services
- Reproduce a Test, QA or Third type of environment for different use cases
- Efficiently allocate infrastructure resources, with support for rapid, dynamic reallocations
- Simplified, Docker friendly workflow
Bamboo provides a minimal set of features to automate services discovery and web request load balancing to web services running in Mesos cluster. It features:
- A user interface for configuring DNS mapping to Marathon id
- A REST API for configuring DNS mapping; during automated application deployment via Marathon with DNS management services an API comes very handy
- Automatic HAProxy configuration based on templates; you can provision your own template in production to enable SSL and protect HAProxy stats interface, or even configuring different load balance strategy
- Optional handling of health check endpoint if Marathon application is configured with Healthchecks
- Stateless daemon; enables horizontal replication and scalability
- Developed in Golang, deployment on HAProxy instance has no additional dependency
- Optional integration with StatsD to monitor configuration restart/reload events
Bamboo ensures HAProxy understands DNS and directs traffic to the correct web application in Mesos Cluster. When scaling up or down a number of applications running in Marathon, each Bamboo web daemon auto reconfigures HAProxy process and only direct traffic to the application when it passes health check.
If you have a small scale web service to manage, the user interface is useful to manually manage and visualize current state of DNS settings. You can find out if a DNS is assigned or missing from the interface. The template is flexible to customize based on your infrastructure requirement. Of course, you can configure HAProxy template to load balance Bamboo web interface too.