Download and Run Karamel
A Hadoop Cluster on Amazon EC2
You will need to have an account on Amazon Web Services. You should have an account-key (id), and an account secret-key (password). The secret-key is very long - 20 characters.
You should then open a Cluster Definition YAML file. The Cluster Definition file defines the machines you are going to create and the software that will be installed. Here is an example of an Apache Hadoop cluster - the hadoop.yml file. It can be found in the "examples" directory included in the Karamel download. Here is what the YAML file looks like:
name: ApacheHadoop ec2: type: m3.medium region: eu-west-1 cookbooks: hadoop: github: "hopshadoop/apache-hadoop-chef" branch: "master" attrs: hadoop: version: 2.7.1 groups: namenodes: size: 1 recipes: - hadoop::nn - hadoop::rm - hadoop::jhs datanodes: size: 2 recipes: - hadoop::dn - hadoop::nm
Launching Clusters from the command-line in Linux/Mac
You can either set environment variables containing your EC2 credentials or enter them from the console. We recommend you set the environment variables, as shown below.
export AWS_KEY=... export AWS_SECRET_KEY=... ./bin/karamel -launch examples/hadoop.yml
After you launch a cluster from the command-line, the client loops, printing out to stdout the status of the install DAG of Chef recipes every 20 seconds or so. Both the GUI and command-line launchers print out stdout and stderr to log files that can be found from the current working directory in:
tail -f log/karamel.log
How to write a cluster yaml?
Karamelize a cookbook
- Deploy your cookbook into a public github repository.
- Karamel assumes that each cookbook has install and default recipes. By default all the recipes of each cookbook are locally dependent on the install recipe, so your cookbook must contain the install recipe.
- Define all the public recipes and attributes in the metadata.rb file, only those are visible to Karamel system. To improve user experience, write a description for recipe and display_name, description, type (data-type) and a default value for attributes to be displayed in Karamel GUI.
- Create the Karamelfile at the root level of each cookbook, it holds the cluster-wide and recipe to recipe decencies. Karamelfile is the source of information for ordering and orchestration in Karamel, therefore, when you are defining dependencies in the Karamelfile you specifying precedence of your recipes among all other recipes in the cluster. When there is a/some prerequisite dependency(yes) for a particular recipe you must mention them in the Karamelfile. Below is a sample example of Karamelfile, local dependencies means this recipe just waits until the dependent recipe is finished on the current machine, but global dependencies means this recipe on every machine waits for the all the dependent machines in the cluster to be run first.
dependencies: - recipe: ndb::ndbd global: - ndb::mgmd - recipe: ndb::mysqld global: - ndb::mgmd - ndb::ndbd - recipe: ndb::memcached global: - ndb::mgmd - ndb::ndbd - ndb::mysqld