Using Dockerfiles to setup our Elasticsearch nodes

In my previous two articles, Elasticsearch using Docker and Adding HAproxy to our Elasticsearch Docker example we walked through manually creating Docker images containing Elasticsearch,
and then connecting those with HAProxy. Let’s now look into using Dockerfiles to make automated builds of these environments.

So what are Dockerfiles? A Dockerfile is a plain text file that docker build command will take, and create Docker images. Lets get started.

I’m going to start by making a new directory structure for our Dockerfiles:

# mkdir Dockerfiles
# cd Dockerfiles/

Then with in the Dockerfiles directory I will create a directory for the elasticsearch node:

# mkdir search

Next lets enter the directory and create a empty Dockerfile:

# cd search/
# touch Dockerfile

Now lets download the elasticsearch executables from their upstream site:

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.0.tar.gz

Now our /root/Dockerfiles/search path should look something like this:

# ls -l
total 27120
-rw-r--r-- 1 root root 0 Nov 24 17:52 Dockerfile
-rw-r--r-- 1 root root 27730896 Nov 5 14:37 elasticsearch-1.4.0.tar.gz

We are now ready to populate our configuration in the Dockerfile:

FROM debian:wheezy

RUN mkdir /app
WORKDIR /app

RUN apt-get update
RUN apt-get install -y openjdk-7-jre

ADD elasticsearch-1.4.0.tar.gz /app/

EXPOSE 9200 9300
CMD ["/app/elasticsearch-1.4.0/bin/elasticsearch"]

This is a very simple and straight forward Dockerfile for our elasticsearch nodes, lets go ahead and build it into a image:

# docker build -t elasticsearch .
 .....
 .....
Successfully built 69ba38fbd056

This will take a couple of minutes as it is pulling down the openjdk package, but once done we should have a shiny new image!

# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
elasticsearch latest 69ba38fbd056 22 seconds ago 582.6 MB
debian wheezy f6fab3b798be 2 weeks ago 85.1 MB

Now lets fire up one of these images:

# docker run -d -p 9201:9200 -p 9301:9300 -t elasticsearch
d40eec4ce3f2632ed1a10c4956a9f03197ce9f1ce28b9321b88f2b3a3b108388

# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d40eec4ce3f2 elasticsearch:latest "/app/elasticsearch- 2 seconds ago Up 2 seconds 0.0.0.0:9201->9200/tcp, 0.0.0.0:9301->9300/tcp focused_sammet

And lets make sure we are HTTP accessible on port 9201:

# curl -XGET http://localhost:9201
{
 "status" : 200,
 "name" : "Fer-de-Lance",
 "cluster_name" : "elasticsearch",
 "version" : {
 "number" : "1.4.0",
 "build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
 "build_timestamp" : "2014-11-05T14:26:12Z",
 "build_snapshot" : false,
 "lucene_version" : "4.10.2"
 },
 "tagline" : "You Know, for Search"
}

Share on Facebook Share on Twitter Share on Google+

Bender’s Apartment Number

bender

In the hit series Futurama Bender’s apartment number of 00100100.

I heard the other day this represents the ASCII character $

Let quickly see how to validate that using Python!

First lets use the int function to pass in our binary as a string, then we make sure to specify this as a base 2 numbering:

In [1]: int('00100100', 2)
Out[1]: 36

Next we can use the built in chr function to return the ASCII character for the number 36:

In [2]: chr(36)
Out[2]: '$'

And there you have it, this checks out!

 


Share on Facebook Share on Twitter Share on Google+

Adding HAproxy to our Elasticsearch Docker example

In the last article titled “Elasticsearch using Docker” we walked through creating a custom docker image including elasticsearch, we then ran multiple instances across multiple ports. In this walk through I want to add haproxy to the mix.

First thing we need to do is fire up a few docker images:

# docker run -d -p 9201:9200 -p 9300:9300 2a523f874a5c /root/elasticsearch-1.4.0/bin/elasticsearch

# docker run -d -p 9202:9200 -p 9301:9300 2a523f874a5c /root/elasticsearch-1.4.0/bin/elasticsearch

# docker run -d -p 9203:9200 -p 9302:9300 2a523f874a5c /root/elasticsearch-1.4.0/bin/elasticsearch

We can verify the processes are running with the docker ps command:

# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d243fbc78bd7 jness/elasticsearch:v1 "/root/elasticsearch 5 minutes ago Up 5 minutes 0.0.0.0:9203->9200/tcp, 0.0.0.0:9303->9300/tcp jolly_mestorf
5187ae75c473 jness/elasticsearch:v1 "/root/elasticsearch 5 minutes ago Up 5 minutes 0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp dreamy_kowalevski
65fbed0d3174 jness/elasticsearch:v1 "/root/elasticsearch 5 minutes ago Up 5 minutes 0.0.0.0:9201->9200/tcp, 0.0.0.0:9301->9300/tcp dreamy_archimedes

We could also hit each to verify JSON is being returned:

# curl -XGET 127.0.0.1:9201
{
 "status" : 200,
 "name" : "Harmonica",
 "cluster_name" : "elasticsearch",
 "version" : {
 "number" : "1.4.0",
 "build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
 "build_timestamp" : "2014-11-05T14:26:12Z",
 "build_snapshot" : false,
 "lucene_version" : "4.10.2"
 },
 "tagline" : "You Know, for Search"
}

# curl -XGET 127.0.0.1:9202
{
 "status" : 200,
 "name" : "White Tiger",
 "cluster_name" : "elasticsearch",
 "version" : {
 "number" : "1.4.0",
 "build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
 "build_timestamp" : "2014-11-05T14:26:12Z",
 "build_snapshot" : false,
 "lucene_version" : "4.10.2"
 },
 "tagline" : "You Know, for Search"
}

# curl -XGET 127.0.0.1:9203
{
 "status" : 200,
 "name" : "Lightbright",
 "cluster_name" : "elasticsearch",
 "version" : {
 "number" : "1.4.0",
 "build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
 "build_timestamp" : "2014-11-05T14:26:12Z",
 "build_snapshot" : false,
 "lucene_version" : "4.10.2"
 },
 "tagline" : "You Know, for Search"
}

So far everything is looking good! Lets go ahead and install haproxy and configure it to roundrobin balance our three elasticsearch instances:

# apt-get install haproxy

Next lets open up the haproxy configuration and append our configuration for the elasticsearch processes:

vim /etc/haproxy/haproxy.cfg

First lets configure our frontend, this will be where we configure the listen port. This configuration creates a frontend named elasticsearch, binds it to port 9200, and passes all traffic to the backend named elasticsearch-backend

frontend elasticsearch
  bind 127.0.0.1:9200
  default_backend elasticsearch-backend

Next we need backend, the backend is where we specify our balance algorithm, and which servers are in the rotation:

backend elasticsearch-backend
 balance roundrobin
 mode tcp
 server es1 127.0.0.1:9201 check
 server es2 127.0.0.1:9202 check
 server es3 127.0.0.1:9203 check

We can now start our haproxy service and pull a couple GETs:

# service haproxy start

At this point we should be able to hit our haproxy port (9200) and get response from roundrobin balanced elasticsearch processes. Notice the name change on our elasticsearch nodes:

# curl -XGET 127.0.0.1:9200
{
 "status" : 200,
 "name" : "Harmonica",
 "cluster_name" : "elasticsearch",
 "version" : {
 "number" : "1.4.0",
 "build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
 "build_timestamp" : "2014-11-05T14:26:12Z",
 "build_snapshot" : false,
 "lucene_version" : "4.10.2"
 },
 "tagline" : "You Know, for Search"
}

# curl -XGET 127.0.0.1:9200
{
 "status" : 200,
 "name" : "White Tiger",
 "cluster_name" : "elasticsearch",
 "version" : {
 "number" : "1.4.0",
 "build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
 "build_timestamp" : "2014-11-05T14:26:12Z",
 "build_snapshot" : false,
 "lucene_version" : "4.10.2"
 },
 "tagline" : "You Know, for Search"
}

# curl -XGET 127.0.0.1:9200
{
 "status" : 200,
 "name" : "Lightbright",
 "cluster_name" : "elasticsearch",
 "version" : {
 "number" : "1.4.0",
 "build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
 "build_timestamp" : "2014-11-05T14:26:12Z",
 "build_snapshot" : false,
 "lucene_version" : "4.10.2"
 },
 "tagline" : "You Know, for Search"
}

# curl -XGET 127.0.0.1:9200
{
 "status" : 200,
 "name" : "Harmonica",
 "cluster_name" : "elasticsearch",
 "version" : {
 "number" : "1.4.0",
 "build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
 "build_timestamp" : "2014-11-05T14:26:12Z",
 "build_snapshot" : false,
 "lucene_version" : "4.10.2"
 },
 "tagline" : "You Know, for Search"
}

Excellent, we have balancing! Lets add some data:

# curl -XPOST 127.0.0.1:9200/flip-edesign/blog/1/ -d '
{
 "foo" : "bar"
}
'
{"_index":"flip-edesign","_type":"blog","_id":"1","_version":1,"created":true}

And we should be able to pull it:

# curl -XGET 127.0.0.1:9200/flip-edesign/blog/1/
{"_index":"flip-edesign","_type":"blog","_id":"1","_version":1,"found":true,"_source":
{
 "foo" : "bar"
}
}

Share on Facebook Share on Twitter Share on Google+

Elasticsearch using Docker

Elasticsearch is a distributed RESTFul search tool over the HTTP protocol. And we are going
to use Docker to spin up multiple nodes in the cluster. First we need a server node running Docker. I’m using a Debian server so the command I need is apt-get:

# apt-get install docker.io

After installing the package make sure the docker command is available:

# docker version
Client version: 1.3.1
Client API version: 1.15
Go version (client): go1.3.2
Git commit (client): 4e9bbfa
OS/Arch (client): linux/amd64
Server version: 1.3.1
Server API version: 1.15
Go version (server): go1.3.2
Git commit (server): 4e9bbfa

Excellent we now have docker, lets start by downloading a image, the below command will
download the latest Debian docker image:

# docker create debian
Unable to find image 'debian' locally
debian:latest: The image you are pulling has been verified
511136ea3c5a: Pull complete
f10807909bc5: Pull complete
f6fab3b798be: Pull complete
Status: Downloaded newer image for debian:latest
6cf83ed03695134d2a606f63c494dcf6e9dedcb2fe0db2768d8d5a95baac50c1

We can verify the debian image is available using the docker command images:

# docker images
REPOSITORY   TAG     IMAGE ID      CREATED      VIRTUAL SIZE
debian       latest  f6fab3b798be  8 days ago   85.1 MB

Next I will shell into the debian image and install some packages:

# docker run -t -i f6fab3b798be /bin/bash
root@85bcc90e1983:/#

You will want to take note of the running process id (85bcc90e1983) from the prompt.

Next lets update your apt repository cache and installed java runtime environment:

root@85bcc90e1983:/# apt-get update
root@85bcc90e1983:/# apt-get install openjdk-7-jre

From here we can get the latest statically compiled binaries from the elasticsearch download page:

root@85bcc90e1983:~# apt-get install wget
root@85bcc90e1983:~# wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.0.tar.gz
root@85bcc90e1983:~# tar -zxvf elasticsearch-1.4.0.tar.gz

From here lets test starting up the elasticsearch process:

root@85bcc90e1983:~# elasticsearch-1.4.0/bin/elasticsearch
[2014-11-15 00:18:30,616][INFO ][node ] [Ape-Man] version[1.4.0], pid[6482], build[bc94bd8/2014-11-05T14:26:12Z]
[2014-11-15 00:18:30,617][INFO ][node ] [Ape-Man] initializing ...
[2014-11-15 00:18:30,620][INFO ][plugins ] [Ape-Man] loaded [], sites []
[2014-11-15 00:18:32,805][INFO ][node ] [Ape-Man] initialized
[2014-11-15 00:18:32,805][INFO ][node ] [Ape-Man] starting ...
[2014-11-15 00:18:32,893][INFO ][transport ] [Ape-Man] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.17.0.2:9300]}
[2014-11-15 00:18:32,905][INFO ][discovery ] [Ape-Man] elasticsearch/-LrLApD4RhyPpz8VYbDAnQ
[2014-11-15 00:18:36,671][INFO ][cluster.service ] [Ape-Man] new_master [Ape-Man][-LrLApD4RhyPpz8VYbDAnQ][85bcc90e1983][inet[/172.17.0.2:9300]], reason: zen-disco-join (elected_as_master)
[2014-11-15 00:18:36,700][INFO ][http ] [Ape-Man] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.17.0.2:9200]}
[2014-11-15 00:18:36,700][INFO ][node ] [Ape-Man] started
[2014-11-15 00:18:36,711][INFO ][gateway ] [Ape-Man] recovered [0] indices into cluster_state

Everything looks good, lets CTRL+C out of the elasticsearch process and CTRL+C out of our docker process.

We then need to commit our change we made to the debian image, but we will save it as a new image name, we need the docker process id mentioned previously:

# docker commit -a 'jness' -m 'Elasticsearch v1.4.0' 85bcc90e1983 jness/elasticsearch:v1

It is now time to run a elasticsearch process using our new image, we will need to make sure
to map our network ports (transport process, and http process). Lets first find the IMAGE ID:

# docker images
REPOSITORY          TAG    IMAGE ID      CREATED             VIRTUAL SIZE
jness/elasticsearch v1     2a523f874a5c  About a minute ago  612.7 MB
debian              latest f6fab3b798be  8 days ago          85.1 MB

Using the above IMAGE ID we can start a process using the elasticsearch binary:

# docker run -d -p 9200:9200 -p 9300:9300 2a523f874a5c /root/elasticsearch-1.4.0/bin/elasticsearch

We should now have a running docker process, lets check using the docker ps command:

# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b621c107a1fb jness/elasticsearch:v1 "/root/elasticsearch 36 seconds ago Up 35 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp stoic_pike

Looks like we have our process running, lets make sure we can access it from the host using curl:

# curl -XGET localhost:9200
{
 "status" : 200,
 "name" : "Franklin Storm",
 "cluster_name" : "elasticsearch",
 "version" : {
 "number" : "1.4.0",
 "build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
 "build_timestamp" : "2014-11-05T14:26:12Z",
 "build_snapshot" : false,
 "lucene_version" : "4.10.2"
 },
 "tagline" : "You Know, for Search"
}

Sweet we have a response! lets have it store some data shall we?

# curl -XPOST localhost:9200/ness/jeff/1/ -d '
{
 "full_name" : "Jeffrey Ness"
}
'
{"_index":"ness","_type":"jeff","_id":"1","_version":1,"created":true}

And we should be able to retrieve that same piece of data:

 curl -XGET localhost:9200/ness/jeff/1/
{"_index":"ness","_type":"jeff","_id":"1","_version":1,"found":true,"_source":
{
 "full_name" : "Jeffrey Ness"
}
}

And finally lets see the true power of elasticsearch by adding a couple more nodes,
we will need to make sure we map the ports to unused ports on the host:

# docker run -d -p 9201:9200 -p 9301:9300 2a523f874a5c /root/elasticsearch-1.4.0/bin/elasticsearch

# docker run -d -p 9202:9200 -p 9302:9300 2a523f874a5c /root/elasticsearch-1.4.0/bin/elasticsearch

And without doing anything these two additional nodes should return the same data:

# curl -XGET localhost:9201/ness/jeff/1/
{"_index":"ness","_type":"jeff","_id":"1","_version":1,"found":true,"_source":
{
 "full_name" : "Jeffrey Ness"
}
}
# curl -XGET localhost:9202/ness/jeff/1/
{"_index":"ness","_type":"jeff","_id":"1","_version":1,"found":true,"_source":
{
 "full_name" : "Jeffrey Ness"
}
}

And there you have it! A single server running three docker processes of elasticsearch.

Hope you enjoyed this little walk-through  !

 


Share on Facebook Share on Twitter Share on Google+