Process Elasticsearch JSON on the Shell

  • 18/4/2016
  • 3-minute read

Lets throw security out the window for a moment. Say we store user accounts with clear text passwords in Elasticsearch , what is the easiest way to use the results in a shell script? We can begin by creating two accounts, one for admin and one for john :

# curl -XPUT localhost:9200/site/people/1?pretty=True -d '
  {"name": "admin", "password": "secret", "admin": "true"}
'
{
  "_index" : "site",
  "_type" : "people",
  "_id" : "1",
  "_version" : 1,
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
 "created" : true
}

# curl -XPUT localhost:9200/site/people/2?pretty=True -d '
  {"name": "john", "password": "password", "admin": "false"}
'
{
  "_index" : "site",
  "_type" : "people",
  "_id" : "1",
  "_version" : 2,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
 "created" : false
}

Using curl this is very easy to query, we just use the id :

# curl localhost:9200/site/people/2?pretty=True
{
  "_index" : "site",
  "_type" : "people",
  "_id" : "2",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "name" : "john",
    "password" : "password",
    "admin" : "false"
  }
}

But what we really want is to get the values of the keys without needing to hack it with sed/awk/grep. We can install a small json parsing package called jq :

# apt-get install jq

Now we can easily pull the keys out:

# curl -s localhost:9200/site/people/2 | jq ._source.name
"john"

# curl -s localhost:9200/site/people/2 | jq ._source.password
"password"

# curl -s localhost:9200/site/people/2 | jq ._source.admin
"false"

Using the power of jq we could easily search Elasticsearch and perform comparisons for login, remember we are throwing security out the window here, this is merely for example :

# curl -s $ip:9200/site/people/_search?q=name:admin | jq -r '.hits.hits'
[
  {
    "_source": {
      "admin": "true",
      "password": "secret",
      "name": "admin"
    },
    "_score": 0.30685282,
    "_id": "1",
    "_type": "people",
    "_index": "site"
  }
]

We can assume this query only returns one user, and we are pretty sure the account exists, so lets grab the password key directly:

# curl -s $ip:9200/site/people/_search?q=name:admin |
  jq -r '.hits.hits[0]._source.password'
secret

If the value is not present, and we attempt to access it, we will get a null result:

# curl -s $ip:9200/site/people/_search?q=name:jack |
  jq -r '.hits.hits[0]._source.password'

null

And here it is all wrapped up in a poorly written shell script , I would not advise using this logic for anything more than toying with: auth.sh

#!/bin/bash

read -p "Login username: " username
read -p "Login password: " password

account=`curl -s localhost:9200/site/people/_search?q=name:$username |
  jq '.hits.hits[0]'`

if [ "$account" != "null" ] ; then

  account_username=`echo $account | jq -r ._source.name`

  if [ "$username" == "$account_username" ] ; then

    account_password=`echo $account | jq -r ._source.password`

    if [ "$password" == "$account_password" ] ; then
      echo "You are in!"
    else
      echo "Wrong username or password"
    fi

  fi

fi

And usage looks like this:

# ./auth.sh
Login username: admin
Login password: password
Wrong username or password

# ./auth.sh
Login username: john
Login password: password
You are in!
DevelopmentElasticsearchBash