File not found!

I’ve seen it before, a customer deletes a file and then needs it restored.

Normally a challenging request, but under special circumstances a process may have the file opened.

While showing my son some fun and exciting Linux security scenarios, I recalled all those times I was able to recover data from the /proc (in memory) filesystem. In order to visualize this scenario I threw together a small Dockerfile and had him poke around:

Once logged in you notice a script, for our scenario this is our companies server daemon. If we browse the code we notice a file named password.txt was opened, but never closed.

Ignore the sleep() function, this is just here to keep the file opened, and to simulate a daemon process.

The next things you should do it verify the server is actively running! If we hope to recover that file the server needs to have a lock on password.txt

root@4df66824ad14:~# ps aux
root   1  0.0  0.0  18504 3396 pts/0  Ss   01:37   0:00 /bin/bash
root  10  0.0  0.1  22548 6076 pts/0  S    01:37   0:00 python
root  24  0.0  0.0  34396 2904 pts/0  R+   01:45   0:00 ps aux

We are in luck!

From here we can use the lsof command to verify password.txt is still opened:

root@4df66824ad14:~# lsof | grep password.txt
python 10 root 3r REG 8,1 30 15860072 /root/password.txt (deleted)

Bingo! we see the file is opened, and it is marked for deletion. Good thing our service is still running!

All that is left is to track down our service’s process id (we see above it’s process id, or PID is 10).

We now have all the pieces to track down our password file from the in memory filesystem.

The /proc filesystem has a number of useful files, it also includes a directory for every running process (named after the PID).

Within the process directory we have a slew of files; however, the directory we care about is fd (standing for file descriptor):

root@4df66824ad14:~# cd /proc/10
root@4df66824ad14:/proc/10# ls -l fd
total 0
lr-x------ 1 root root 64 Sep 28 01:43 0 -> /dev/null
lrwx------ 1 root root 64 Sep 28 01:43 1 -> /dev/pts/0
lrwx------ 1 root root 64 Sep 28 01:43 2 -> /dev/pts/0
lr-x------ 1 root root 64 Sep 28 01:43 3 -> '/root/password.txt (deleted)'

And just like that we found a file representing our in memory password.txt file!

From here we can show its contents, or even restore it using cp:

root@4df66824ad14:/proc/10# cat fd/3

Happy hacking!

MySQL IN Operator Not Using Index

After being alerted for elevated cpu on a staging mysql instance, I logged into the mysql console and ran a SHOW PROCESS LIST:

*************************** 1. row ***************************
     Id: 1
   User: my_user
   Host: x.x.x.x:xxx
     db: my_db
Command: Sleep
   Time: 2
   Info: NULL
*************************** 2. row ***************************
     Id: 3
   User: my_user
   Host: x.x.x.x:xxx
     db: my_db
Command: Query
   Time: 786
  State: Sending data
   Info: SELECT file_id, fs_file_id
FROM my_data
WHERE file_id IN ('123456', 123457);

A query running for 786 seconds quickly caught my eye, my next course of action was to run an EXPLAIN:

EXPLAIN SELECT file_id, fs_file_id FROM my_data WHERE file_id IN ('123456', 123457)\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: my_data
         type: ALL
possible_keys: PRIMARY
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 131144420
        Extra: Using where
1 row in set (0.00 sec)

The EXPLAIN confirmed my suspicions, this query was not using an index, and ultimately caused a full table scan.

I was expecting to not find an index for the file_id field, but as you can see this field has an index by being a PRIMARY KEY:

*************************** 1. row ***************************
Table: my_data
Create Table: CREATE TABLE `my_data` (
`file_id` int(10) unsigned NOT NULL,
`fs_file_id` binary(16) DEFAULT NULL,
PRIMARY KEY (`file_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
1 row in set (0.00 sec)

Something I didn’t immediately identify earlier was the mixed types in my IN operator‘s list, they contained both integers and strings:

file_id IN ('123456', 123457)

Once I noticed the mixed type I began debugging, I finally learned the IN operator works great with sets of integers:

EXPLAIN SELECT file_id, fs_file_id FROM my_data WHERE file_id IN (123456, 123457)\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: my_data
         type: range
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 4
          ref: NULL
         rows: 2
        Extra: Using where
1 row in set (0.00 sec)

It even works with sets of strings:

EXPLAIN SELECT file_id, fs_file_id FROM my_data WHERE file_id IN ('123456', '123457')\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: my_data
         type: range
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 4
          ref: NULL
         rows: 2
        Extra: Using where
1 row in set (0.00 sec)

But as we observed from the begining, the IN operator doesn’t act as expected with mixed strings and integers.

Python traceroute style tool

Recently, while talking with a some techies I was asked to explain how traceroute works. I was quick to answer what results I expect back from the command, and how to understand that data, but for the life of me I was having issues recalling how TTL works at the foundation.

Later that week I spent some time reading up on it, and click, it all came back.

To reinforce this I decided to write some code. I decided to use Scapy in order to make
crafting network packets a breeze (shown below). You should be able to get Scapy right from PyPi:

$ sudo pip install scapy

#!/usr/bin/env python

# avoid printing some ipv6 debug stuff from scapy.
import logging

from scapy.all import IP, ICMP, sr1

from sys import argv

def tracehop(host_or_ip):
  Return the network hops used to arrive at destination.
  # craft our IP portion of the packet with destination address.
  ip = IP(dst=host_or_ip)

  # use a blank ICMP packet.
  icmp = ICMP()

  # set our time to live to 1, this will get our first hop.
  ip.ttl = 1

  # we will loop until getting a echo-response.
  while True:

    # send the ICMP packet and expect one response.
    res = sr1(ip/icmp, verbose=False)

    # print out the responding host source address.
    print res['IP'].src

    # if our ICMP type is 0 we've got
    # the echo response and can break while loop.
    if res['ICMP'].type == 0:

    # if we did not break above we increase the ttl by
    # one, and let the while loop run once again.
    ip.ttl += 1

if __name__ == '__main__':

The above code should be documented well enough to answer most questions.

Python is unable to create network sockets without root privileges. I do not advise running any script you find on the internet as root until reading over it, and fully understanding what may happen to your system and why root privileges are needed.

With that out of the way lets get right to the usage and output:

$ sudo python

We can compare traceroute’s results to ours, and make sure the hops are relatively similar:

$ traceroute -n
traceroute to (, 30 hops max, 60 byte packets
 1 19.107 ms 19.470 ms 19.315 ms
 2 19.208 ms 28.023 ms 28.233 ms
 3 28.142 ms 28.034 ms 27.901 ms
 4 27.799 ms 27.683 ms 27.557 ms
 5 27.426 ms 27.330 ms *
 6 27.095 ms 11.672 ms 18.996 ms
 7 27.821 ms 27.727 ms 32.801 ms
 8 15.742 ms 20.731 ms 17.218 ms
 9 28.956 ms 28.876 ms 28.787 ms
10 31.914 ms 32.112 ms 28.468 ms
11 28.356 ms 31.783 ms 25.326 ms
12 23.522 ms 30.280 ms 26.967 ms
13 26.921 ms 31.626 ms 27.053 ms
14 26.340 ms 26.614 ms 27.644 ms

Then again we could just proof this using traceroute, the -m flag sets our TTL:

$ traceroute -m 1
traceroute to (, 1 hops max, 60 byte packets
 1 ( 3.398 ms 3.592 ms 3.482 ms

And then hack together quickly a bash script:

$ for i in $(seq 1 30) ; do
     traceroute -m $i -n | tail -n 1 | egrep "$i " | awk '{print $2}'

And now the inner-workings of traceroute should be stuck in my head,
never to be tripped up again, or at least that is the plan 😉

Process Elasticsearch JSON on the shell

Lets throw security out the window for a moment. Say we store user accounts with clear text passwords in Elasticsearch, what is the easiest way to use the results in a shell script? We can begin by creating two accounts, one for admin and one for john:

# curl -XPUT localhost:9200/site/people/1?pretty=True -d '
  {"name": "admin", "password": "secret", "admin": "true"}
  "_index" : "site",
  "_type" : "people",
  "_id" : "1",
  "_version" : 1,
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
 "created" : true
# curl -XPUT localhost:9200/site/people/2?pretty=True -d '
  {"name": "john", "password": "password", "admin": "false"}
  "_index" : "site",
  "_type" : "people",
  "_id" : "1",
  "_version" : 2,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
 "created" : false

Using curl this is very easy to query, we just use the id:

# curl localhost:9200/site/people/2?pretty=True
  "_index" : "site",
  "_type" : "people",
  "_id" : "2",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "name" : "john",
    "password" : "password",
    "admin" : "false"

But what we really want is to get the values of the keys without needing to hack it with sed/awk/grep. We can install a small json parsing package called jq:

# apt-get install jq

Now we can easily pull the keys out:

# curl -s localhost:9200/site/people/2 | jq

# curl -s localhost:9200/site/people/2 | jq ._source.password

# curl -s localhost:9200/site/people/2 | jq ._source.admin

Using the power of jq we could easily search Elasticsearch and perform comparisons for login, remember we are throwing security out the window here, this is merely for example:

# curl -s $ip:9200/site/people/_search?q=name:admin | jq -r '.hits.hits'
    "_source": {
      "admin": "true",
      "password": "secret",
      "name": "admin"
    "_score": 0.30685282,
    "_id": "1",
    "_type": "people",
    "_index": "site"

We can assume this query only returns one user, and we are pretty sure the account exists, so lets grab the password key directly:

# curl -s $ip:9200/site/people/_search?q=name:admin |
  jq -r '.hits.hits[0]._source.password'

If the value is not present, and we attempt to access it, we will get a null result:

# curl -s $ip:9200/site/people/_search?q=name:jack |
  jq -r '.hits.hits[0]._source.password'


And here it is all wrapped up in a poorly written shell script, I would not advise using this logic for anything more than toying with:


read -p "Login username: " username
read -p "Login password: " password

account=`curl -s localhost:9200/site/people/_search?q=name:$username |
  jq '.hits.hits[0]'`

if [ "$account" != "null" ] ; then

  account_username=`echo $account | jq -r`

  if [ "$username" == "$account_username" ] ; then

    account_password=`echo $account | jq -r ._source.password`

    if [ "$password" == "$account_password" ] ; then
      echo "You are in!"
      echo "Wrong username or password"



And usage looks like this:

# ./
Login username: admin
Login password: password
Wrong username or password
# ./
Login username: john
Login password: password
You are in!


Telegraf laptop battery plugin

Wanted to expand a little on my previous blog post Custom Telegraf Plugin, and decided to do a simple battery monitor. The end result looks something like this:

Screen Shot 2016-04-11 at 5.40.47 PM

I decided to read from the file /sys/class/power_supply/BAT0/capacity on my Ubuntu 14.04 machine, this file merely shows the current battery percent:

# cat /sys/class/power_supply/BAT0/capacity

All that is needed is a little Python script for converting this output to JSON, my script outputs like this:

# /
{"status": "Charging", "capacity": 63}

The code is very basic Python:


#!/usr/bin/env python
import json

with open('/sys/class/power_supply/BAT0/capacity') as f:
 capacity ='\n')[0]
 capacity = int(capacity)

with open('/sys/class/power_supply/BAT0/status') as f:
 status ='\n')[0]

results = dict(capacity=capacity,

print json.dumps(results)

All that is left is to add this script to our Telegraf configuration directory:

# cat /etc/telegraf/telegraf.d/battery.conf
 command = "/"
 data_format = "json"
 name_suffix = "_battery"
 interval = "5s"

And there you have it, a simple battery tracker.