UNIX SHA-512 Passwords & Python

Well today I spent a bit of time looking up how /etc/shadow created their shadow passwords.

I found a good source for the method at http://www.akkadia.org/drepper/SHA-crypt.txt , and if you take some time to review it you will notice the steps are a bit involved

Lucky for us Python has a Crypt module that works nicely.

First off I started by creating a dummy user with the password of test on one of my Linux computers:

# cat /etc/shadow | grep dummy

And like that we have a password we can attempt to validate.

Next I will need to define our salt in Python. A good article explaining the format of /etc/shadow can be found on cyberciti.biz. Like the article says the salt is found between the $ after the username and before the password hash. For comparison I will also extract the hash:

>>> salt = 'JJ1sbaL/'
>>> hash = '2P4ZcZ10VjIa9f5.8N/GJukYCg.RJVBTP2h30v2mnh0Q5izqc2rtKZ6qtI0Ewiyj0hX9zIVfftOVuJTIsBSKQ/'

Now that we have our salt we can create a hash of our password using that salt and verify it matches the hash shown in /etc/shadow:

>>> import crypt
>>> output = crypt.crypt('test', '$6$%s$' % salt)
>>> print output

And lastly lets compare the new hash with the one found in /etc/shadow:

>>> newhash = output.split('$')[-1]
>>> print newhash
>>> newhash == hash

Wordlist with all Possible ASCII Elements

After a bit of thinking I believe I have a better way to create all possible ASCII pass phrases, rather than create a recursive function that calls itself as I did in the previous post.

This method takes a slightly different approach, but I believe it to return the same results.

First off let’s create a list of all lower case ASCII character numbers:

>>> chars = range(97, 123)
>>> chars
[97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
113, 114, 115, 116, 117, 118, 119, 120, 121, 122]
>>> map(chr, chars)
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p',
'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

Next we can set our max word width and current word width variables:

import itertools
maxwidth = 3
currentwidth = 1

At this point we are ready to run with a while loop:

while int(currentwidth) <= int(maxwidth):
    for i in itertools.product(chars, repeat=currentwidth):
        print ''.join(['%c' % c for c in list(i)])
    currentwidth += 1

The loop above will run until currentwidth is greater than maxwidth (makes sense right), at that point we use itertools’s product method to create all possible orderings for our characters.

Then using list comprehension I print out the %c which returns the alphabetical character.

And lastly we increase the currentwidth by 1

What itertools.product does is creates all possible combinations (including repeated chars):

>>> [i for i in itertools.product(chars, repeat=2)]
[(97, 97), (97, 98), (97, 99), (97, 100), (97, 101), (97, 102), (97,
 103), (97, 104), (97, 105), (97, 106), (97, 107), (97, 108), (97, 109),
 (97, 110), (97, 111), (97, 112), (97, 113), (97, 114), (97, 115), (97,
 116), (97, 117), (97, 118), (97, 119), (97, 120), (97, 121), (97, 122),
 (98, 97), (98, 98), (98, 99), (98, 100), (98, 101), (98, 102), (98,
103), (98, 104), (98, 105), (98, 106), (98, 107), (98, 108), (98, 109),
 (98, 110), (98, 111), (98, 112), (98, 113), (98, 114), (98, 115), (98,
 116), (98, 117), (98, 118), (98, 119), (98, 120), (98, 121), (98, 122),
 (99, 97), (99, 98), (99, 99), (99, 100), (99, 101), (99, 102), (99,

Of course you probably want to see those tuples of characters as letters right:

>>> for i in itertools.product(chars, repeat=2):
...   ''.join(['%c' % c for c in i])

Python Libvirt Domain Configuration

So in my last article I talked a little bit about using libvirt to start and stop QEMU domains.

In this article I would like to go over how domains are created.

QEMU uses XML files for each domain configuration, and using libvirt we can access that data.

First off lets connect to our local running QEMU instance:

>>> import libvirt
>>> conn = libvirt.open('qemu:///system')

Next lets look for and link to a running domain:

>>> conn.listDomainsID()
[35, 36, 37, 41, 38, 39]

>>> domain = conn.lookupByID(35)

The libvirt domain object gives os a function called XMLDesc() which we can use to see our running configuration:

>>> raw_xml = domain.XMLDesc(0)
>>> print raw_xml

<domain type='kvm' id='35'>
    <type arch='x86_64' machine='fedora-13'>hvm</type>
    <boot dev='hd'/>
  <clock offset='utc'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/data/libvirt/datastore01/data01-dev-clone-2.img'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    <interface type='bridge'>
      <mac address='50:00:00:00:00:00'/>
      <source bridge='br1'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target port='0'/>
      <alias name='serial0'/>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes'/>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

Now say you are interested in working with this XML data, we can do just that using the xml module:

>>> from xml.dom import minidom
>>> xml = minidom.parseString(raw_xml)

And something that may be useful in the XML above is the type tag, so lets grab that data:

>>> domainTypes = xml.getElementsByTagName('type')
>>> for domainType in domainTypes:
...   print domainType.getAttribute('machine')
...   print domainType.getAttribute('arch')

And there you have it, we can easily pull out data from this XML using the minidom function.

Using this data it should be pretty easy to deploy a new guest domain using the createXML function:

Help on method createXML in module libvirt:

createXML(self, xmlDesc, flags) method of libvirt.virConnect instance
    Launch a new guest domain, based on an XML description similar
    to the one returned by virDomainGetXMLDesc()
    This function may require privileged access to the hypervisor.
    The domain is not persistent, so its definition will disappear when it
    is destroyed, or if the host is restarted (see virDomainDefineXML() to
    define persistent domains).

    If the VIR_DOMAIN_START_PAUSED flag is set, the guest domain
    will be started, but its CPUs will remain paused. The CPUs
    can later be manually started using virDomainResume.

    If the VIR_DOMAIN_START_AUTODESTROY flag is set, the guest
    domain will be automatically destroyed when the virConnectPtr
    object is finally released. This will also happen if the
    client application crashes / loses its connection to the
    libvirtd daemon. Any domains marked for auto destroy will
    block attempts at migration, save-to-file, or snapshots.

Consuming Jenkins API using Python

Lately I’ve been using Jenkins as a central hub for automated task across multiple machines, think a central cron system with a slick web interface.

In a nutshell Jenkins CI is the leading open-source continuous integration server. Built with Java, it provides over 400 plugins to support building and testing virtually any project.

One thing that is very nice about Jenkins is the easy to use REST JSON API.

Since our Jenkins system is used internally I’ve allowed the anonymous user full read access, this allows our API read call operations in without the need to use an API KEY.

Each Jenkins page has a REST API hyperlink at the bottom, this is because each page has its own endpoint.

I’ll go over getting a list of all Projects, this uses the end point of /api/json:

>>> from urllib2 import urlopen
>>> from json import loads
>>> req = urlopen('')
>>> res = req.read()
>>> data = loads(res)

At this point we have our JSON data loaded in to a Python dict (just like every other JSON API):

>>> data.keys()
[u'unlabeledLoad', u'jobs', u'description', u'views', u'quietingDown', u'assignedLabels',
u'nodeDescription', u'useCrumbs', u'numExecutors', u'mode', u'slaveAgentPort',
u'overallLoad', u'useSecurity', u'primaryView', u'nodeName']

We can go ahead and parse over these Jobs:

>>> for job in data['jobs']:
...   print job['name']
Pull Logs from Cloud
Jenkins Job Status Notify

Or we can look directly at the dict object of the first job:

>>> data['jobs'][0].keys()
[u'url', u'color', u'name']

This will give us the full url to the job, all we need to do is append api/json to get our object:

>>> req = urlopen('%s/%s' % (data['jobs'][0]['url'], 'api/json'))
>>> res = req.read()
>>> job = loads(res)

And again we have a Python dict object:

>>> job.keys()
[u'scm', u'color', u'lastSuccessfulBuild', u'actions', u'lastCompletedBuild',
u'lastUnsuccessfulBuild', u'upstreamProjects', u'lastFailedBuild', u'healthReport',
u'queueItem', u'lastBuild', u'lastStableBuild', u'description', u'downstreamProjects',
u'concurrentBuild', u'lastUnstableBuild', u'buildable', u'displayNameOrNull',
u'inQueue', u'keepDependencies', u'name', u'displayName', u'builds', u'url',
u'firstBuild', u'nextBuildNumber', u'property']

Let say for example we want to see if the last ran build for this job was success, to do that we use the lastCompletedBuild key:

>>> job['lastCompletedBuild'].keys()
[u'url', u'number']

And like last time lets make the API call:

>>> req = urlopen('%s/%s' % (job['lastCompletedBuild']['url'], 'api/json'))
>>> res = req.read()
>>> build = loads(res)

And yet again a Python dict:

>>> build.keys()
[u'building', u'changeSet', u'builtOn', u'description', u'artifacts', u'timestamp',
u'number', u'actions', u'id', u'keepLog', u'url', u'culprits', u'result', u'executor',
u'duration', u'fullDisplayName', u'estimatedDuration']

But from here we can see the build number and status:

>>> build['number'], build['result']
(16, u'SUCCESS')

I hope this was helpful.

Diablo III API

So I was doing a bit of research on a Diablo III API, and stumbled across this article:


It seems they are still in the early stages, but there is a Character Calculator API, and it is in JSON.

First off we need to load in our Python libraries:

>>> from urllib2 import urlopen
>>> from json import loads

We can make our request using common URL methods:

>>> res = urlopen('http://us.battle.net/d3/en/data/calculator/monk').read()
>>> monk = loads(res)

Now we are given a simple Python dictionary:

>>> monk.keys()
[u'skills', u'traits']

And like most APIs these keys contain a list of objects:

>>> type(monk['skills'])
<type 'list'>
>>> type(monk['skills'][0])
<type 'dict'>

>>> monk['skills'][0].keys()
[u'tooltipParams', u'name', u'runes', u'simpleDescription', u'description', u'primary', u'slug',
u'requiredLevel', u'categoryName', u'categorySlug', u'icon']

So far so good, lets go ahead and itter over all our skills and get their names:

>>> for skill in sorted(monk['skills']):
...   print skill['name']
Blinding Flash
Breath of Heaven
Inner Sanctuary
Mystic Ally
Cyclone Strike
Seven-Sided Strike
Mantra of Conviction
Mantra of Healing
Mantra of Retribution
Mantra of Evasion
Deadly Reach
Way of the Hundred Fists
Fists of Thunder
Crippling Wave
Tempest Rush
Lashing Tail Kick
Wave of Light
Dashing Strike
Exploding Palm
Sweeping Wind

So there you have it, using one of the new Diablo III APIs to get class information.