Downloading and uploading dashboards to/from a graphite server

April 24th, 2014

I’ve been using graphite and statsd lately, and over lunch quickly whipped up this script to download/upload dashboard configurations to/from a graphite server. Here’s the code, use as you wish.

#!/usr/bin/env python
#
# Author: Dustin Spicuzza
# 
# Use this script to download/upload dashboards to/from a graphite server.
#

import json

import sys
import urllib
import urllib2


download_url = 'http://%s/dashboard/load/%s'
upload_url = 'http://%s/dashboard/save/%s'


def download(ip, name, fname):
    
    uf = urllib2.urlopen('http://%s/dashboard/load/%s' % (ip, name))

    data = json.loads(urllib2.unquote(uf.read()))
    data_str = json.dumps(data['state'], sort_keys=True, indent=4, separators=(',',': '))

    if fname == None:
        print data_str
    else:
        with open(fname, 'w') as fp:
            fp.write(data_str)

    return 0


def upload(ip, name, fname):

    if fname is None:
        data_str = sys.stdin.read()
    else:
        with open(fname, 'r') as fp:
            data_str = fp.read()

    data = json.loads(data_str)

    if data['name'] != name:
        print "ERROR: dashboard name doesn't match name in state file"
        return 1

    post_data = urllib.urlencode([('state', json.dumps(data))])

    request = urllib2.Request(upload_url % (ip, name), post_data)
    request.add_header("Content-type", "application/x-www-form-urlencoded")

    print urllib2.urlopen(request).read()
    return 0


def usage():
    print "Usage: dasher.py [upload graphite_host name filename] | [download graphite_host name [filename]]"
    exit(1)

if __name__ == '__main__':

    if len(sys.argv) < 4:
        usage()

    action = sys.argv[1]
    ip = sys.argv[2]
    name = sys.argv[3]
    fname = None

    if len(sys.argv) > 4:
        fname = sys.argv[4]

    if action == 'upload':
        retval = upload(ip, name, fname)

    elif action == 'download':
        retval = download(ip, name, fname)

    else:
        usage()

    exit(retval)

Global shared development folder on Vagrant

December 20th, 2013

Vagrant is pretty awesome for development. One thing that I’ve ran into is that I use a lot of vagrant instances at various times, and much of the time I want to access my development files from inside the VM. One thing that is nice about vagrant is that by default it maps the folder where the Vagrantfile is located to /vagrant inside the VM. However, most of the time the content I want to access isn’t in that folder, so I found a good way to allow me to access stuff without needing to copy content all over the place.

What you can do is setup a global Vagrantfile, and all of the VMs that are stood up by your username will get the settings inside that VM. Just create a file  ~/.vagrant.d/Vagrantfile so it looks like the following. This will map some local folder to /src on the vagrant VM — but of course, you should set the paths to values that make sense for you.

Vagrant.configure("2") do |config|

    config.vm.provider :virtualbox do |vbox, override|

        # path on your local machine
        host_folder_name = "~/local/path/to/somewhere"

        # path where the local folder is mapped to inside the VM
        vm_folder_name = "/src"

        # In newer versions of Vagrant, you should use "type" otherwise
        # you may find it rsyncing your computer to the VM
        override.vm.synced_folder File.expand_path(host_folder_name), vm_folder_name, type: "virtualbox"
    end

end

Of course, if you set something like this up, definitely use the vagrant-rekey-ssh plugin to make sure that nobody else is able to access your VM via SSH using the default insecure vagrant keys.

Beware of using Vagrant VMs on a bridged network

December 10th, 2013

I love Vagrant. If you haven’t used it, Vagrant is pretty awesome. It lets you manage VMs + configurations really easily, and it’s a great development tool to use when you need to do rapid iterative development on disposable environments.

However, last week I realized that because of the way it’s implemented, all vagrant VMs share the same set of credentials to access them. Since most people only do local development with them, this is mostly ok (though it could be used to jump process boundaries and escalate privileges if you were creative and already running on the box) — however, in a bridged configuration, this is a huge security vulnerability as *anyone* on the internet could potentially use this key to SSH into your VM.

Since some of the stuff I do involves using bridged VMs, I wrote a Vagrant plugin to fix this problem. It replaces the default vagrant SSH key with one randomly generated for your user on that host. If you want to fix this security vulnerability on your vagrant installation, just do:

vagrant plugin install vagrant-rekey-ssh

Hope you find it useful! The github site for the plugin has more useful information if you want to read more.

Tool to implement ruby-inspired DSLs in python

November 17th, 2013

This weekend, I created library to aid developers in creating neat little embedded DSLs when using python, without having to do any complex parsing or anything like that. The resulting DSLs look a bit more like english than python does.

The idea for this was inspired by some ruby stuff that I’ve been using lately. I’ve been using ruby quite a bit lately, and while I am still not a huge fan of the language, I do like the idea of easy to code up DSLs that I can use to populate objects without too much effort. Since I wanted a DSL for a python project I was working on, I played with a few ideas and ended up with this tool. Read the rest of this entry »

Versioned chef environments

November 3rd, 2013

I was just recently introduced to chef, and it has turned out to be a pretty useful tool for automating infrastructure. One feature that I’m using in our environment is chef environments. A bunch of developers are all updating this file, and their cookbooks depend on attributes in it*.  I keep getting bit by various user errors that all could be solved if a cookbook could state ‘make sure you have at least this version of the environment’.

I’m sure there’s a good reason for it, but chef does not currently support versioning environments. To work around this problem, I created a cookbook with a library function that can check a node attribute and compare the version information there to determine if the environment is out of date. It’s a bit of a hack for now, but it gets the job done.

I thought someone else might find this useful, so I uploaded the code to github and to the opscode community site. If you have a need for versioned environments in chef, this might work for you too. Let me know if you find it useful!

*Yes, this could be considered an anti-pattern, but for our use case using environments to override attributes makes perfect sense, and allows us to not have to fork every community cookbook that we want to use.

Another Python SIP wrapper for the tesseract OCR library

July 26th, 2013

Tesseract is a pretty decent open source OCR engine that was developed by HP back in the day, but is now maintained as open source by Google. It has a C++ API that you can program to, and as you would expect there are a number of wrappers (of varying quality) that allow you to use libtesseract from Python. For various reasons, none of those fit my needs, so I created my own SIP-based wrapper instead. I will not be maintaining this as a format project, but if you want an apache licensed python wrapper, you can find the code on github. :)

GitHub repository for Python Tesseract SIP wrapper

Award Winning FIRST Robotics Robot Control Interface

April 8th, 2013

KwarqsDashboard is an award-winning control system developed in 2013 for FIRST Robotics Team 2423, The Kwarqs. For the second year in a row, Team 2423 won the Innovation in Control award at the 2013 Boston Regional. The judges cited this control system as the primary reason for the award.

It is designed to be used with a touchscreen, and robot operators use it to select targets and fire frisbees at the targets. Check out the shiny screenshot below:

Kwarqs Dashboard ScreenshotThere are a lot of really useful features that we built into this control interface, and once we got our mechanical and electrical bugs ironed out the robot was performing quite well. We’re looking forward to competing with it at WPI Battle Cry 2013 in May!  Here’s a list of some of the many features it has:

  • Written entirely in Python
    • Image processing using OpenCV python bindings, GUI written using PyGTK
    • Cross platform, fully functional in Linux and Windows 7/8
  • All control/feedback features use NetworkTables, so the same robot can be controlled using the SmartDashboard instead if needed
    • SendableChooser compatible implementation for mode switching
  • Animated robot drawing that shows how many frisbees are present, and tilts the shooter platform according to the current angle the platform is actually at.
  • Allows operators to select different modes of operation for the robot using brightly lit toggle switches
  • Operators can choose an autonomous mode on the dashboard, and set which target the robot should aim for in modes that use target tracking
  • Switches operator perspective when robot switches modes
  • Simulated lighted rocker switches to activate robot systems
  • Logfiles written to disk with errors when they occur

Target acquisition image processing features:

  • Tracks the selected targets in a live camera stream, and determines adjustments the robot should make to aim at the target
  • User can click on targets to tell the robot what to aim at
  • Differentiates between top/middle/low targets
  • Partially obscured targets can be identified
  • Target changes colors when the robot is aimed properly

Fully integrated realtime analysis support for target acquisition:

  •  Adjustable thresholding, saves settings to file
  • Enable/disable drawing features and labels on detected targets
  • Show extra threshold images
  • Can log captured images to file, once per second
  • Can load a directory of images for analysis, instead of connecting to a live camera

So, as you can see, lots of useful things. We’re releasing the full source code for it under a GPL license, so go ahead, download it, play with it, and let me know what you think! Hope you find this useful.

A lot of people made this project possible:

  • Some code structuring ideas and PyGTK widget ideas were derived from my work with Exaile
  • Team 341 graciously open sourced their image processing code in 2012, and the image processing is heavily derived from a port of that code to python.
  • Sam Rosenblum helped develop the idea for the dashboard, and helped refine some of the operating concepts.
  • Stephen Rawls helped refine the image processing code and distance calculations.
  • Youssef Barhomi created image processing stuff for the Kwarqs in 2012, and some of the ideas from that code were copied.

The included images were obtained from various places:

  • Linda Donoghue created the robot image
  • The fantastic lighted rocker switches were created by Keith Sereby, and are distributed with permission.
  • The green buttons were obtained via google image search, I don’t recall where

Download it on my FRC resources page!

VirtualKD 2.8 with VirtualBox 4.2

March 11th, 2013

If you try to install VirtualKD 2.8 on VirtualBox 4.2.x, you get an error similar to this:

Unable to cast COM object of type ‘VirtualBox.VirtualBoxClass’ to interface type ‘VirtualBox.IVirtualBox’. This operation failed because the QueryInterface call on the COM component for the interface with IID ‘{…}’ failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).

It turns out it’s a pretty easy thing to fix. The Interop.VirtualBox.dll distributed with VirtualKD is built for the 4.1 VirtualBox interface, so you just have to rebuild it for your version of VirtualBox. Create a C# project, and paste the following code into the Program.cs file to build a new Interop.VirtualBox.dll for 4.2.x.

using System;
using System.Reflection;
using System.Reflection.Emit;
using System.Runtime.InteropServices;

namespace ConvertTypeLibToAssembly
{
    public class App
    {
        private enum RegKind
        {
            RegKind_Default = 0,
            RegKind_Register = 1,
            RegKind_None = 2
        }
    
        [ DllImport( "oleaut32.dll", CharSet = CharSet.Unicode, PreserveSig = false )]
        private static extern void LoadTypeLibEx( String strTypeLibName, RegKind regKind, 
            [ MarshalAs( UnmanagedType.Interface )] out Object typeLib );
        
        public static void Main()
        {
            Object typeLib;
            LoadTypeLibEx( @"C:\Program Files\Oracle\VirtualBox\VBoxC.dll", RegKind.RegKind_None, out typeLib ); 
            
            if( typeLib == null )
            {
                Console.WriteLine( "LoadTypeLibEx failed." );
                return;
            }
                
            TypeLibConverter converter = new TypeLibConverter();
            ConversionEventHandler eventHandler = new ConversionEventHandler();
            //AssemblyBuilder asm = converter.ConvertTypeLibToAssembly( typeLib, "Interop.Virtualbox.dll", 0, eventHandler, null, null, null, null );
            AssemblyBuilder asm = converter.ConvertTypeLibToAssembly(typeLib, "Interop.VirtualBox.dll", TypeLibImporterFlags.SafeArrayAsSystemArray, eventHandler, null, null, "VirtualBox", null); //using assembly name "VirtualBox" and SafeArrayAsSystemArray to be compatible to VisualStudio-Generated Interop-Assembly

            asm.Save("Interop.Virtualbox.dll");
        }
    }

    public class ConversionEventHandler : ITypeLibImporterNotifySink
    {
        public void ReportEvent( ImporterEventKind eventKind, int eventCode, string eventMsg )
        {
            // handle warning event here...
        }
        
        public Assembly ResolveRef( object typeLib )
        {
            // resolve reference here and return a correct assembly...
            return null; 
        }    
    }
}

Some of this code was grabbed from MSDN, and a single line was grabbed from VirtualBoxService. As the latter is GPL, this code may be GPL. I don’t claim any rights to it.

Problems with file descriptors being inherited by default in Python

February 6th, 2013

Have you ever run into a traceback that ends with something like this?

  File "C:\Python27\lib\logging\handlers.py", line 141, in doRollover
    os.rename(self.baseFilename, dfn)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process

I certainly have, in a few places. The basic problem is that when python creates file objects on Windows (and I think on *nix as well), by default Python will mark the handle as being inheritable (I’m sure there’s a reason why… but, doesn’t make a whole lot of sense for this to be the default behavior to me). So if your script spawns a new process, that new process will inherit all the file handles from your script — and of course since it doesn’t realize that it even has those handles, it’ll never close them. A great example of this is launching a process, then exiting. When you launch your script again and try to open that handle… the other process still has it open, and depending on how the file was opened, you may not be able to open it due to a sharing violation.

It looks like they’re trying to provide ways to fix the problem in PEP 433 for Python 3.3, but that doesn’t help those of us still using Python 2.7. Here’s a snippet that you can put at the very beginning of your script to fix this problem on Windows:

import sys

if sys.platform == 'win32':
    from ctypes import *
    import msvcrt
    
    __builtins__open = __builtins__.open
    
    def __open_inheritance_hack(*args, **kwargs):
        result = __builtins__open(*args, **kwargs)
        handle = msvcrt.get_osfhandle(result.fileno())
        windll.kernel32.SetHandleInformation(handle, 1, 0)
        return result
        
    __builtins__.open = __open_inheritance_hack

Now, I admit, this is a bit of a hack… but it solves the problem for me. Hope you find this useful!

Notepad++ plugin to enable unindent on backspace key

December 3rd, 2012

When I first started using editors, I was a tabs person. I always set the tab size to 4 since I liked the way that looked, and used tabs instead of spaces. And it wasn’t really a philosophical reason behind it, it was very simple actually:

I really hate hitting backspace 4 times to unindent

Seems simple enough. Yes, I know you can use SHIFT-BACKSPACE or other voodoo to make it work, but I want to hit *one* key.

Because of this, I always used 4-space tabs instead of inserting spaces instead of tabs. However, at some point I started writing a project in python, and python tends to encourage spaces vs tabs. Notepad++ is my favorite editor, so I decided to write a plugin for Notepad++ to make using spaces a lot less annoying — and in particular, to make it so that when I hit backspace, it would magically eat the spaces back to the next indentation level.

In the process of trying to do this, I discovered that the Scintilla editing component actually has this functionality built-in — you just need to enable SCI_SETBACKSPACEUNINDENTS on the editor component. So I took the sample plugin for Notepad++ and added two lines of code that enabled this functionality, and it works great! Ever since then, I’ve used spaces instead of tabs, since I really can’t tell the difference — and that’s how I like it :)

I actually wrote this thing a rather long time ago, and never got around to actually cleaning it up and releasing it. I’ve decided to release it in its present form, since it’s so useful and very simple. It would be even better if Notepad++ had an option to enable this, but this works just as well until that happens.

This plugin is known to work in versions of Notepad++ since 4.x, and works in Notepad++ running under Wine also. Download the Notepad++ tabs plugin at my usual software page.