Archive for the ‘Software’ Category

« Older Entries

Introducing pyhcl

Wednesday, October 15th, 2014

HCL is a configuration file format created by Hashicorp to “build a structured configuration language that is both human and machine friendly for use with command-line tools”. They created it to use with their tools because they weren’t satisfied with existing solutions, and I think they did a really good job with it.

I also have similar opinions. JSON has parsers in just about any language one can think of, but is a really terrible format for humans to deal with since it doesn’t support comments. YAML is sometimes mentioned as a better choice, but I think it’s generally pretty terrible to use. I’ve only had to use YAML a few times, but in my opinion it’s tricky to get YAML files of any complexity correct without a bit of futzing.

Instead, HCL is super easy to type out and read, and looks quite nice:

variable "ami" {
    # this is a comment
    description = "the AMI to use"
}

I liked the idea of HCL a lot, so I created a python implementation of the parser using ply, which I’ve called pyhcl. When you read an HCL file, pyhcl turns it into a python dictionary (just like JSON), and the dictionary representation for the above HCL is like this:

{
  "variable": {
    "ami": {
      "description": "the AMI to use"
    }
  }
}

To get that result, parsing HCL using pyhcl is super easy, and is pretty much the same as parsing JSON:

import hcl

with open('file.hcl', r) as fp:
     obj = hcl.load(fp)

So far, I’ve been really happy with HCL, and I’ve been using it for some projects with complex configuration requirements, and the end-users of those projects have been quite happy with the simplicity that HCL provides.

pyhcl is provided under the MPL 2.0, just like the golang parser, and can mostly be used in the same places one might use JSON. It’s probably not terribly performant — but parsing files that humans read shouldn’t really require performance. If you do, python’s JSON parser is written in C and should meet your needs quite nicely.

One thing that would be nice is if there was an actual specification for HCL, and in particular it isn’t very well defined on how to convert HCL to/from JSON… but lacking that, pyhcl currently tries to match the golang implementation bug for bug, and its test suite has stolen most of the fixtures from the golang parser to ensure maximum compatibility.

Github site for pyhcl
Pypi site for pyhcl

Posted in python, Software | No Comments »

Transparently use avro schemata (.avsc) files in a python module

Sunday, June 8th, 2014

One of the cool things about avro is that it has bindings in a couple of different languages. However, I think the only one that has native code generation support for working with avro objects is Java, which makes working with avro in the other languages a bit harder. Here’s a simple way to load your schemata dynamically (and if you don’t want to write your schemata by hand, then using maven you can generate it from AVDL files using the tip from my previous post).

What this bit of code does is overrides the __getattr__ function on the module, so anytime you try to access a type on the module, it will attempt to load the avro schema from a file of the same name with the avsc extension. To use this code, create a file called __init__.py in your directory of .avsc files, and paste the following code in.

import sys
from os.path import join, dirname
import avro.schema

class AvroSchemaLoader(object):
    '''
        This object allows us to lazily load schemata files in the current
        directory and parse them as needed.
        
        It is intended to be used as a replacement of the current module in
        sys.modules, so usage of this object should be transparent to users.
        
        For example, to access the Foo wrapper object, you would do the
        following:
        
            >>> from this_dir_name import Foo
            >>> print type(Foo)
            <avro.schema.RecordSchema at ...>
            >>>
    '''
    
    def __init__(self, module):
        # things break in odd ways if you don't keep a reference to the module here
        self.__module = module  

    def __getattr__(self, name):
        if name.startswith('__'):
            return object.__getattr__(self, name)
        
        with open(join(dirname(__file__), '%s.avsc' % name), "r") as fp:
            schema = avro.schema.parse(fp.read())
        
        setattr(self, name, schema)
        return schema


# Replace this module instance with the dynamic loader
sys.modules[__name__] = AvroSchemaLoader(sys.modules[__name__])

There’s a lot you can do to make this better — like load a wrapper around the schema instead of using the schema directly. I’ll leave that as an exercise for the reader. 🙂

Posted in avro, python, Software, tips | No Comments »

Automatically generating avro schemata (avsc files) using maven

Sunday, June 8th, 2014

I’ve been using avro for serialization a bit lately, and it seems like a really useful, flexible, and performant technology. To use avro containers, you have to define a schema for them — but writing out JSON files is a bit of a pain. Avro provides an IDL that you can use to specify the object types instead, and it’s much easier to work with. The avro-maven-plugin is quite useful because you can automatically generate Java objects from the IDL files — but what if you’re working with the same Avro files in a different language that can’t use the IDL?

Until they add the functionality to the maven plugin, there’s an easy way you can automate this yourself using a bit of maven magic. First thing to do is add the following dependency to your project’s pom.xml

<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-tools</artifactId>
  <version>1.7.6</version>
</dependency>

Next, you need to add a simple little class that converts an entire directory from avdl files to avsc files. Avro-tools ships with a useful class called IdlToSchemataTool that will convert a single file for you, so converting an entire directory is just a simple wrapper around that. There is a bit of improvement that could be done here, but this gets the job done assuming your directory only has avdl files in it.

package main;

import java.io.File;
import java.util.ArrayList;
import java.util.List;

import org.apache.avro.tool.IdlToSchemataTool;

/**
 * Converts an entire directory from Avro IDL (.avdl) to schema (.avsc)
 */
public class ConvertIdl {

	public static void main(String [] args) throws Exception {
		IdlToSchemataTool tool = new IdlToSchemataTool();
		
		File inDir = new File(args[0]);
		File outDir = new File(args[1]);
		
		for (File inFile: inDir.listFiles()) {
			List<String> toolArgs = new ArrayList<String>();
			toolArgs.add(inFile.getAbsolutePath());
			toolArgs.add(outDir.getAbsolutePath());
			
			tool.run(System.in, System.out, System.err, toolArgs);
		}
	}
}

Finally, you add the following to the plugins section of pom.xml to actually generate the avsc files. This uses the exec-maven-plugin to run the class we created above during compilation. This configuration assumes that you are storing your avdl files in src/main/avro, and that you want to place the files in schemata. Obviously you can reconfigure this however you want.

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <version>1.3</version>
    <executions>
        <execution>
            <phase>compile</phase>
            <goals>
                <goal>java</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <mainClass>main.ConvertIdl</mainClass>
        <arguments>
            <argument>${project.basedir}/src/main/avro/</argument>
            <argument>${project.basedir}/schemata/</argument>
        </arguments>
    </configuration>
</plugin>

And that’s it! To actually convert your avdl files to avsc files, run ‘mvn compile’ and the output directory should be filled with avsc files containing the JSON schema for your avro containers. Hope this helps you out, let me know if you find any bugs or have improvements.

Posted in avro, Java, maven, Software, tips | No Comments »

Award Winning FIRST Robotics Robot Control Interface

Monday, April 8th, 2013

KwarqsDashboard is an award-winning control system developed in 2013 for FIRST Robotics Team 2423, The Kwarqs. For the second year in a row, Team 2423 won the Innovation in Control award at the 2013 Boston Regional. The judges cited this control system as the primary reason for the award.

It is designed to be used with a touchscreen, and robot operators use it to select targets and fire frisbees at the targets. Check out the shiny screenshot below:

Kwarqs Dashboard ScreenshotThere are a lot of really useful features that we built into this control interface, and once we got our mechanical and electrical bugs ironed out the robot was performing quite well. We’re looking forward to competing with it at WPI Battle Cry 2013 in May!  Here’s a list of some of the many features it has:

  • Written entirely in Python
    • Image processing using OpenCV python bindings, GUI written using PyGTK
    • Cross platform, fully functional in Linux and Windows 7/8
  • All control/feedback features use NetworkTables, so the same robot can be controlled using the SmartDashboard instead if needed
    • SendableChooser compatible implementation for mode switching
  • Animated robot drawing that shows how many frisbees are present, and tilts the shooter platform according to the current angle the platform is actually at.
  • Allows operators to select different modes of operation for the robot using brightly lit toggle switches
  • Operators can choose an autonomous mode on the dashboard, and set which target the robot should aim for in modes that use target tracking
  • Switches operator perspective when robot switches modes
  • Simulated lighted rocker switches to activate robot systems
  • Logfiles written to disk with errors when they occur

Target acquisition image processing features:

  • Tracks the selected targets in a live camera stream, and determines adjustments the robot should make to aim at the target
  • User can click on targets to tell the robot what to aim at
  • Differentiates between top/middle/low targets
  • Partially obscured targets can be identified
  • Target changes colors when the robot is aimed properly

Fully integrated realtime analysis support for target acquisition:

  •  Adjustable thresholding, saves settings to file
  • Enable/disable drawing features and labels on detected targets
  • Show extra threshold images
  • Can log captured images to file, once per second
  • Can load a directory of images for analysis, instead of connecting to a live camera

So, as you can see, lots of useful things. We’re releasing the full source code for it under a GPL license, so go ahead, download it, play with it, and let me know what you think! Hope you find this useful.

A lot of people made this project possible:

  • Some code structuring ideas and PyGTK widget ideas were derived from my work with Exaile
  • Team 341 graciously open sourced their image processing code in 2012, and the image processing is heavily derived from a port of that code to python.
  • Sam Rosenblum helped develop the idea for the dashboard, and helped refine some of the operating concepts.
  • Stephen Rawls helped refine the image processing code and distance calculations.
  • Youssef Barhomi created image processing stuff for the Kwarqs in 2012, and some of the ideas from that code were copied.

The included images were obtained from various places:

  • Linda Donoghue created the robot image
  • The fantastic lighted rocker switches were created by Keith Sereby, and are distributed with permission.
  • The green buttons were obtained via google image search, I don’t recall where

Download it on my FRC resources page!

Posted in Announcements, FIRST, Innovation, PyGTK, python, Software | 1 Comment »

Notepad++ plugin to enable unindent on backspace key

Monday, December 3rd, 2012

When I first started using editors, I was a tabs person. I always set the tab size to 4 since I liked the way that looked, and used tabs instead of spaces. And it wasn’t really a philosophical reason behind it, it was very simple actually:

I really hate hitting backspace 4 times to unindent

Seems simple enough. Yes, I know you can use SHIFT-BACKSPACE or other voodoo to make it work, but I want to hit *one* key.

Because of this, I always used 4-space tabs instead of inserting spaces instead of tabs. However, at some point I started writing a project in python, and python tends to encourage spaces vs tabs. Notepad++ is my favorite editor, so I decided to write a plugin for Notepad++ to make using spaces a lot less annoying — and in particular, to make it so that when I hit backspace, it would magically eat the spaces back to the next indentation level.

In the process of trying to do this, I discovered that the Scintilla editing component actually has this functionality built-in — you just need to enable SCI_SETBACKSPACEUNINDENTS on the editor component. So I took the sample plugin for Notepad++ and added two lines of code that enabled this functionality, and it works great! Ever since then, I’ve used spaces instead of tabs, since I really can’t tell the difference — and that’s how I like it 🙂

I actually wrote this thing a rather long time ago, and never got around to actually cleaning it up and releasing it. I’ve decided to release it in its present form, since it’s so useful and very simple. It would be even better if Notepad++ had an option to enable this, but this works just as well until that happens.

This plugin is known to work in versions of Notepad++ since 4.x, and works in Notepad++ running under Wine also. Download the Notepad++ tabs plugin at my usual software page.

Note: I’ve now posted the code as a project on Github

Posted in Innovation, Notepad++, Software | 9 Comments »

BPM Counter Plugin for Exaile 0.3.2

Monday, October 3rd, 2011

I’ve been starting to use my desktop machine at home a lot lately, so I’ve been looking for a cross-platform audio player that I can use that doesn’t annoy me. After a long search (and discarding most of the linux audio players: amarok, banshee, etc as too annoying or not the right features or whatever) I finally stumbled across Exaile, which not only doesn’t annoy me greatly, but it’s written in python so its way easy to modify and figure out what it does 🙂

I’ve been doing a bit of swing dancing lately (in particular Lindy Hop), so I’ve been gathering music together to listen to, and I need BPM for the music… one thing Exaile did not already have was a BPM counter, so I wrote a manual beat counter for it. You can write plugins for Exaile, though the documentation is rather sparse. I must say though, using the GLADE widget editor thing has to be the most annoying GUI design tool ever…


Download it at my usual software site.

PS: In case anyone asks, I’m not interested in writing an automated beat counter… this works well enough for me 🙂

Posted in exaile, python, Software | No Comments »

Changing variables using a web interface and embedded HTTP server

Tuesday, April 21st, 2009

When walking around during the Boston Regional, I had been talking to some people about code, and they mentioned that LabView was great because they could tune their PID controllers on the fly while the robot was operating. So I thought to myself, “why can’t I do this with C++?”. And… so I did. WebDMA was created to allow our FIRST Robotics team to tune our robot in an easy to use and intuitive way via any modern web browser.

Using C++ operator overloading, WebDMA provides proxy objects that your application can use as normal variables which can be manipulated or displayed by your application via a configurable jQuery/javascript powered Web 2.0 interface hosted by an lightweight embedded web server.

Despite that WebDMA was specifically created for use in FIRST Robotics on the NI-cRio/vxWorks platform, it uses the Boost ASIO portable networking library and Boost Thread portable threads library and is usable on any platform supported by these Boost libraries (tested on Boost 1.38, requires a patch for vxWorks).

A non-functional (but very shiny) demo of the interface is available at http://www.virtualroadside.com/botface/index.html

Visit the Google Code project site for WebDMA

Update: Go here for a video: http://www.virtualroadside.com/blog/index.php/2009/04/25/webdma-demo-video/

Posted in FIRST, Howto, Innovation, Javascript, jQuery, Software, WebDMA | No Comments »

WPILib Test Harness Released!

Saturday, March 14th, 2009

So after *far* too much time spent on this, I am happy to announce the release of my WPILib Test Harness! For those who aren’t aware, WPILib is the name of the C++ library that FIRST gives to FIRST Robotics Teams to control the robots. The control system is a PowerPC running on a cRio platform designed and produced by National Instruments.

It had occurred to me towards the end of the build season this year that it was really annoying that I could only easily test code while our team was meeting, and of course I stay up late (as evidenced by this post time). So I wrote some stubs, but at some point I realized it would just be easier to make a giant stub for WPILib, since its just a set of classes with some hardware interfaces. It was extremely useful, and as I evolved it I was even able to find bugs in WPILib and the program for our bot.

At some point I decided to bring it in the direction that it is now, and I’ve spent the last week adding things and making the GUI a bit shinier. I’m pretty happy with the results so far, but theres a long way to go before every bot program will run under it. There’s a lot of things that do work, but there’s also a lot of things that don’t work.
(more…)

Posted in FIRST, Software, WPILib | 3 Comments »

Debug STL errors with __GLIBCXX_DEBUG

Friday, March 13th, 2009

Ever got one of those *really* annoying glibc errors that don’t give you a whole lot of information except “something bad happened, and it was memory related, but we’re not going to tell you where the problem is!”? You know, in the form of

*** glibc detected *** ./test: double free or corruption (out): 0x0000000001e52440 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f27a9fd5a58]
/lib/libc.so.6(cfree+0x76)[0x7f27a9fd80a6]
./test[0x401e51]
./test[0x401e83]
./test[0x402118]
./test[0x402159]
./test[0x40155b]
./test[0x40167d]
/lib/libc.so.6(__libc_start_main+0xe6)[0x7f27a9f7a466]
./test[0x400cd9]
======= Memory map: ========
... and so on ...

I’ve encountered this cryptic and annoying error (I mean c’mon, usually the memory addresses referenced in the trace don’t resolve to anything in gdb) a few times for really random and weird memory-related errors, many times when I’m doing something with the STL.

Turns out, you can define __GLIBCXX_DEBUG in your program (using -D__GLIBCXX_DEBUG on the g++ command line), and instead of an annoying message you may get an assert that is far more helpful and looks like this:

/usr/include/c++/4.3/debug/vector:267:error: attempt to access an element
    in an empty container.

Objects involved in the operation:
sequence "this" @ 0x0x7fff5ee0f420 {
  type = NSt7__debug6vectorIiSaIiEEE;
}

Now at least you know what you did wrong. However, for optimal usage you should be attached with GDB, and then as soon as the error occurs you can do a backtrace with the debugger and you should know exactly where the bug is. The only caveat is that there is a noticeable performance hit when you enable this (especially if you use the STL heavily), so in general you don’t want to define it all the time.

Hope that saves you some time! I know it helped me!

Posted in GCC, Software | No Comments »

C++ template-fu: delete_if

Sunday, November 9th, 2008

I’ve been doing a lot of work with templates lately, for my R* tree implementation (which is morphing into a R* tree and a B tree library) and some things that I’ve been playing around with for Roadnav, especially in regards to serialization.

Heres a neat snippet that I created:

template <bool>
struct delete_if {
	template <typename T>
	static void del(T t) { delete t; }
};

template <>
struct delete_if
{
	template <typename T>
	static void del(T t) { /* no-op, doesn't delete */ }
};

This uses a partial specialization to do the decision portion, and a static member template to eliminate some of the typing (you could get rid of the member template, and you would end up calling delete_if<type, delete_item>::del ).

So what this is intended for is those cases where you are implementing some generic container, and you don’t know whether the type you have is a pointer or not. You *could* use a normal bool as a parameter:

if (delete_it) delete var;

But primary problem with this is that it won’t compile if the type isn’t a pointer. So a sample of something similar to what I’m using it for is something like this:

template < typename T, bool delete_item >
class test {
public:

	~test()
	{
		// only release the item if the client specifies
		// that they want to
		delete_if<delete_item>::del(val);
	}

private:
	T val;
};

Obviously this is just a really simple example of how you can use this. One could also use boost::is_pointer to detect whether the type is a pointer, and pass *that* to delete_if while using a ‘normal’ variable to determine whether the client wants to have the item be deleted or not. I leave that as an exercise for the reader.

Posted in Snippet, Software | No Comments »

« Older Entries