Recent Updates

  • February 08, 2014 Permalink

    Converting Bytes to Hex Strings In Java

    There seems to be a lot of misinformation and overly complicated solutions to the problem of converting byte values (and arrays) to hex strings in Java floating around on the internet. In this post I am to clear up the confusion and provide a clean solution to the problem. To illustrate I will provide a fairly straightforward approach, but I'll provide a bit more detail to help illustrate why it works.

    First off, what are we trying to do? We want to convert a byte value (or an array of bytes) to a string which represents a hexadecimal value in ASCII. So step one is to find out exactly what a byte in Java is:

    The byte data type is an 8-bit signed two's complement integer. It has a minimum value of -128 and a maximum value of 127 (inclusive). The byte data type can be useful for saving memory in large arrays, where the memory savings actually matters. They can also be used in place of int where their limits help to clarify your code; the fact that a variable's range is limited can serve as a form of documentation.

    What does this mean? A few things: First and most importantly, it means we are working with 8-bits. So for example we can write the number 2 as 0000 0010. However, since it is two's complement, we write a negative 2 like this: 1111 1110. What is also means is that converting to hex is very straightforward. That is, you simply convert each 4 bit segment directly to hex. Note that to make sense of negative numbers in this scheme you will first need to understand two's complement. If you don't already understand two's complement, you can read an excellent explanation, here: http://www.cs.cornell.edu/~tomf/notes/cps104/twoscomp.html

     

    Converting Two's Complement to Hex In General

    Once a number is in two's complement it is dead simple to convert it to hex. In general, converting from binary to hex is very straightforward, and as you will see in the next two examples, you can go directly from two's complement to hex.

    Examples

    Example 1: Convert 2 to Hex.

    1) First convert 2 to binary in two's complement:

        2 (base 10) = 0000 0010 (base 2)

    2) Now convert binary to hex:

        0000 = 0x0 in hex
        0010 = 0x2 in hex

        therefore 2 = 0000 0010 = 0x02.

    Example 2: Convert -2 (in two's complement) to Hex.

    1) First convert -2 to binary in two's complement:

        -2 (base 10) = 0000 0010 (direct conversion to binary)
                       1111 1101 (invert bits)
                       1111 1110 (add 1)
        therefore: -2 = 1111 1110 (in two's complement)

    2) Now Convert to Hex:

        1111 = 0xF in hex
        1110 = 0xE in hex

        therefore: -2 = 1111 1110 = 0xFE.

     

    Doing this In Java

    Now that we've covered the concept, you'll find we can achieve what we want with some simple masking and shifting. The key thing to understand is that the byte you are trying to convert is already in two's complement. You don't do this conversion yourself. I think this is a major point of confusion on this issue. Take for example the follow byte array:

        byte[] bytes = new byte[]{-2,2};

    We just manually converted them to hex, above, but how can we do it in Java? Here's how:

    Step 1: Create a StringBuffer to hold our computation. 

        StringBuffer buffer = new StringBuffer();

    Step 2: Isolate the higher order bits, convert them to hex, and append them to the buffer

    Given the binary number 1111 1110, we can isolate the higher order bits by first shifting them over by 4, and then zeroing out the rest of the number. Logically this is simple, however, the implementation details in Java (and many languages) introduce a wrinkle because of sign extension. Essentially, when you shift a byte value, Java first converts your value to an integer, and then performs sign extension. So while you would expect 1111 1110 >> 4 to be 0000 1111, in reality, in Java it is represented as the two's complement 0xFFFFFFFF!

    So returning to our example:

        1111 1110 >> 4 (shift right 4) = 1111 1111 1111 1111 1111 1111 1111 1111 (32 bit sign-extended number in two's complement)

    We can then isolate the bits with a mask:

        1111 1111 1111 1111 1111 1111 1111 1111 & 0xF = 0000 0000 0000 0000 0000 0000 0000 1111
        therefore: 1111 = 0xF in hex.

    In Java we can do this all in one shot:
     

        Character.forDigit((bytes[0] >> 4) & 0xF, 16);

    The forDigit function just maps the number you pass it onto the set of hexadecimal numbers 0-F.

    Step 3: Next we need to isolate the lower order bits. Since the bits we want are already in the correct position, we can just mask them out:

        1111 1110 & 0xF = 0000 0000 0000 0000 0000 0000 0000 1110 (recall sign extension from before)
        therefore: 1110 = 0xE in hex. 

    Like before, in Java we can do this all in one shot:

        Character.forDigit((bytes[0] & 0xF), 16);

    Putting this all together we can do it as a for loop and convert the entire array:

        for(int i=0; i < bytes.length; i++){
            buffer.append(Character.forDigit((bytes[i] >> 4) & 0xF, 16));
            buffer.append(Character.forDigit((bytes[i] & 0xF), 16));
        }

    Hopefully this explanation makes things clearer for those of you wondering exactly what is going on in the many examples you will find on the internet. Hopefully I didn't make any egregious errors, but suggestions and corrections are highly welcome! 

     
  • January 18, 2013 Permalink

    Little Known Secrets of Java: Better Constants

    Sometimes, every once in a while, I come across something in the dark recesses of Java that surprises me. Most recently I was reading through the code for Netflix's Curator library and I came across a little gem so interesting, so tantalizing, that I had to read the JDK's javadocs just to be sure it was right.

    So what is this little secret? Ladies and Gentleman, let's talk about constants.

    What's in a Constant?

    First, let's clarify what I mean by a constant. Let's say that we have some useful value we wish to define. Furthermore, let's say this value is something I intend not to change during the execution of my program. You might even go so far as to say it is constant.

    Our First Try

    A first attempt to define such a value might look something like this:

    public class Foo {
     
     
         // set the session timeout to 1 hour (in ms)
         public static final int SESSION_TIMEOUT = 60*60*1000;
     
     
         // other methods elided // 
     
    }

    Wonderful. Welcome to junior programmer land. You can pick up your ID badge with Jane in HR.

    A Better Idea

    While the above is quick and dirty and gets the job done, alas, Dijkstra would not like it.

    dijkstra.jpg

    Perhaps we can improve on this. First, let's examine: is that constant really a constant?

    Is that Constant REALLY a Constant?

    It's pretty presumptuous of us to assume we know what is constant, isn't it? Like, it's a tad cheeky. See, when I think of a constant I think of, say: π. These are things that, given the task of manipualting them, we can safely code them into our programs–hard code even–and never think twice about it.

    But alas, what about our session timeout? If you consider the true purpose of session timeout, you will find that it is really something that is subject to change, based on some arbitrary business rules. In fact, session timeout is really a variable; a variable that doesn't change at runtime.

    Defining Integer Constants

    So how to do this in Java? XML files? Databases? Property files? Close, but then we have to build all that code to read the properties, serialize them back out to files… Not ideal. What about our old friend System.getProperty()? Close, but then we have to do String conversion anywhere we want to use it. Things brings us to…

    The Secret

    The Java implementation of Integer offers an often overlooked method: getInteger(). Here's how you use it, by way of an example, taken from Netflix Curator:

     
    public class CuratorFrameworkFactory
    {
        private static final int        DEFAULT_SESSION_TIMEOUT_MS = Integer.getInteger("curator-default-session-timeout", 60 * 1000);
        private static final int        DEFAULT_CONNECTION_TIMEOUT_MS = Integer.getInteger("curator-default-connection-timeout", 15 * 1000);
     
        private static final byte[]     LOCAL_ADDRESS = getLocalAddress();
     
        private static final CompressionProvider        DEFAULT_COMPRESSION_PROVIDER = new GzipCompressionProvider();
        private static final DefaultZookeeperFactory    DEFAULT_ZOOKEEPER_FACTORY = new DefaultZookeeperFactory();
        private static final DefaultACLProvider         DEFAULT_ACL_PROVIDER = new DefaultACLProvider();
        private static final long                       DEFAULT_INACTIVE_THRESHOLD_MS = (int)TimeUnit.MINUTES.toMillis(3);
     
    // other methods elided //
     
    }

    Essentially, what the getInteger() method does is allow you to access a value in the system properties, but without littering your code with the String to Integer conversion. What's better is you can name these values and even specify default arguments. Very nice indeed. Oh, and for those of you who don't know how to set these properties, you can do it a number of ways, but most easily on the command line when invoking your Java program.

    Example: java -Dmyproperty=myvalue MyClass

    You can learn more by looking at the documentation for System.getProperty().

    Wrapping Up

    So there you have it. One little function in the Integer class that may just end up making your code just a little bit more awesome.

    And, because I feel like I have to say it: Please don't write me about using enums. Trust me, I know they exist. I just think that for this application the getInteger() approach is much better.

     
  • December 12, 2012 Permalink

    How to Fix Emacs CPU Problems within Gnome Shell on Ubuntu 12.10

    A strange problem seems to afflict Ubuntu 12.10 when you are running Gnome Shell. When running a fresh gnome session, if you fire up Emacs, the gnome-shell process will start pegging the CPU. Badness.

    After several hours of hacking, it looks like it is simply a matter of the Emacs package having been compiled against the GTK 2 libraries. To fix it, download a new source package of Emacs from the gnu.org site and configure it with the following options:

    # ./configure --with-x-toolkit=gtk3 --prefix=/opt/sw/emacs-24.3
    

    Of course you will probably need to install the dev headers and so on for GTK3, and the other associated libraries. To do that:

    
    # sudo apt-get install libgtk-3-dev libjpeg-dev libgif-dev libtiff-dev libXpm-dev
    

    Enjoy!

     
  • July 09, 2012 Permalink

    Opening a Garage Door With Erlang... and a Motorcycle?

    I ride a motorcycle. It's great except for when it comes to the little things that most cagers (people who ride in cars) take for granted. Take for example the simple problem of opening your garage door when you come home. You just clip your clicker onto your sun visor and never think about it again. But if you ride a motorcycle things aren't so easy. In the best case you can throw a clicker in your bag but you'll have to fumble with it when you get home and try to get at it with your gloves on. Worst case you just get off your bike, open the door, ride the bike in.

    Kind of a pain.

    While riding around one day I got the idea for a garage door opener which could work based only on the sound of my motorcycle's horn. Even cooler, I built it in Erlang. I just finished my first prototype of the system and thought I'd share a little video that shows the device in action and even gives you a peek behind the scenes of how it all works. I'll be posting more pictures here, but for now, check out the video!

     
  • February 09, 2012 Permalink

    Parsing Nested JSON in Haskell with AESON

    Parsing JSON in Haskell is one of the most painless experiences one can have as far as parsing goes... except for one little thing. Aeson, which is one of the best JSON libraries for Haskell out there, is so under-documented so as to make it almost impossible to use for all the the most trivial case.

    My case was that I needed to parse the Yammer JSON feed, which had quite a few nested fields. The schema looked, basically like this:

     
    {
      "messages": [
        {
          "client_url": "https://www.yammer.com/",
          "created_at": "2011/03/28 20:39:12 +0000",
          "system_message": false,
          "body": {
            "parsed": "message with photo attachment.",
            "plain": "message with photo attachment."
          },
          "sender_type": "user",
          "network_id": 104604,
          "thread_id": 84402777,
          "web_url": "https://www.yammer.com/yammerdeveloperstestcommunity/messages/84402777",
          "direct_message": false,
          "id": 84402777,
          "url": "https://www.yammer.com/api/v1/messages/84402777",
          "client_type": "Web",
          "message_type": "update",
          "sender_id": 4022984,
          "replied_to_id": null,
          "attachments": [
            {
              "type": "image",
              "content_type": "",
              "uuid": null,
              "web_url": "https://www.yammer.com/yammerdeveloperstestcommunity/uploads/857663/Firefly.jpg",
              "y_id": 857663,
              "image": {
                "thumbnail_url": "https://www.yammer.com/api/v1/file/857663/Firefly.jpg?view=thumbnail",
                "url": "https://www.yammer.com/api/v1/file/857663/Firefly.jpg",
                "size": 0
              },
              "name": "Firefly.jpg",
              "id": 974915
            }
          ],
          "liked_by": {
            "count": 0,
            "names": []
          },
          "privacy": "public"
        },
        {
          "client_url": "http://www.yammer.com",
          "created_at": "2011/03/25 00:49:29 +0000",
          "system_message": false,
          "body": {
            "parsed": "new test message 1",
            "plain": "new test message 1"
          },
          "network_id": 104604,
          "thread_id": 83957686,
          "sender_type": "user",
          "direct_message": false,
          "web_url": "https://www.yammer.com/yammerdeveloperstestcommunity/messages/83957686",
          "id": 83957686,
          "url": "https://www.yammer.com/api/v1/messages/83957686",
          "client_type": "testingtest",
          "sender_id": 4022984,
          "replied_to_id": null,
          "message_type": "update",
          "liked_by": {
            "count": 0,
            "names": []
          },
          "attachments": [],
          "privacy": "public"
        }
      ],
     
        {
          "type": "user",
          "stats": {
            "followers": 1,
            "updates": 14,
            "following": 2
          },
          "web_url": "https://www.yammer.com/yammerdeveloperstestcommunity/users/mikealrogers-guest",
          "mugshot_url": "https://assets3.yammer.com/images/no_photo_small.gif",
          "url": "https://www.yammer.com/api/v1/users/4022984",
          "full_name": "mikeal",
          "name": "mikealrogers-guest",
          "state": "active",
          "job_title": "Test Title",
          "id": 4022984
        },
        {
          "type": "user",
          "stats": {
            "followers": 1,
            "updates": 4,
            "following": 2
          },
          "web_url": "https://www.yammer.com/yammerdeveloperstestcommunity/users/mknopp",
          "mugshot_url": "https://assets1.yammer.com/user_uploaded/photos/p1/0141/2640/n1644278019_46479_62_small.jpg",
          "url": "https://www.yammer.com/api/v1/users/1452329",
          "full_name": "Matt Knopp",
          "name": "mknopp",
          "state": "active",
          "job_title": null,
          "id": 1452329
        }
     
      ]
    }

    Not so nice. Particularly, I was interested in extracting each message, a list of users, as well as data on likes. Turns out this isn't too hard in Haskell.

    First, we define three types to hold this data.

    {-# LANGUAGE OverloadedStrings #-}
     
    module Yamulator where
     
    import           Control.Applicative
    import           Control.Monad
    import           Data.Aeson
    import           Data.Aeson.Types
    import qualified Data.HashMap.Strict  as HM
    import qualified Data.ByteString.Lazy.Char8 as C
    import qualified Data.Text as T
     
     
    data Message = Message {
          mid       :: Integer,
          plainText :: T.Text,
          byUserId       :: Integer,
          likes   :: Integer,
          inReplyTo :: Maybe Integer,
          createdAt :: T.Text
     
    } deriving (Eq, Show)
     
    data User = User {
          name :: T.Text,
          userId  :: Integer
     
    } deriving (Eq, Show)
     
     
    data Yammers = Yammers {
          messages :: [Message],
          users    :: [User] 
    } deriving (Eq, Show)

    This part is important. Note that Yammers contains [Message] and [User].

    Next, we define the instances.

    instance FromJSON Yammers where
      parseJSON (Object o) = do
          messages <- parseJSON =<< (o .: "messages")
          users <- mapM parseJSON . filter (\(Object ref) -> HM.lookup "type" ref == Just (String "user")) =<< o .: "references"
          return $ Yammers messages users
      parseJSON _ = mzero
     
    instance FromJSON Message where
        parseJSON (Object v) = Message <$>
                              v .: "id" <*>
                              ((v .: "body") >>= (.: "plain")) <*>
                              v .: "sender_id" <*>
                              ((v .: "liked_by") >>= (.: "count")) <*> -- note how we can keep drilling into nested structures like this.
                              v .:? "replied_to_id"  <*>
                              v .: "created_at"
     
        parseJSON _ = mzero

    Then, reading it in is a piece of cake. We can define a method like the following to serialize things out:

     
    decodeYammers :: C.ByteString -> Maybe Yammers
    decodeYammers response = decode response
     
    decodeUsers :: C.ByteString -> Maybe [User]
    decodeUsers response = decode response

    It doesn't get any easier. I hope that helps some other Haskellers out there. Many thanks to #haskell for all the help while I was putting this together!

     
  • September 26, 2011 Permalink

    Who Moved My UI? Or: When is it Okay to Change Something? Thoughts on Rethinking the Rule Builder

    Nothing makes my nerd heart beat harder than good interface design. And one of the most interesting aspects about interface design is the issue of reinventing or improving well-established UI elements and conventions.

    But while everyone agrees that designs can always be improved it’s not always clear what that means and more importantly, if it’s even possible.

    I’ve overheard many heated arguments in which someone eventually says something like: “Everyone expects x to work like y, so you can’t change it!!” (This is usually the deathblow to the other person. They walk away, tail between legs, to go silently Facebook-stalk their ex-girlfriend.)

    And you know what, this sort of system seems to work. Most of the time. But that’s only because most of the time we come up with terrible, terrible ideas. But sometimes new controls and ways of doing things do need to get designed. How to settle this age-old argument?

    The first thing is to understand that it is okay to change things. This should be obvious. But when? Companies like Apple have invented entire new widget standards with phenomenal success while others have floundered. What did they do differently?

    As a framework for discussing that, let’s start by classifying our UI elements. I suggest the following classification for thinking about UI elements:

    • Semiotic Elements (eg: Radio Buttons vs Circles)
    • Paradigm Elements (eg: Click/Drag vs Swipe/Touch)
    • Functional Elements (eg: Clicking on a table header to sort vs pulling down a drop down.)

    Naturally, each of these refers to either general or specific parts of a user experience as indicated by the image, below.


    The Semiotic Element – eg: Radio Buttons vs Circles.

    The most core relationship a UI element has to its user is its Semiotic meaning. This can be a property as simple as “is this element selected?” or “what does activating this control mean?” Controls in this category are things like check boxes, radio buttons, highlighted table cells. Why? Because they all mean one thing: “This thing right here.” And though, to a programmer/designer, the difference between a radio button and a checkbox is oceanic, to a user, all they mean to say when the click or check something is “this thing right here.”

    If you want to convince yourself of this, check out the image, below:

    The only difference between these two is the fact that on the programmatic form, you are not allowed to select more than one. On the paper form you are on the honor system.

    So can components of this category be reinvented? Of course! With the following caveat:

    Never change the meaning of a click or a highlight or placement meaning “This thing right here.”

    Imagine if to select something a user had to unselect the item he wanted? Workable, but chaos. And the reasons for this are probably so hardwired into our brains that it’s probably not even worth exploring the alternatives. That is, if your goal is to make usable user interfaces.

    The Paradigm Element – eg: Click/Drag vs Swipe/Touch

    Paradigm elements have to do with the way we think about things. A good example of something in this category is the concept of basing the storage of a hard disk off of 1950’s office technology: Files and folders.

    While “files and folders” are an overwhelmingly popular way to think about your data, it’s hardly universal. For example, Plan 9 doesn’t really use “file and folders” at all. And of course, more famously, the iPhone, from a user perspective, doesn’t even have a filesystem!

    It’s surprising to realize it, but even things like the filesystem, or even files can be changed. That’s because as humans we are really, really good at inventing new ways of doing things. In fact, the term “Paradigm Shift” has practically become as bourgeois as the word bourgeois!

    So paradigm shifts are ok. Just so long as the paradigm you replace it with is as complete as the one it replaces—or has really good reasons for not being.

    The Functional Element eg: Clicking on a table header to sort vs pulling down a drop down.

    There are lots of ways to skin a cat. Or at least that’s what elements belonging to the Functional Element group think.

    Examples of this type of element are things like the OSX Dock, the Windows Start Menu, and sortable tables—all things that can be done a million ways. Also, all elements that use a combination of elements to provide the user with a much higher level function than saying “this thing right here.”

    Functional components give the user the ability to say “I want all of the albums by 1972 David Bowie from the Ziggy Stardust tour, but I don’t know how to spell “Bowie” or what year the Stardust tour was.” Or even simply, “Is my package on time?”

    These are arguably some of the most malleable components. Mainly because in creating their own language they use so much of the language of the Paradigm and Semiotic elements. For example, clicking on a table header might cause a highlight, which then highlights the sorted column. The meaning of “this thing right there” is preserved.

    Really, the general rule of an element of the Functional type is simply that they allow the user to perform the function they expect to perform or that the function is possible given proper training.

    This is where the problems start.

    Designers and Programmers sense that this element is “ok” to change and they do. And because there appear to be no rules for doing so, the results are often terrible, unworkable, illogical interfaces that make users feel stupid and cost your company or project money and users.

    Elements of this category are often incredibly hard to make and often represent the bulk of work done by programming and design teams.

    My Dilemma – Designing the Rule Builder

    So what is the practical impact of all this you ask? It’s something you can apply to your day-to-day work.

    I recently had to do this on a control I was making: A Rule Builder.

    The goal was to make a control, similar to the way a mailbox filter builder that would allow a user to build a very complex set of interactions.

    So the question for me was: is it okay to reinvent this control? Is it even possible? Seeing as a rule builder would fall under a “Functional” element, I decided that it would be.

    I asked myself: What is wrong with the current paradigm? The answer? A lot. The current paradigm works something like this:

    One builds a “ruleset” by adding together these elements in rows. For an example of one of the better implementations of this, check out the Apple Mail filter builder:

    Take the example of Apple Mail’s Rule Builder:

    Here they’ve done nice stuff like added natural user language to the process and placed all the controls in the right places and even made the action of the filter very, very clear. But there were several things I didn’t like about Apple’s approach.

    First, Apple’s implementation relies on over simplification of the problem. One cannot group together the logic of the filter.

    Second this design uses the same “double drop down” approach that the standard UI convention uses. This itself has a number of problems. Mainly, a user has to click twice, and must then move between the “what to filter” and “how to compare” options of the form if they wish to get a complete picture of all of their filtering options. In other words, a user has no context.

    And then there’s other approaches, some of which are just too bad for words:

    Kill me now.

    So in designing my own rule builder, I sought to overcome the flaws I found in Apple’s builder, as well as add a few more constraints. The original design document I wrote up went something like this:

    1. The element should allow the user to use natural language to describe what they want.
    2. It should absolutely minimize the number of clicks needed to complete the action
    3. It should be built in Javascript/CSS/HTML and require no images and a very minimal amount of markup.
    4. It should give the user the ability to see ALL of their options at once.
    5. The value of the control should be expressed as in a single value as a aggregate – in my case JSON.
    6. The state of the control should be easy to restore.
    7. The control should use all standard html form elements.

    And you can see the result of all this work, below:

    In the rule builder, the first drop down is replaced with a string label, and the user, when they click the drop down, can select from a hierarchical menu that shows them all of their options at once.

    A couple of simple changes, but a lot better. In fact, I liked the control so much, I decided to open source it and a few other controls as a part of my Prototype-Widgets project.

    You can see the source as well as an interactive demo over on github, here: http://jsinglet.github.com/Prototype-Widgets/

    Well there you have it. Thanks for reading! Happy hacking!