Web Innovators Group 17 in Cambridge

April 3rd, 2008

I got a chance to attend the Web Innovators Group in Cambridge last night, and it was an absolute blast! This was my first time attending, and the experience was very, very different from other events I’ve attended in the area.

The event starts off with a bit of a networking mixer in the bar area of the Royal Sonesta in Cambridge, and then into the ballroom for a set of “main dish” and “side presentations.” Main dishers get about 10 minutes to demo their latest and greatest creations (PowerPoint seems to be strictly forbidden… if you don’t have something to demo that makes people go “Oooh!” and “Aaah!”, don’t present). Side dish folks get a minute-long elevator pitch, directing people to a couple of tables in the back for demos after the presentations are over. After that, it’s basically just a bunch of techies and investors hanging out for a couple of hours over a couple of drinks… simply awesome.

There must have been at least 500 folks in attendance… far larger than any other non-conference event I’ve been to in the area, but it didn’t FEEL like a large, impersonal event. It was very open and social.

Probably the biggest shock for me was that I never once ran across a service provider pimping their own services. I’m sure they were there, but there were so many other folks that I never ran across one. Granted, the service providers are often the folks that end up sponsoring these types of events, and they keep the gears moving, but after you’ve talked to your nineteenth or twentieth IP attorney, it gets a little tiresome. If I was looking for a new patent attorney, that would be one thing, but I’ve been very happy with Brett.

I once knew a guy who said he would meet a couple of attorneys and accountants at the networking events, and then introduce them to each other with the line “I think you guys might be a good match!”, and then he would wait and see how long the “dueling salesman” scenario would play out before they figured it out! I doubt it’s a true story, but it certainly would be interesting to watch. Anywho, I digress…

The real thing that tipped me off to attending was that Artur Janc was one of the presenters. Artur and I both went to WPI, and we crossed paths about four times at various events without ever actually “meeting”. It was great to spot him (Artur’s pretty easy to spot in the crowd… he has the most bad-ass hair you’ll ever find on an entrepreneur), and catch up a bit. His new project is absolutely awesome; it’s called Lingro, and it’s “the coolest dictionary known to hombre!“.

Lingro.com

It’s a service that let’s you visit a web page in any language, and all of the words become “clickable” to translate and/or lookup their definitions in a nice, unobtrusive panel on the page. The service is free for general usage, but if you’re someone who is actively trying to learn a new language, they’ll be offering an upgraded version of the service that will make life easier for you. Watch out Rosetta!

I thoroughly enjoyed all of the main and side dish presenters… and without further delay, here are my six-word summaries for each dish:

  • Good2Gether - non-profits meet for-profits for social profit.
  • PicMe Photo Sharing - flying photo stacks sharable with friends.
  • Jack Cards - e-cards suck; easy hand-written cards rule.
  • Traackr - measure your pimp status online. word.
    • (PS - I miss the days when people used vowels… I’m convinced folks are just trying to get a leg-up in Wheel of Fortune down the line)
  • Moborazzi - visual mobile twitter… visual mobile twitter.
  • Yamli - search in Arabic using English letters.
  • Entrecard - blogger networking meets targeted sidebar ads.
  • Flimp - these folks will make a fortune.
  • StylePath - hopefully my wife never finds this!
    • Okay that one’s a joke, but it was my first thought! The real six-word summary is: visual search when words aren’t enough.

Very much looking forward to the next event, currently scheduled for July. I hope to see you there!

WebReboot Mention in PC Magazine

April 2nd, 2008

As always, I get my news fresh from Ken Jamaca at Silverback Migration Solutions, who let me know that we had a mention in this month’s PC Magazine!

So, if you see this in your mailbox:

April 2008 Cover

Turn to page 96, and we’re in the Five Steps to a Greener SMB article.

April 2008 Article

Click here for the full article text on PC Magazine’s web site.

Many thanks to Oliver Rist from PC Magazine for the mention!

REST Web Service Code Generation

March 24th, 2008

I’m really excited about this:

http://code.google.com/p/rest-api-code-gen/

Embedded SOAP Web Services: Where’s the Value?

March 22nd, 2008

Steve Vinoski sent over the following comment on my previous article:

… WS-* is *not* preferable to REST. WSDL creates a tight coupling between the client and service which no amount of XML versioning can hope to alleviate. Code generation is brittle, can create unnecessary data definition dependencies, and inhibits extensibility. The industry has by and large learned this well over the past few years, which is why I mentioned 2004 in my blog about this. WS-* is practically dead.

WSDL and code generation try to give distributed systems the appearance of being local, ala RPC, only for the purpose of making the developer’s life easier, but in doing they create big problems. That’s why I mentioned 1994 and pointed to the “Note on Distributed Computing” paper, which covers this quite well.

You might want to read my latest IEEE Internet Computing columns on this topic, as those who don’t know history are doomed to repeat it:

Assuming you apply it correctly, REST is far, far preferable to what you’ve described…

Here are my thoughts…

Hi Steve,

Thanks for taking the time to write back. I appreciate it. I whole-heartedly agree with each of your points:

WSDL creates tight coupling, and this is bad.

Code generation is brittle, can create unnecessary data definition dependencies, and inhibits extensibility. All bad.

I know you’ve touched upon how REST is intrinsically more scalable than WS-*. I agree.

I believe my key failing here is that I failed to adequately explain the problem I am trying to solve.

I make embedded systems. More specifically, I make a device that can remotely reboot or power control a server in the same fashion as pressing the reset or power switch on the front of the server’s chassis. The device, named the WebReboot, can also measure the temperature inside a server, and tell you if the server is powered on or off. It’s core application is the remote recovery of crashed servers.

The interface for the WebReboot is tightly coupled to the hardware. We use a relay to control the power switch or reset switch of a server. We use a thermistor to measure temperature. The hardware has remained relatively unchanged for quite a long time, largely because it is simple, reliable, and cost-effective. Since the hardware has remained relatively unchanged, the interface has remained relatively unchanged. The extent of changes are limited to adding complimentary functionality that does not affect existing functionality (for instance, if I added a humidity sensor). In this scenario, I don’t think the tight coupling forced upon you by WSDL really makes you any worse off.

Scalability is largely not a concern for this application. In practice, it is not necessary to have many folks or systems trying to reboot a crashed server or retrieve a server’s temperature simultaneously.

Since my functionality is limited to what’s implemented in relatively simple physical hardware (e.g., sensors and relays), I’ve found limited application for complicated data structures and dependencies in controlling this hardware. I’ve also found limited application for extensibility. I’ve come up with many hypothetical ways for their application, but I haven’t found a need to use them. For these reasons, I haven’t in practice found the generated code for Java, Python, PHP or .NET to be brittle.

The biggest challenge we’ve had as a small company (< 10 people) competing in a large market (IT hardware) is competing with organizations that have significantly more resources than we do (Servprise was bootstrapped, and several of my competitors are public companies). One thing I noticed was that my competitors were largely offering stove-pipe solutions that were designed only for human operators to manually use. We decided from the get-go to make ease of integration a key selling point for the solution. This selling point has been the largest contributor to our growth.

There are two main ways our customers integrate our solution:

1) The functionality is made available in a centralized system for ease of access and control (e.g., a control panel at a hosting company).

2) Monitoring solutions, such as Nagios, that can tell you whether or not a server has crashed or a server is overheating, take automated recovery steps to either correct the problem or prevent escalation of the problem by sending commands to the WebReboot.

Our customers obtain financial benefits by making the relatively simple hardware functionality of our solution accessible in larger systems such as control panels and monitoring solutions. The problem for me was enabling this integration faster and cheaper than the competition.

Our competition, for the most part, chose the route of either developing and maintaining client libraries to manage communication to and from their embedded systems in Java, PHP, Python, etc., to ease integration for their customers, or left the customer to develop solutions on their own. I did not and still do not have the resources to develop and maintain client libraries for my customers. It’s code generation that has enabled me to compete on this level.

For my application, as well as numerous other embedded applications where the interface is tightly coupled to physical hardware, where the means of control on the physical world (e.g., physical relays and sensors, embedded systems) remain largely unchanged over time (it’s very cost-prohibitive to change), where scalability is not a major concern, and where capability and ease of integration are key selling points, WSDL and SOAP may be just what the doctor ordered, even if the only reason is code generation.

Clearly, those that put forth WS-* had much grander ambitions. I firmly believe REST is better suited for obtaining those ambitions. But I also believe this approach has a real future in making it easier and more cost effective for people to bridge the gap between software and the simple embedded hardware that enables control of the physical world.

Kick More Butt, Use Less Effort: Web Apps on Web Services (Part 1)

March 21st, 2008

Web services have changed my development world forever. Sounds kind of silly, I know… but I’m a true believer.

I still remember developing web applications the old-fashioned way… you know, some HTML forms posting some data to the server… doing some server-side magic, and then pushing out a hot ‘n fresh new page of HTML, ripe for the user’s viewing pleasure! Here’s a professional artist’s depiction of what I’m talking about:

Old Fashioned Way

The user enters their name on a form (”Bob”). Clicks submit. The data is POSTed to the server as a name=value pair (”Name=Bob”). Your choice from a number of server-side technologies then generates a new page with a response (”Hi Bob!”) and sends it to the browser.

It was easy to understand. Quick to implement. Enjoyable to tinker with. It worked, and I made money doing it. At the time, I didn’t have too much to complain about. You make a form. You post the data. You process the data. You return HTML. Rinse and repeat.

Then came the wave of AJAX. Ho-ly crap.

I remember the first time I saw an AJAX web app. It was Gmail. It was the freakin’ auto-complete fields in Gmail. I couldn’t believe it. How did it work that fast? It’s faster than… my desktop email client! Un-freakin-believable! What sorcery was at work?

Disgustingly simple, and disgustingly effective sorcery. Just a little snippet of client-side code, in the background and without interrupting the user, sent little tidbits of information to the server, and received and processed responses in return. If all you needed were these little tidbits, you didn’t even have to generate a new page on the server. Just let the client-side code manipulate the page as needed.

Ajax

Brilliant. This had HUGE implications for me. I come from an embedded hardware world. A world where you need to squeeze the maximum performance possible out of your hardware to deliver the greatest feature set possible to the customer, without spending too much money on super-fast processors. When it came to web-enabled embedded hardware, this opened up many, many new doors.

I now didn’t need the embedded processors in our remote reboot hardware to spend all of their time dynamically building HTML. Instead of dynamically building HTML files line-by-line in C, I could serve static HTML and JavaScript files. The static HTML would contain the forms for the user to interact with the system. The JavaScript would just submit the data from the forms, receive the response, and display the response for the user on the same page. I went from having to dynamically process tens of kilobytes to tens of bytes. That’s three orders of magnitude less data to process.

As a result, the system was much “snappier” (as end users often describe), and I had extra processing power to burn for new features. But it didn’t stop there. See, the operating system we eventually settled on for the WebReboot Enterprise product line was the Evolution OS from Lantronix. They actually had an AJAX interface right out-of-the-box. Worked just like described above. When you were using it, it didn’t even feel like there was an embedded processor on the serving end. Just what the doctor ordered. But there was more to come…

About the same time, a bit of buzz was being generated on web services, promising brand new worlds of interoperability, making it easier to get systems to talk to each other. If I can make it easier for people to write software against my remote reboot hardware, that would surely give me a leg up in the marketplace. It wasn’t the first technology on the block to make such promises, but hey, it’s worth a look, right?

There were two major camps of web service folks at the time. The REST folks, and the SOAP folks. Without getting too technical, the main differences, as I understood them, were that REST promised a very loose, informal way of exchanging data that was very quick and easy to get up and running, while SOAP promised more rigid, explicit, and formal data exchange that took a little more effort to get started with.

Sticking with my “What’s Your Name?” web application example, someone writing code against a REST web service might access a URI like the following:

http://whatisyourname.com/set-name/Bob

Pretty simple. You just invoked this REST web service, telling the service your name is Bob. In response, the web service could return:

Hi Bob!

That’s it… I mean it could be a fully generated HTML page as a response, it could include other stuff, but really, that shows you what I mean by quick and easy. You invoke the web service by including the parameters in the URI, and the response you read back from the request is the result. You can even manually invoke the web service in your web browser just by plopping that URI in the address bar and seeing the response displayed in the browser… in fact, I find that to be one of the major appeals of this approach.

The SOAP approach, on the other hand, requires a little more setting up… see, you’re actually going to send a request to the web service formatted in XML, and then get a response back in XML.

You might send the following to the web service at the URI http://whatisyourname.com/services/NameService:

<soapenv:Envelope xmlns:soapenv=“http://schemas.xmlsoap.org/soap/envelope/”>
<soapenv:Body><q0:SetName xmlns:q0=“http://whatisyourname.com/schemas/2008/02/04/NameService.xsd”>
<q0:Name>Bob</q0:Name>
</q0:SetName>
</soapenv:Body>
</soapenv:Envelope>

The response would be:

<soapenv:Envelope xmlns:soapenv=“http://schemas.xmlsoap.org/soap/envelope/”>
<soapenv:Body><q0:SetNameResponse xmlns:q0=“http://whatisyourname.com/schemas/2008/02/04/NameService.xsd”>
<q0:Response>Hi Bob!</q0:Response>
</q0:SetNameResponse>
</soapenv:Body>
</soapenv:Envelope>

Wow. What a difference, huh? Looks like REST is the clear winner. I mean, with REST, I just access the URI containing the parameter, and get the response back with the data. With SOAP, I have to build an XML document with VERY strict formatting requirements, submit it, get an XML document back as the response, parse it, and I have my data.

Well, not for me, and not for what I wanted to accomplish. You see, there’s a lot more that goes into SOAP web services than REST web services, but you get plenty back. I chose SOAP, and never looked back.

For my SOAP web service, I documented the format of the request and response messages using the Web Service Description Language (WSDL… just another XML document). Basically, I specified that the request XML should contain a string with the name (”Bob”), and the response XML should contain a string with, well, the response (”Hi Bob!”). The WSDL in effect documents the API. It tells you what commands and data you can invoke on the service, and what you’ll receive in response.

Now here’s the absolute coolest part (at least in my opinion). Armed with the WSDL document, you can use freely available, open source tools to automatically generate stub code to send requests to and receive responses from the SOAP web service in just about any modern programming language of your choosing.

For both the client and the server.

The stub code generates and parses all of the XML. As a developer working in the language of your choice, you are completely abstracted from the creation of the code for the sending and receiving of data on the wire (I had a gross omission here… thanks owed to Steve Vinoski for pointing it out).

If I were to write Java client code against this web service, I would use the open source WSDL2Java utility, and as output I would get super easy to use setName(String name) and getResponse() Java methods to interact with the service.

I could just as easily write code against the service in Python, .NET, PHP, Perl, C/C++, JavaScript, and many more. By spending the time to formally document the API in WSDL, you can now automatically generate stub code to communicate with the web service for your language of choice. Well worth the extra effort.

To build a web application on top of a XML SOAP web service, the JavaScript client creates the XML SOAP requests and parses the XML SOAP responses, just like the AJAX described above.

SOAP

For our remote reboot hardware, I decided to not use the built-in AJAX support of the Evolution OS (which relied on sending and receiving data by using POST and reading the response), and instead built an XML SOAP web service in C in the Evolution OS, and a JavaScript web service client for the web browser interface. It took a while to get up and running, but has paid dividends.

What are some examples of how we’ve benefited?

  • We have complete freedom to choose the right programming tool for each job, without having to worry about maintaining multiple API libraries. We don’t need to distribute JARs and worry about compiling in Java 1.4, 1.5, or 1.6. We don’t need to distribute PHP that was developed against version 4 but is buggy against 5. We don’t impose “DLL Hell” on users. We just maintain the web service as the API, put out a guide showing people how to generate the stub code in the language of their choice, and everything else just works.
  • The web browser interface, once fully loaded, is extremely responsive, considering it is an embedded processor based, SSL-enabled system.
  • All functionality of the WebReboot Enterprise is regression tested using unit tests written in Java using TestNG against the XML SOAP web service. That’s right… test-driven development of an embedded system using Java, even though the embedded system was written in C. Every change to the C code that’s committed to Subversion gets checked out by Bamboo, our continuous integration server, built and deployed on actual WebReboot Enterprise hardware, and ran against a suite of checkin, verification, and functional regression tests (depending on what changed). All automatic. Cool beans.
  • We are abstracted from dealing with the data on the wire. I don’t have to care about sending 0×0A hex for a reboot, and 0×0B hex for a power on… I just call my nice clean doReboot() and doPowerOn() methods in a Java client I’m integrating with the web service, and it just works.
  • After implementing the server in C and the client in JavaScript, I wanted to create a reference implementation for the server in a language I can develop much faster than C in. Accordingly, I took the WSDL file, ran WSDL2Java on it to generate the server stub code, and filled in the blanks for the various requests and responses in Java. The net result is I was able to create a fully functional “mock” WebReboot Enterprise in a Java servlet container. We now distribute this for folks to get started with development without needing a full, physical hardware WebReboot Enterprise in front of them, and we also use it for our live public demonstration, enabling each user that connects to get a unique “session,” such that any configuration changes they make are confined to their session and don’t affect other users accessing the demo. Super cool beans.
  • In the field, we’ve whipped together Python applications in a matter of minutes against the XML SOAP web service that automated several of the mundane processes for deploying large installations… such as setting Syslog servers, DNS servers, etc.
  • XML SOAP messages are versioned by namespace. Each request contains a namespace, and each response contains a namespace. The namespaces (if you designed your service well) correspond to which version of the WSDL document (in other words, the API) they were designed for. When our web service code released in December, 2007 receives a message from a client that was written against an older version of the web service, it automatically determines this, and automatically degrades functionality. Upgrades won’t break existing systems. The more I can prevent things from breaking, the happier I am (since I don’t have to hack fixes together to get stuff working in the field), and the happier my customers are (since they don’t need to rewrite their code when they upgrade firmware… if they want the latest features made available in the latest version of the web service they’ll need to make some code changes, but they won’t break anything just by upgrading).
  • In the factory, we’ve developed a test system written in Python against the XML SOAP web service to do a full hardware inspection to ensure each unit off the line meets quality standards. The test system is even integrated with the web service built-in to JIRA, our issue tracking system (those Atlassian guys are so smart), so quality issues are automatically tracked.
  • We’ve developed plugins in Python for Nagios that use the data available from the XML SOAP web service on the WebReboot Enterprise (like server temperature, server power status, and server connection status) to make decisions about corrective actions that can be taken to automate the datacenter (turn off overheating servers, turn on servers that should never be turned off, reboot crashed servers, etc.).
  • Our customers have integrated the WebReboot Enterprise functionality into their client control panels using everything under the sun… from ASP.NET to PHP to Java servlets. We’re talking huge datacenters filled with thousands and thousands of servers, and the folks using those servers now have nice little reboot, power control, power monitoring, and temperature monitoring displays in the control panels they were already using to monitor and control those servers. When customers are happy, those are the coolest beans of all.

For these reasons, and several more I plan on going into more depth on, web services have enabled me to kick more butt with less effort in the embedded hardware world. In Part 2, I’ll show you how I’ve used this approach to kick more butt with less effort for full-blown, production web applications, and Part 3 will discuss the huge benefits this approach has for test-driven development of web applications. I’ll probably wrap up with some lame “TOP 10 WAYS WEB SERVICES RULEZ!!1!!!”, since that’s the only way to get on Digg anymore.

Stay tuned!

Making Intangible Benefits… Tangible

January 27th, 2008

We’re in a fairly benefit-intangible business. Selling WebReboots is a lot like selling insurance; buying one won’t make you any prettier, nor will it spice up your love life… but when you have a server failure and you aren’t nearby to directly handle it, you’ll be darn glad you have one!

To combat this intrinsic intangibility, we have a two-fold strategy:

  1. Convey the benefits through story-telling.
  2. Make the features and functionality as tangible as possible.

We have a pretty interesting story to tell of how we came to be. We were a small hosting company trying to make it in a big industry. We had a need for a product that would make us more competitive that wasn’t available at the time, so we made something that met our needs. Turns out, other folks had the same need, and wanted to buy the product from us. Voila! WebReboot!

Building upon the core “how we got here” story when face-to-face with potential customers, I try to add one or two customer-relevant stories to convey the pain that I felt when I was in a role similar to the customer’s… a pain that the customer would likely share. Once on common “Man, I hate that!” ground, I tell a story of how I alleviated the specific pain, and what I can do to alleviate their pain. It certainly makes for more interesting and productive conversation, both for me and for the potential customer, than the naive product-centric “here’s what the widget does” sales technique I used when I first started. But telling a good story about alleviating pain is only part of the process.

We have very, very smart customers. They know snake-oil when they see it. Firing back “we have the solution to that” to each of their voiced problems, pains, and concerns will be heard by increasingly deaf ears. They don’t want to be told what a new widget does, or how it’s all-that-and-a-bag-of-potato-chips; they want to be shown. They want to see their pain alleviated before their very eyes.

Unfortunately, that makes it very, very difficult to sell something with an intangible “you’ll be glad you have it when you need it” benefit. While we could have a successful sales experience by hacking into our customers’ datacenter the night before a meeting, crashing their servers, and then showing up all bright and chirpy the next morning with a welcomed opening line of “Ever have a server crash in the middle of the night?” as they stare bleary-eyed up at us, still wearing their slippers and ducky-pajamas from the all-nighter they just pulled… I can think of one or two legal and technical reasons why that may not be feasible.

Instead, I bring a computer I can crash and reboot with us to each customer meeting. This computer also happens to be covered in well-lit viewing windows that shows how all of the WebReboot magic works.

Servprise Demo Computer

I’ll save the technical details behind this setup for another post, but for the those on the edges of their seats, it’s a custom-built small form factor PC running Ubuntu with a WebReboot Advanced Server Card installed inside, as well as an LCD control panel on the front showing the hostname, IP address, and uptime of the PC, and I can “crash” the PC at-will from a menu option on the LCD control panel (really, it runs a script that sends Linux into single-user mode, but it LOOKS and BEHAVES like a crash from the perspective of a remote user).

You may now return from the edges of your seats. Read the Servprise Demo Computer Overview if you’re truly captivated.

Servprise Demo Computer LCD

WebReboot Advanced Server Card

When we visit with customers, we always bring our demonstration gear with us. Always, always, always! With it, I can demystify everything about the WebReboot. I can show crashes and recoveries. I can show overheating alerts, temperature monitoring, and automatic reboots with Nagios. I can show how customers won’t accidentally reboot the wrong server when cables get crossed. I even turn control over to customers and let them play with it.

Once it became tangible, it not only became easy, but it also became a lot of fun. One customer once connected a monitor and keyboard to the demo computer, and through some holy knowledge of Linux that far exceeds my own, actually crashed the computer much more severely than my meager “go into single user mode” script… and then became truly impressed when the reboot was successful.

We’ve had requests for quite a while to roll all of this into a video, and that’s what some of my recent efforts have been focused on. I’ll follow-up in a few days for some more details on my first experiences creating a video like this, but without further ado, I present… the Servprise WebReboot Enterprise Demo Video:

Row, Row, Row Your Datacenter

January 8th, 2008

Ken Jamaca and Derek Baker from Silverback Migration Solutions pointed me to a new datacenter company (PDF datasheet showcasing their first ship) that is converting refurbished cargo container ships into full-fledged datacenters. The company, IDS, is currently in stealth mode.

Major benefits include:

  1. Rapidly deployable datacenter real-estate in the most physically and electrically congested areas of the port-accessible nation.
  2. Readily available and cheap chill water (mind the salt!).
  3. It’s real… first ship coming online this April.

For more details, see Ken’s blog, where he points out some contact information for the folks. I’m willing to sign up for a tour come this April!

Ever Use Example Code From a Book That Did Not Work?

January 8th, 2008

The challenge of keeping example code updated and working when it is destined for printed form was addressed during my update over the weekend with a quick glimpse into how we’ve overcome the problem. In a nutshell, we keep our example code in source control, and use brief tags in Microsoft Word along with a Word macro that pulls the code directly out of source control and places it inline with our descriptive content. See my previous article for the technical nitty-gritty.

Now I want to highlight a couple areas where I think this approach will be applicable, as well as a demonstration movie showing how we use it.

  1. Technical Books. Big, big problem. I can’t remember the last time I picked up a programming book and didn’t find a flaw somewhere in the example code. Just like we have a spell-checker to catch a good portion of typos, we can now have a compilability-and-testability-checker, since our code would be coming straight from source control, where the automated build system and suite of unit tests made certain everything was working.

    And really, knowing that the printed code works is only a small benefit of this approach. I see a much bigger benefit being when edition 2 of a book comes out, and now making all of the code in the book up to date is as simple as… making the actual code in source control up to date. Then just run the macro to pull the new code into the book. Much, much simpler than having to go through the book and manually update all of the code examples.
  2. Web Sites. This might be even more of a problem than with the technical books, since there often isn’t quite the same review process for web sites as there is for mass-produced technical books.

    Probably about 90% of the work I had to do to make this happen in Word was spent overcoming my own lack of knowledge in macro development and VB/VBA. This would have been significantly easier for me to have implemented with server-side software that I am more accustomed to; but alas, that was not where I needed it the most. I needed to address our printed documentation. I leave it to someone else to benefit in the online world of code examples.
  3. User Guides and Developer Guides. I think I beat this one to death already. Let’s move on to the video.

Here I walk through the API manual for our WebReboot Enterprise, and show how a technical writer would add code examples into their documentation, and have those code examples come right out of source control.

You Are Frustrating Your API Users

January 6th, 2008

It’s time for a show of hands. How many folks have ever had the following happen to them:

  1. Get excited over a shiny new component, object, or service to integrate with your project.
  2. Download the SDK and API docs.
  3. Copy and paste some trivial examples into your project to learn about the new API.
  4. Son of a …. The examples don’t work!

If you’re fortunate, it’s just a missing semi-colon or other typo that’s easy to find, easy to fix. If fortune is not on your side, it will be the following scenario that happened to me roughly this time last year.

The examples I was using were from the most recent documentation available from the culprit’s web site. The Java code just would not compile; the methods being used just did not exist in the classes that the examples were using. Sure, those classes existed. And sure, those methods existed in other classes. Many other classes. But not as described in the example.

I thought for certain the problem was by my own doing. Was there something else on the classpath screwing me up? Did I copy and paste the wrong example? Was I using an incompatible JDK? The longer I spent on it, the more and more outrageous the possible causes I came up with were.

I did not assume the example was flawed. Why? Call it over-confidence in the documentation, call it a personal trait to first assume that I am the cause of the problem, or perhaps call it a combination of the two.

At this point, having no example, and just figuring it all out myself from scratch would have been far more efficient. It’s sad, but it’s true. It would have been better for me to have no documentation at all than to have started from flawed documentation. The ultimate problem was trivial to say the least… the example code was using the wrong class. Digging through the javadocs, and about an hour or two of trial and error to find the right class to use, finally saved the day.

Not fun.

The most likely cause of the whole problem? I see a common scenario going something like this:

  1. Documentation folks say “Gee whiz! It would be great to show our users some examples of this new whiz bang widget!”. They ask the programmers to whip something together.
  2. Programming folks create a set of examples. They’re designed, documented, compiled, tested, code reviewed, and shipped off to the documentation folks.
  3. Documentation folks insert the code snippets into Word, InDesign, Quark, or whatever publishing platform is used, never to see a compiler again… until the end-user gets their hands on it and puts it through theirs.

Now what can happen between the last time the programmer runs the code, and the first time the end-user does? Well, a lot, quite frankly. Everything from an incomplete insertion of the code into the documentation by the technical writers, to a seemingly benign last-minute global Find and Replace All by the Chief Editor that trips up on a portion of the code, to some creative modifications by the technical writing staff (”Bob in software couldn’t have meant ++i, surely he meant i++! I’ll fix it for him.”).

Even if the first version of the code survives the trip from the programmers to the technical writers and then to the end-user, what happens when the new version of the API is released? Can someone guarantee that they will go back and update the examples in the document? Who’s going to do it and make certain it’s done correctly? The valuable programmers? None-sense! They’re working on new whiz bang widgets! Chances are, this is where most of the problems come from… the actual code behind the API and the API itself are updated, but the examples that use the API are forever trapped in Word, and do not get updated. It’s a classic case of information living in separate places, and thus maintained separately; they are practically guaranteed to grow out of sync over time.

No matter what the cause, however, the result is the same: Frustration for the end user trying to work with the examples. And frustration is bad.

After all, the end-user was gracious enough to try out your software. Heck, they might even want to open up their wallets, take money out, and put that money into your wallet! And you’re going to ask them to debug your code?!? Try not to go out of business too quickly.

A Proposed Solution

Now, here’s where we go from ranting and raving to something a little more productive. I decided that I would do everything I reasonably could to ensure no user of my code suffers the same frustration.

The solution I came up with is two-fold:

  1. Keep the example code where code should live: in source control, where it’s frequently checked out, built by the build server, and regularly run through a gamut of unit tests.
  2. Automate the process of putting code in documentation. No emailing of code, no copying and pasting.

From an implementation standpoint, it’s disgustingly simple. My solution consists of a Word macro that automatically pulls example code out of source control, places it in the appropriately tagged locations in the documentation, and formats it as “code” (e.g., Courier New, special indenting, etc.).

Benefits of This Approach

Before I dive into implementation details, it’s helpful to describe several benefits of this approach to putting example code in documentation.

Easy to Maintain

I am willing to bet that for most organizations, example code is throwaway code. That means that after it’s copied-and-pasted, it’s never edited again… perhaps the original source files aren’t even saved. Why? Because if the cost of properly maintaining the code exceeds a certain threshold such that it impacts other revenue-generating development, it won’t be maintained. No one will maintain example code that they have to copy and paste out of Word into a new project, update and debug, and then once again copy and paste the example code back into Word, right before the new release of the software ships. Won’t happen.

The trick is to make it easy to maintain; to treat the example code as a real project, so there is real motivation to maintain it yet it doesn’t significantly impact other development. Don’t start off on the wrong foot by assuming the code is throwaway. You will create more headaches for yourself in the end.

What do I mean by making the example code easy to maintain? Easy to maintain means source control. Easy to maintain means test-driven development that guarantees unit tests exist for all of your example code. Easy to maintain means that a build server checks out, compiles, and tests the code every time a change is made. Easy to maintain means that when your programmers inevitably make a change to the API or to the underpinnings of the API that breaks the example code, they find out that the example code stopped working as soon as possible, so they can make the corresponding updates to the example code, too.

Support Change

Refactoring is good. It keeps your code lean, mean, and no more complicated than it needs to be. Refactoring should save you work in the long run, not add more work.

Typically, a user’s guide for an API will have the names of classes and methods sprinkled throughout. Not just in code snippets, but inline in paragraphs as actual subjects of sentences (e.g., “The WidgetFactory class is rather nifty because it does A, B, and C.”). If there’s a need to rename this class, say from the WidgetFactory to the GadgetFactory, have fun keeping your documentation in sync with the change. Even a global “Find and Replace All” operation for changing “WidgetFactory” to “GadgetFactory” in the documentation can introduce more problems than it solves (e.g., it will trip up on “Note: The WidgetFactory class was renamed to the GadgetFactory class as of release 4.”).

With the approach shown here, the code itself will still live in source control. The documentation pulls the example code from source control. Refactoring and committing a change in source control would coincidentally update the documentation when it is next accessed.

Consistent Formatting

The formatting of code is different from the formatting of your descriptive content. Programmers should have their way for code, technical writers and graphic design folks should have their way for the rest. Crossing the two will only lead to problems.

To combat this, all the programmers need to do is adhere to predefined line-width restrictions provided by the design folks, and display consistency in their usage of tabs versus spaces, and life will be good. For me, this was as simple as adding a code formatter in Eclipse that capped the line-width for my example code project, and imposing a rule for converting all tabs to spaces.

With this approach, the code will appear in the documentation exactly as the programmer intended. No word wrapping or inconsistent indentations across lines that may break the code (think Python), no mismatched curly braces, no none-sense.

Implementation

As earlier described, the bulk of the solution lives in the form of a Word macro. As a programmer, all I’ve done is created and tested my example code, formatted it within the parameters defined by the design staff, tagged it appropriately to indicate what portions of code should be pulled out into the documentation (more on this in a moment), and checked it into source control.

As a technical writer preparing the documentation, I’ve undoubtedly asked the programmer for example code showing a couple of common workflows, to which I receive in response a list of tag names that correspond to the examples. All I have to do is put those tags into my documentation, and run the macro.

Let’s walk through an example. We’ll have Calvin, a technical writer, and Bob, our programmer, working together to make a Developer’s Guide for the WebReboot Enterprise API. The most common reason to use this API is to programmatically reboot servers attached to a WebReboot Enterprise. You may reboot them because your monitoring software told you they crashed, or you received a command from your support desk software that indicated a user has filed a trouble ticket for a reboot. Details aside, as the technical writer, Calvin will want to show our users how easy it is to do this very common task with the API. For completeness, we’ll show how to get status information for a server, and power it off and on as well.

Calvin will fire off an email to the programmer to get the ball rolling:

Howdy Bob,

I would like to get some code examples to insert into our developer documentation how easy it is to use our API. Can I get a couple of examples showing:

  • How to reboot a server
  • How to get status information on the server, such as whether the server is on or off
  • How to power off and on a server

That should do it. Thanks!

Calvin the Technical Writer

A couple dozen cups of coffee later, Bob will shoot back to Calvin:

Hola Calvin,

All set! Here are the tags that correspond to the code examples you asked for:

  • How to reboot a server
    • Usage example tag:
      { PRIVATE
      code_java/src/com/Servprise/documentcode/wreapimanual/Examples.java:refJCodeShowPowerSwitchReboot
      }
  • How to get status information on the server, such as whether the server is on or off
    • Method name tag:
      { PRIVATE
      code_java/src/com/Servprise/documentcode/wreapimanual/Examples.java:refJMethodGetPortConfiguration }
    • Usage example tag:
      { PRIVATE
      code_java/src/com/Servprise/documentcode/wreapimanual/Examples.java:refJCodeShowServerStatus
      }
  • How to power off a server
    • Method name tag:
      { PRIVATE
      code_java/src/com/Servprise/documentcode/wreapimanual/Examples.java:refJMethodDoPowerOff
      }
    • Usage example tag:
      { PRIVATE
      code_java/src/com/Servprise/documentcode/wreapimanual/Examples.java:refJCodeShowPowerOff
      }
  • How to power on a server
    • Method name tag:
      { PRIVATE
      code_java/src/com/Servprise/documentcode/wreapimanual/Examples.java:refJMethodDoPowerOn
      }
    • Usage example tag:
      { PRIVATE
      code_java/src/com/Servprise/documentcode/wreapimanual/Examples.java:refJCodeShowPowerOn
      }

All you have to do is use our standard Servprise documentation Word template, and insert the above tags where you would like the code to appear, and then run the GetCodeExcerpts macro whenever you want to update the code in the document. That’s it!

Enjoy!

Bob the Programmer

Now, for this to all come together, here’s what Bob did prior to sending that email back to the technical writer. He wrote the Java example code for each requested operation, and put all of the code in a class called Examples. The tag names he emailed back (shown above) specify two pieces of information:

  1. What file the code is in. In this case, each tag specifies a common file: SOURCE_CONTROL_PATH/java/src/com/servprise/documentcode/wreapimanual/Examples.java
  2. What portion of code in that file to pull into Word. In the case of powering off a server:
    1. The method name to use is tagged:
      refJMethodDoPowerOff
    2. The example snippet is marked with the tag:
      refJCodeShowPowerOff

What does the actual code in Examples.java look like? Here’s the relevant portion for powering off a server:

  /**
   * Example showing how to perform a power off
   *
   * @throws RemoteException
   */
public void showPowerOff() throws RemoteException
{
    // refJCodeShowPowerOff .

  // Prompt user for information.
int portNumber = getPortNumber();
GetWebRebootPortRequest infoRequest = new GetWebRebootPortRequest();
infoRequest.setPortNumber(portNumber);
GetWebRebootPortResponse infoResponse = wr.getWebRebootPort(infoRequest);
WebRebootPort port = infoResponse.getWebRebootPort();

// Check that the server supports power operations.
if (true == port.getServerPowerSwitchSupported())
{
  System.out.println(”You are about to power-off: ” + port.getName());
  // Prompt user for confirmation of power toggle action.
  boolean confirm = confirmAction();

  if (true == confirm)
  {
    PowerOffRequest powerOffRequest = new PowerOffRequest();
    powerOffRequest.setPortNumber(port.getPortNumber());
    PowerOffResponse powerOffResponse = wr.doPowerOff(powerOffRequest);

    // Check if there was an error executing the command.
    CommandResult result = powerOffResponse.getCommandResult();
    if (result.isError())
    {
      System.err.println(result.getResultMessage());
    }
  }
}

    // refJCodeShowPowerOff .

PowerOffRequest powerOffRequest = new PowerOffRequest();
PowerOffResponse powerOffResponse = wr.
doPowerOff // refJMethodDoPowerOff .
(powerOffRequest);
Assert.assertNotNull(powerOffResponse);

}

You can see the tag refJCodeShowPowerOff exists twice… surrounding a code segment that shows how to do a power off operation.

You can also see the tag refJMethodDoPowerOff exists only once in this code… next to the relevant method name.

That is really all that’s required from a code standpoint to pull the code out into our documentation. The macro in our Word document will parse this file looking for the relevant tag names. When two tags are found, the macro pulls out the lines in between the tags. That’s how we show a code snippet. When a single tag is found, it pulls out the content preceding the tag on the same line. That’s how we show just a method name (or a variable, or a single line of code in general).

From the technical writer’s perspective, all he has to do is enter in the tag names that Bob sent over into Word as field codes. The field codes are a “hidden” portion of the document that is viewable by selecting Show Symbols in Word.

Here’s an example with Show Symbols turned on, and the field code to insert the method name for powering off a server is selected:

Field Code in Word

And now, with Show Symbols turned off, we just see the example code, exactly as our programmer intended us to:

Text in Word

Implement for Yourself

Since most of the magic exists in the Word document, that’s really all you’ll need. Use our API manual document as a starting point:

WebReboot Enterprise API Manual

I’ve released the macro under the Apache License, Version 2.0.

For your own needs, do the following:

  1. Create your example source code, and insert the tags as appropriate to indicate which portions should get pulled into Word. You can see the Examples.java file I’ve used for our own examples, but this won’t be directly applicable to your project.
  2. Copy the GetCodeExcerpts Macro into your own document. Make sure macros are enabled in Word.
  3. Insert the tags as described above into your Word document as field codes where you want the example code to be placed. It may be easiest to enable Show Symbols in your Word document, and then copy and paste the tags from the API Manual document, and then modify the tag as necessary.
  4. Add a Custom Property to the Document Properties in Word to indicate where your source code is stored. The macro needs this to locate your source code. The property name is “codeBaseUrl”, and the value should be to your source path.
    1. For Word 2003 and older, go to File -> Properties. Select the Custom tab.
    1. For Word 2007, this can be found by clicking the Office Pearl button in the top-left corner, then going to Prepare -> Properties. On the tab that appears at top, click Document Properties, then go to Advanced Properties. Finally, select the Custom tab. Some upgrade, huh?

For my needs, the value for the property “codeBaseUrl” is “D:\servprise\wre\documentCode\wreApiManual\”. The paths specified in the tags above are relative to the path specified in this property. Accordingly, the absolute path to the Examples.java file reference above would be:
“D:\servprise\wre\documentCode\wreApiManual\java/src/com/Servprise/documentcode/wreapimanual/Examples.java”.

The Document Properties

Results

We’ve been using this approach since the WebReboot Enterprise API was first released in November, 2006. We’ve had a number of updates and enhancements to the API since then, as well as a number of improvements to our code examples we provide to customers.

To say the least, knowing that after the code changes are complete, having updated documentation to distribute just a click away (rather than a multi-hour editing session away) has made the whole process a lot more pleasant. We know we’re giving examples to our customers that work, so they can get started on their own development without delay, and without unnecessary frustration.

Why So Productive in Massachusetts?

December 18th, 2007

I think I get more accomplished in the winter months in New England than throughout the rest of the year. And it’s no wonder…

We managed to time our return from Washington DC right between two monster storms. When it came time to decide what the weekend’s activities should be, a simple look outside on our porch pretty much summed it up:

Hunker Down

Internet? Check.

Power? Check.

Caffeine? Check. Check. Check.

Let’s get to work.


Home - Remote Reboot - Contact - Copyright 2007 Cory von Wallenstein. All rights reserved.