Standing on the Brink

Kick More Butt, Use Less Effort: Web Apps on Web Services (Part 1) 

Web services have changed my development world forever. Sounds kind of silly, I know… but I’m a true believer.

I still remember developing web applications the old-fashioned way… you know, some HTML forms posting some data to the server… doing some server-side magic, and then pushing out a hot ‘n fresh new page of HTML, ripe for the user’s viewing pleasure! Here’s a professional artist’s depiction of what I’m talking about:

The user enters their name on a form (“Bob”). Clicks submit. The data is POSTed to the server as a name=value pair (“Name=Bob”). Your choice from a number of server-side technologies then generates a new page with a response (“Hi Bob!”) and sends it to the browser.

It was easy to understand. Quick to implement. Enjoyable to tinker with. It worked, and I made money doing it. At the time, I didn’t have too much to complain about. You make a form. You post the data. You process the data. You return HTML. Rinse and repeat.

Then came the wave of AJAX. Ho-ly crap.

I remember the first time I saw an AJAX web app. It was Gmail. It was the freakin’ auto-complete fields in Gmail. I couldn’t believe it. How did it work that fast? It’s faster than… my desktop email client! Un-freakin-believable! What sorcery was at work?

Disgustingly simple, and disgustingly effective sorcery. Just a little snippet of client-side code, in the background and without interrupting the user, sent little tidbits of information to the server, and received and processed responses in return. If all you needed were these little tidbits, you didn’t even have to generate a new page on the server. Just let the client-side code manipulate the page as needed.

Brilliant. This had HUGE implications for me. I come from an embedded hardware world. A world where you need to squeeze the maximum performance possible out of your hardware to deliver the greatest feature set possible to the customer, without spending too much money on super-fast processors. When it came to web-enabled embedded hardware, this opened up many, many new doors.

I now didn’t need the embedded processors in our remote reboot hardware to spend all of their time dynamically building HTML. Instead of dynamically building HTML files line-by-line in C, I could serve static HTML and JavaScript files. The static HTML would contain the forms for the user to interact with the system. The JavaScript would just submit the data from the forms, receive the response, and display the response for the user on the same page. I went from having to dynamically process tens of kilobytes to tens of bytes. That’s three orders of magnitude less data to process.

As a result, the system was much “snappier” (as end users often describe), and I had extra processing power to burn for new features. But it didn’t stop there. See, the operating system we eventually settled on for the WebReboot Enterprise product line was the Evolution OS from Lantronix. They actually had an AJAX interface right out-of-the-box. Worked just like described above. When you were using it, it didn’t even feel like there was an embedded processor on the serving end. Just what the doctor ordered. But there was more to come…

About the same time, a bit of buzz was being generated on web services, promising brand new worlds of interoperability, making it easier to get systems to talk to each other. If I can make it easier for people to write software against my remote reboot hardware, that would surely give me a leg up in the marketplace. It wasn’t the first technology on the block to make such promises, but hey, it’s worth a look, right?

There were two major camps of web service folks at the time. The REST folks, and the SOAP folks. Without getting too technical, the main differences, as I understood them, were that REST promised a very loose, informal way of exchanging data that was very quick and easy to get up and running, while SOAP promised more rigid, explicit, and formal data exchange that took a little more effort to get started with.

Sticking with my “What’s Your Name?” web application example, someone writing code against a REST web service might access a URI like the following:

http://whatisyourname.com/set-name/Bob

Pretty simple. You just invoked this REST web service, telling the service your name is Bob. In response, the web service could return:

Hi Bob!

That’s it… I mean it could be a fully generated HTML page as a response, it could include other stuff, but really, that shows you what I mean by quick and easy. You invoke the web service by including the parameters in the URI, and the response you read back from the request is the result. You can even manually invoke the web service in your web browser just by plopping that URI in the address bar and seeing the response displayed in the browser… in fact, I find that to be one of the major appeals of this approach.

The SOAP approach, on the other hand, requires a little more setting up… see, you’re actually going to send a request to the web service formatted in XML, and then get a response back in XML.

You might send the following to the web service at the URI http://whatisyourname.com/services/NameService:

<soapenv:Envelope xmlns:soapenv=“http://schemas.xmlsoap.org/soap/envelope/”>
<soapenv:Body><q0:SetName xmlns:q0=“http://whatisyourname.com/schemas/2008/02/04/NameService.xsd”>
<q0:Name>Bob</q0:Name>
</q0:SetName>
</soapenv:Body>
</soapenv:Envelope>

The response would be:

<soapenv:Envelope xmlns:soapenv=“http://schemas.xmlsoap.org/soap/envelope/”>
<soapenv:Body><q0:SetNameResponse xmlns:q0=“http://whatisyourname.com/schemas/2008/02/04/NameService.xsd”>
<q0:Response>Hi Bob!</q0:Response>
</q0:SetNameResponse>
</soapenv:Body>
</soapenv:Envelope>

Wow. What a difference, huh? Looks like REST is the clear winner. I mean, with REST, I just access the URI containing the parameter, and get the response back with the data. With SOAP, I have to build an XML document with VERY strict formatting requirements, submit it, get an XML document back as the response, parse it, and I have my data.

Well, not for me, and not for what I wanted to accomplish. You see, there’s a lot more that goes into SOAP web services than REST web services, but you get plenty back. I chose SOAP, and never looked back.

For my SOAP web service, I documented the format of the request and response messages using the Web Service Description Language (WSDL… just another XML document). Basically, I specified that the request XML should contain a string with the name (“Bob”), and the response XML should contain a string with, well, the response (“Hi Bob!”). The WSDL in effect documents the API. It tells you what commands and data you can invoke on the service, and what you’ll receive in response.

Now here’s the absolute coolest part (at least in my opinion). Armed with the WSDL document, you can use freely available, open source tools to automatically generate stub code to send requests to and receive responses from the SOAP web service in just about any modern programming language of your choosing.

For both the client and the server.

The stub code generates and parses all of the XML. As a developer working in the language of your choice, you are completely abstracted from the creation of the code for the sending and receiving of data on the wire (I had a gross omission here… thanks owed to Steve Vinoski for pointing it out).

If I were to write Java client code against this web service, I would use the open source WSDL2Java utility, and as output I would get super easy to use setName(String name) and getResponse() Java methods to interact with the service.

I could just as easily write code against the service in Python, .NET, PHP, Perl, C/C++, JavaScript, and many more. By spending the time to formally document the API in WSDL, you can now automatically generate stub code to communicate with the web service for your language of choice. Well worth the extra effort.

To build a web application on top of a XML SOAP web service, the JavaScript client creates the XML SOAP requests and parses the XML SOAP responses, just like the AJAX described above.

For our remote reboot hardware, I decided to not use the built-in AJAX support of the Evolution OS (which relied on sending and receiving data by using POST and reading the response), and instead built an XML SOAP web service in C in the Evolution OS, and a JavaScript web service client for the web browser interface. It took a while to get up and running, but has paid dividends.

What are some examples of how we’ve benefited?

• We have complete freedom to choose the right programming tool for each job, without having to worry about maintaining multiple API libraries. We don’t need to distribute JARs and worry about compiling in Java 1.4, 1.5, or 1.6. We don’t need to distribute PHP that was developed against version 4 but is buggy against 5. We don’t impose “DLL Hell” on users. We just maintain the web service as the API, put out a guide showing people how to generate the stub code in the language of their choice, and everything else just works.
• The web browser interface, once fully loaded, is extremely responsive, considering it is an embedded processor based, SSL-enabled system.
• All functionality of the WebReboot Enterprise is regression tested using unit tests written in Java using TestNG against the XML SOAP web service. That’s right… test-driven development of an embedded system using Java, even though the embedded system was written in C. Every change to the C code that’s committed to Subversion gets checked out by Bamboo, our continuous integration server, built and deployed on actual WebReboot Enterprise hardware, and ran against a suite of checkin, verification, and functional regression tests (depending on what changed). All automatic. Cool beans.
• We are abstracted from dealing with the data on the wire. I don’t have to care about sending 0x0A hex for a reboot, and 0x0B hex for a power on… I just call my nice clean doReboot() and doPowerOn() methods in a Java client I’m integrating with the web service, and it just works.
• After implementing the server in C and the client in JavaScript, I wanted to create a reference implementation for the server in a language I can develop much faster than C in. Accordingly, I took the WSDL file, ran WSDL2Java on it to generate the server stub code, and filled in the blanks for the various requests and responses in Java. The net result is I was able to create a fully functional “mock” WebReboot Enterprise in a Java servlet container. We now distribute this for folks to get started with development without needing a full, physical hardware WebReboot Enterprise in front of them, and we also use it for our live public demonstration, enabling each user that connects to get a unique “session,” such that any configuration changes they make are confined to their session and don’t affect other users accessing the demo. Super cool beans.
• In the field, we’ve whipped together Python applications in a matter of minutes against the XML SOAP web service that automated several of the mundane processes for deploying large installations… such as setting Syslog servers, DNS servers, etc.
• XML SOAP messages are versioned by namespace. Each request contains a namespace, and each response contains a namespace. The namespaces (if you designed your service well) correspond to which version of the WSDL document (in other words, the API) they were designed for. When our web service code released in December, 2007 receives a message from a client that was written against an older version of the web service, it automatically determines this, and automatically degrades functionality. Upgrades won’t break existing systems. The more I can prevent things from breaking, the happier I am (since I don’t have to hack fixes together to get stuff working in the field), and the happier my customers are (since they don’t need to rewrite their code when they upgrade firmware… if they want the latest features made available in the latest version of the web service they’ll need to make some code changes, but they won’t break anything just by upgrading).
• In the factory, we’ve developed a test system written in Python against the XML SOAP web service to do a full hardware inspection to ensure each unit off the line meets quality standards. The test system is even integrated with the web service built-in to JIRA, our issue tracking system (those Atlassian guys are so smart), so quality issues are automatically tracked.
• We’ve developed plugins in Python for Nagios that use the data available from the XML SOAP web service on the WebReboot Enterprise (like server temperature, server power status, and server connection status) to make decisions about corrective actions that can be taken to automate the datacenter (turn off overheating servers, turn on servers that should never be turned off, reboot crashed servers, etc.).
• Our customers have integrated the WebReboot Enterprise functionality into their client control panels using everything under the sun… from ASP.NET to PHP to Java servlets. We’re talking huge datacenters filled with thousands and thousands of servers, and the folks using those servers now have nice little reboot, power control, power monitoring, and temperature monitoring displays in the control panels they were already using to monitor and control those servers. When customers are happy, those are the coolest beans of all.
  

For these reasons, and several more I plan on going into more depth on, web services have enabled me to kick more butt with less effort in the embedded hardware world. In Part 2, I’ll show you how I’ve used this approach to kick more butt with less effort for full-blown, production web applications, and Part 3 will discuss the huge benefits this approach has for test-driven development of web applications. I’ll probably wrap up with some lame “TOP 10 WAYS WEB SERVICES RULEZ!!1!!!”, since that’s the only way to get on Digg anymore.

Stay tuned!