We’ve been having quite a bit of fun recently putting together videos at the office. We’ve got some decent equipment, but one thing has been bugging me for quite some time… the quality of our audio.

Here’s an example shot with a Logitech QuickCam Pro 9000 webcam using the built-in microphone, standing roughly six feet away from the camera:

It’s OK, but not the greatest. Here’s the same Logitech webcam, but this time sitting in front of the camera, roughly two feet away, with the direction of speaking always toward the camera (and thus, the built-in microphone):

MUCH better! But we still want to be able to do recordings at the whiteboard, walking around, etc.

Here’s an example using the mic built-in with one of our camcorders (an Insignia HD Flash camcorder, which provides amazing video quality for the money):

What’s the ultimate solution? Well, a high-quality lavalier microphone, of course! And we have one… it’s the Audio-Technica ATR-35S.

So, why then, on our latest (and also most popular) videos, do we still have not the greatest sound in the world?

Insignia camcorder with built-in mic, but noisy environment in the middle of office, with lots of moving around by the speaker (relative to the mic):

Same Insignia camcorder and built-in mic, but quieter area with more control of what’s going on:

Same Insignia camcorder and built-in mic, but a better job of noise reduction in post-production:

So, I ask again, if we have the lavalier microphone, why haven’t we been using it? The answer is … we don’t have any place convenient to plug in the lavalier microphone when using our camcorders.

The camcorders themselves unfortunately do not have external microphone inputs.

We’ve tried using the microphone inputs for some laptops and desktops we had handy (and synching the audio and video in post-production), but the poor quality sound cards built-in clearly had some grounding issues, because a hefty hum could be heard (and often overheard) in the background. Plus, this often meant a long cable from the lavalier microphone to the computer, which was likely to be tripped over. No good.

But as of today, that changes. I just picked up a Sony ICD-UX70 digital voice recorder for the video production team, with built-in stereo microphones, an external microphone jack, and MP3 encoding of audio files. Now, we can throw this in the shirt pocket for whoever is speaking on video (when we’re in a rush), or for the best quality, just have the actress or actor hold it/put it in a pocket and wire up the lavalier microphone to walk around freely.

We’re shooting some video this week… I’ll let you know how it works out (hopefully you’ll be able to hear the difference for yourself!).

Here’s to good quality audio.

A little background is in order. Recently, my team released the DynDNS Spring Server^SM VPS platform based on Xen, with one of the main features being dual-stack IPv4 and IPv6 connectivity out-of-the-box. For networking the virtual servers, we’ve chosen a bridging approach, which makes it pretty easy to get IPv4 and IPv6 working; as long as all OSes involved support IPv6, it kind of “just works.”

What’s not so easy is trying to secure and lock down the individual virtual servers so they cannot interfere with one another’s network connection. The efforts spent on IPv6 also exposed some issues for IPv4 and ARP that could arise when used in a VPS environment.

Note: We’re going to get down and dirty into the technical nitty gritty, and it’s beyond the scope of this post to bring all of my readers up to speed. The faint of heart may wish to turn back now.

Problem Cases for IPv6

Two main network spoofing cases come to mind that need to be protected against for running IPv6 on Xen in a virtual private server environment:

1) A user changing the IPv6 address of their network interface to an IPv6 address we haven’t authorized them to use, or adding/aliasing additional IPv6 addresses to their network interface that they did not pay for.

2) A user changing/adding/aliasing an IPv6 address that’s in use by another customer, which can wreak all sorts of routing havoc, interfering with the ability for our customers to reliably communicate with others on the Internet.

Perhaps Luke Crawford put it best with his response to my question, “Have people already solved and dealt with IPv6 in Xen successfully (i.e., is it a non-issue at this point)? If not, I’d be happy to submit the changes and a guide to making it work and work well.” on the Xen developers email list:

Guides would be awesome. Also, if you made anti-spoof work for IPv6, that would also be awesome. I’ve had IPv6 going for a while with no problems (though the xen antispoof rules only work with IPv4), but I didn’t even need to enable IPv6 in the Dom0; it’s a layer 2 bridge, so it just works. In fact, stateless autoconfiguration even worked. I asked for an IPv6 allocation from my provider and my customers noticed it was working before I got around to doing any setup at all.

Problem Cases for IPv4 and ARP

When we initially went down the IPv6 road, we started paying closer attention to what was already implemented for IPv4 protection to get an idea for where to begin. In doing so, we found a couple of scenarios where the existing anti-spoof protection leaves Xen a bit too exposed for using as a VPS platform:

1) If a user aliases an IPv4 gateway IP address for the same subnet that user is on, all network connectivity can go down for that subnet. As an example, if your IP address is 1.2.3.5/24 and your gateway is at 1.2.3.1, and you run “ifconfig 1.2.3.1″ to alias the gateway IP address on your default network interface, you will now respond to ARP requests trying to find the gateway with your MAC address… meaning no other users on that subnet can reliably talk to the gateway… meaning their network connectivity is pretty much useless.

2) If a user aliases an IPv4 address that is already in use by another user, all network connectivity can go down for that other user, for the same reasons it goes down when aliasing the gateway IP address (i.e., you’ll be responding to ARP requests for their IP address).

3) Users can deliberately perform ARP poisoning attacks, by sending arbitrary ARP replies on the subnet, which can interfere with authorized communication for the same reasons as 1 and 2 above.

4) Users can see the network traffic for other domUs on the same dom0 by putting their interfaces into promiscuous mode (i.e., packet sniffing). 

My intent here is not to knock the Xen development community; they’ve put together a fantastic platform, and no one expects our business requirements for using Xen to host virtual private servers to be inline with their development requirements. I’m not even certain that we’ve squashed the issues entirely, but we have at least squashed the pressing issues for IPv6, IPv4, and ARP outlined above.

Now that we have it working in the field, I’d like to share the handful of changes with the community so they can benefit, and perhaps suggest better/more complete/more succinct ways of doing it. The solution we implemented was modeled after the current IPv4 anti-spoof protection.

I will start off with an overview of the current IPv4 anti-spoofing approach used by Xen, followed by the specific commands used to alter the behavior for what we needed for reliable and secure communication, concluded by the actual changes we made to the Xen networking scripts to enable this behavior modification.

Current IPv4 Anti-spoofing Protection for Xen

The current IPv4 anti-spoof protection scheme relies on iptables, in particular the FORWARD chain that each packet destined for a virtual server hits on its way in or out. Rules get added each time a virtual network interface comes up, and removed when the interface does down. 

For the sake of the remaining discussion, where I’ll be walking through some command examples, we’ll assume:

• peth0 is the physical network interface that the bridge connects to
• vif1.0 will be a virtual network interface connected to a domU, and the domU sees that device as eth0
• The domU is “allowed” to use IPv4 address 216.146.46.43/24, with a gateway IP of 216.146.46.1 for IPv4. 
• The domU is “allowed” to use IPv6 address 2607:f590:0:ffff::60/64, with a gateway IP of 2607:f590:0:ffff::1 for IPv6.
• The MAC address of the virtual network interfaced vif1.0 is 00:16:3E:38:B4:AC.
• The domU hostname is www.standingonthebrink.com.

Yes, you guessed it… all of the settings above are the real-world settings for the Xen VPS that is hosting this blog (with the exception of the virtual network interface name of vif1.0, which will change over time as the VPS is powered off and on).

In a nutshell, when you enable anti-spoofing as a parameter in the network-bridge script, the following iptables commands are issued when the bridge comes online:

  # Default policy for packets in the FORWARD chain is DROP.
  iptables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  iptables -F FORWARD  

  # Accept packets that are entering the bridge from the physical
  #   network interface peth0.
  iptables -A FORWARD -m physdev --physdev-in peth0 -j ACCEPT

Now, when each domU virtual network interface comes online, the following iptables command is run for each IP address that the domU is allowed to use, as specified in the www.standingonthebrink.com.cfg file found in /etc/xen/:

  # Accept packets coming into the bridge from the domU if its
  #   source IP address is authorized.
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --source 216.146.46.43 -j ACCEPT

When the domU’s virtual network interface goes offline, the same command is run except the -A is changed to -D to delete the rule.

Into the Bridge? Out of the Bridge? What’s the Difference?

It’s important to point out the difference between “into” and “out of” the bridge. For the purpose of thinking about rules governing the packets that are allowed to traverse the bridge, it can be helpful to think of the bridge as a little switch connecting the physical network interface of the server to the virtual network interface of the dom0 as well as all of the virtual network interfaces for each of the domUs. The rules we write in the FORWARD chain will tell the bridge what packets to allow to enter and/or leave the bridge.

All of the rules we’ll write are from the perspective of the network bridge. Perhaps a couple examples will help convey the concept:

• A packet coming from the Internet destined for the first domU -> Comes IN peth0 and goes OUT vif1.0.
• A packet coming from the first domU destined for the second domU -> Comes IN vif1.0 and goes OUT vif2.0.
• A packet sent from the first domU to somewhere on the Internet -> Comes IN vif1.0 and goes OUT peth0.

  # Accept packets coming into the bridge from the domU if its
  #  source IP address is authorized.
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --source 216.146.46.43 -j ACCEPT

Proposed Solution for ARP Poisoning Protection for Xen

Quite simply, we use arptables in a similar manner to using iptables. It can be a little tricky getting the rules right for what valid ARP traffic should be allowed, but with quite a bit of trial-and-error, we think we have it.

When the bridge comes online, we do:

  # Default policy for packets in the FORWARD chain is DROP.
  arptables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  arptables -F FORWARD

When each virtual network interface comes online, we do:

  # Accept ARP requests coming from the domU into the bridge.
  arptables -A FORWARD --opcode Request --in-interface vif1.0 -j ACCEPT

  # Accept ARP requests coming out of the bridge into the domU.
  arptables -A FORWARD --opcode Request --out-interface vif1.0 -j ACCEPT

  # Accept ARP replies coming out of the bridge from the physical
  #  network into the domU.
  arptables -A FORWARD --opcode Reply --out-interface vif1.0 \
    --in-interface peth0 -j ACCEPT

In addition to the above rules that get added when the virtual network interface comes online, we do the following for each valid IPv4 address:

  # Accept ARP replies coming from the domU into the bridge if they
  #  provide a valid and authorized IP address to MAC address pair.
  arptables -A FORWARD --opcode Reply --in-interface vif1.0 \
    --source-ip 216.146.46.43 --source-mac 00:16:3E:38:B4:AC -j ACCEPT

It’s this last rule that prevents the ARP poisoning attacks and the havoc wreaking by aliasing gateway IP addresses. If invalid ARP replies are denied, as far as I can tell, no harm can be done. When the virtual network interface goes offline, the same commands are run with -D instead of -A to delete the rules.

Proposed Solution for Preventing IPv4 Packet Sniffing

As mentioned earlier, the current IPv4 rules that prevent anti-spoofing are:

  # Accept packets that are entering the bridge from the
  #  physical network interface peth0.    
  iptables -A FORWARD -m physdev --physdev-in peth0 -j ACCEPT

  # Accept packets coming into the bridge from the domU if its
  #  source IP address is authorized.
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --source 216.146.46.43 -j ACCEPT

These work great for preventing a user from using an unauthorized IPv4 address, but the rule for incoming traffic from peth0 is not sufficiently explicit, in my opinion. It allows users to see the network traffic of other users on the same dom0. By tightening up the rules a bit, we can prevent this.

The proposed rules are as follows, for the bridge coming online:

  # Default policy for packets in the FORWARD chain is DROP.
  iptables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  iptables -F FORWARD

For each valid IPv4 address as each virtual network interface comes online:

  # Accept packets leaving the bridge going to the domU only if
  #  the destination IP for that packet matches an authorized IPv4
  #  address for that domU.
  iptables -A FORWARD -m physdev --physdev-out vif1.0 \
    --destination 216.146.46.43 -j ACCEPT

  # Accept packets coming into the bridge leaving the physical
  #  network interface peth0 only if the source IP for that packet
  #  matches an authorized IPv4 address for that domU.  
  iptables -A FORWARD -m physdev --physdev-in vif1.0 \
    --physdev-out peth0 --source 216.146.46.43 -j ACCEPT

Proposed Solution for IPv6 Anti-spoofing Protection for Xen

Quite simply, we use ip6tables with the exact same rule structure as shown above for IPv4. For the bridge coming online:

  # Default policy for packets in the FORWARD chain is DROP.
  ip6tables -P FORWARD DROP

  # Flush all existing rules in the FORWARD chain.
  ip6tables -F FORWARD

For each valid IPv6 address as each virtual network interface comes online:

  # Accept packets leaving the bridge going to the domU only if
  #  the destination IP for that packet matches an authorized IPv6
  #  address for that domU.
  ip6tables -A FORWARD -m physdev --physdev-out vif1.0 \
    --destination 2607:f590:0:ffff::60 -j ACCEPT

  # Accept packets coming into the bridge leaving the physical
  #  network interface peth0 only if the source IP for that packet
  #  matches an authorized IPv6 address for that domU.  
  ip6tables -A FORWARD -m physdev --physdev-in vif1.0 \
    --physdev-out peth0 --source 2607:f590:0:ffff::60 -j ACCEPT

Simple, eh?

Modifications to Xen Network Scripts

So, the above outlined the original IPv4 anti-spoofing approach, and used that as a guideline to IPv6 anti-spoofing, as well as protecting against ARP poisoning attacks. Now, here are the changes to make to actually get this new behavior.

It’s worth noting that these changes were done in a manner to minimize impact to the existing networking scripts (i.e., no refactoring was done). This was done in case these changes never made it out to the community, we still need them in place to effectively use Xen for VPS hosting, and we would have to merge in community changes alongside our changes with each new community release. Thus, there’s a lot of duplicated code. Should these changes make it into Xen, I would recommend refactoring to remove this duplication!

It’s worth noting also that a method was needed for storing the IPv6 address for a domU. This was done by adding the address to the www.standingonthebrink.com.cfg file under /etc/xen/ in the following format:

  #ipv6=2607:F590:0000:FFFF:0000:0000:0000:0060

The modifications to the Xen networking scripts include the ability to read this into the xenstore for usage.

Without further ado, the network script diffs, which were performed against xen-unstable on October 8, 2008:

• vif-common.sh.diff
• vif-bridge.diff
• network-bridge.diff

I would also like to acknowledge the DynDNS team for helping to test and put together these modifications, and in particular Pierre Beaumier and Matthew Horsfall from Dynamic Network Services Inc., who discovered the vulnerabilities and greatly helped in putting a solution together.

The last time I put together a demo video, I submitted it for review by the folks on Business of Software forum on the Joel on Software web site.

Some of their feedback included:

• Audio required post processing, since background noise interfered with the spoken voice.
• Video was too long, and less than 3 minutes in length would be best.
• Transitions from one screen to the next were abrupt and obtrusive.
• The last slide showing the URL for more info disappeared too quickly.
• Wear a plain-colored dress shirt instead of a striped dress shirt, which will help video compression.

With their feedback in mind, I set about creating demo videos showing a nifty feature of the Spring Server platform: If you lose network connectivity to your virtual private server, you can log in to springconsole.com with any SSH client, and get an out of band console connection to your server to fix the network problem. Essentially serial console access to a VPS with an SSH client.

Originally, I prepared a single demo video, but after review with some in-house folks, came to the conclusion that this would not be a “one size fits all” situation. Some people knew what serial console was and how it could be used, and just wanted to learn how to use it here. Others would have no idea what I was talking about.

Accordingly, two videos were put together. The first is a “I’ll show you on the whiteboard” informal introduction to what serial console is and what it’s used for. The second is “I’ll show you on my laptop” informal demo on actually using it.

Whiteboard Intro: What is Spring Console?

VPS Serial Console Demo

We’re going to put these in a DynDNS-branded Flash player and on the DynDNS content delivery network before they go live on our site, but wanted to get some initial feedback from folks here on the blog (even though they’re just hosted with veoh.com above).