Ovidiu Predescu's Weblog

Rsync over ssh: the dreaded "writefd_unbuffered failed to write 4 bytes to socket" error

I use rsnapshot to implement a backup solution for the computers in my home. Rsnapshot runs as a cron job on a ReadyNAS Pro Business Edition system with 6 2TB drives inside (a total of 7.4TB in an X-RAID2 configuration). It backs up the data from a Mac Pro using rsync over ssh. I would have used TimeMachine but its inability to reliably run on non-Apple hardware and with volumes larger than 2TB drives me nuts.

Every once in a while, especially when I have some new fresh data to be backed up, I see rsync start up but it mysteriously dies after a short while (in in /var/log/rsnapshot.log). Running the same command in a terminal gives the following error:

rsync: writefd_unbuffered failed to write 4 bytes to socket [generator]: Broken pipe (32)
rsync error: timeout in data send/receive (code 30) at io.c(1530) [generator=3.0.7]
rsync error: received SIGUSR1 (code 19) at main.c(1306) [receiver=3.0.7]

Both the source and the destination machines were using rsync 3.0.7. The command line I was running on the source machine (readynas):

root@readynas:~> /usr/bin/rsync -a -v --iconv=UTF-8 --timeout=180 --archive \
    --compress --delete --numeric-ids --relative --delete-excluded --copy-unsafe-links \
    --rsync-path="/opt/local/bin/rsync" --rsh="/usr/bin/ssh -p 22 \
    -o 'ClearAllForwardings yes' -o 'ServerAliveInterval 60'" \
    root@monster:/Volumes/BigDisk /backup/hourly.0/monster/BigDisk/

I upgraded rsync both on the source machine (the ReadyNAS), as well as on my Mac Pro to 3.0.8, by manually compiling the latest version. However the error still persisted.

Googling around didn't reveal any solution to the problem, which apparently goes back all the way to at least 2008. On one forum a poster suggested removing the compression from rsync and let ssh handle it. This works for small files but it tends to break on the larger files (larger than 15GB in size).

What seems to work best however is remove compression altogether. Another thing I've done is to use rsh instead of ssh. Here is the new command line (note the new paths to rsync 3.0.8 on each machine).

root@readynas:~> /usr/local/bin/rsync -a -v --iconv=UTF-8 --timeout=180 --archive \
    --delete --numeric-ids --relative --delete-excluded --copy-unsafe-links \
    --rsync-path="/usr/local/bin/rsync" --rsh="/usr/bin/rsh" \
    root@monster:/Volumes/BigDisk /backup/hourly.0/monster/BigDisk/

Arduino Tiny Web Server - part 3

Update (September 3rd, 2011): The code is on Github: https://github.com/ovidiucp/TinyWebServer

A zip file of the latest version of Arduino TinyWebServer can be found here https://github.com/ovidiucp/TinyWebServer/zipball/master.

What is it?

Arduino TinyWebServer is a library implementation of a small web server running on an Arduino Duemilanove or Uno using the new Ethernet Shield released at the end of September 2010. The new Ethernet Shield adds a microSD card right on the board, in addition to fixing a number of hardware bugs from the previous iteration.

The library is meant to provide a Web interface to your Arduino project. It is not meant to be a full Web server implementation, just the minimum amount to let you get started quickly.

To save precious memory space on your Arduino, TinyWebServer encourages you to develop your web applications as AJAX applications. This means that you should delegate the page generation and complex UI interactions to the Web browser, instead of trying to handle it on the Arduino side of your project. The only thing the Arduino should implement is retrieving and displaying the internal state, as well as modifying it using HTTP POST requests.

Changes from the previous release

As you're probably aware, a new Ethernet Shield for Arduino was released at the end September 2010. I bought one and I'm pleased to report that it fixes an annoying hardware bug present in the old shield. In addition it has a new microSD slot that is actually accessible through a dedicated SPI SS pin. This means that you can use an Arduino Duemilanove or Uno with the Ethernet Shield with no modifications whatsoever.

I finally had some time to update the Arduino TinyWebServer library to make it work with the new shield. As you can see in the picture above, the setup is much easier than with the old Ethernet Shield and the data logging shield. The new library version drops support for this kludgy setup.

• Compared to the previous release, the new Arduino TinyWebServer release removes the dependency on the EthernetDHCP library. As a result you need to configure the IP address of the Ethernet shield manually inside the examples to match your network. Of course you can still add EthernetDHCP support to your own sketches if you wish.
• The library now uses the SdFat library instead of the Fat16. This allows the use of microSD cards larger than 2GB, formatted with FAT32, not only cards less than 2GB formatted with FAT16.
• The new release no longer uses a modified Ethernet library. It instead uses the existing Ethernet library that comes with the Arduino IDE.
• The code was updated to work with the latest Arduino IDE, version 021.
• I fixed a bug related to memory allocation and how the Ethernet Client objects were returned through the API.

In the new release I've added a new example that shows how you can control an LED from a regular Web browser. The video below demonstrates how it works.

Documentation

Arduino TinyWebServer is a library and as such requires you to write code customized for your own application. Some almost ready-to-go examples can be found inside the distribution in the examples/ directory. Follow the instructions in the README.txt files for each of the examples.

An explanation of the library's API can be found in this older post. Some things like the EthernetDHCP code no longer apply, so please look at the new examples for the updated code. Still most of the code is still applicable.

Transcode AVCHD to MPEG using ffmpeg

I recently bought a Panasonic Lumix DMC-LX5 camera, which records video in both MPEG and AVCHD formats. The camera has some nice improvements over the previous LX3 camera. I really like the extended zoom range, 24-90mm versus the 24-60mm in the old camera. Another feature I really love is the ability to zoom while shooting a movie. The sounds quality improved significantly as well, some videos I took with the old camera had some strange sound issues.

One thing that drove me nuts however was the inability of iMovie 7 or Final Cut Express to import the AVCHD movies I made with the camera. They would import only the first few minutes of the movie and invariably stop after that. The MacOS X software that comes with the camera doesn't handle movie files, and the Windows version sucks badly.

So I was stuck with no way of converting the AVCHD movies to an MPEG file format that can be viewed on MacOS X. Adobe Photoshop Lightroom 3, the program I use to catalog the photos and videos I take, does not understand AVCHD either.

You can however convert the .MTS files from the AVCHD directory created by the camera using the open source ffmpeg tool. To do this, make sure you have MacPorts installed on your computer, then install ffmpeg like this:

sudo port install ffmpeg +gpl +lame +x264 +xvid +mp3 +aac

Once you have ffmpeg installed, you can convert the MTS files like this:

ffmpeg -i <file>.MTS -vcodec mpeg4 -f mp4 -sameq <file>.mpg

How to build an Arduino project from the command line

For the projection screen controller I'm building I decided to use an Arduino hooked up to a GuruPlug. This way I offload the communication with the outside world to the GuruPlug, which runs a full Linux server, complete with Ethernet and WiFi connectivity. The Arduino is now used to control the motors and relays, and to receive input from an IR sensor, limit switches and infrared proximity sensors. This solution replaces the relatively cumbersome solution involving an Ethernet shield hooked up to the Arduino.. The software on the Arduino becomes a lot simpler too, without the Arduino TinyWebServer running on the Arduino board itself.

However it does have its own complications. Since the Arduino is now hooked up to a remote computer, I need to be able to build the C++ code for the Arduino on my host development machine (a MacOS X), push it to the GuruPlug running Linux and have the GuruPlug program the Arduino with the new binary.

The build process for the Arduino is handled internally by the Arduino IDE. The logic needed to build a typical Arduino project is hidden inside the Java program, which knows about the Arduino built-in libraries, as well as any additional user-defined libraries. I was hoping I could write a simple makefile to do the actual build, but the library dependencies are determined implicitly from the C header files included by your project's .pde file. It could be done using a combination of shell and awk, and a recursive invocation of make, but it's slightly more complicated. A major problem I hit was with file names containing spaces, which abound at least on MacOS X. Make uses a lot of built-in functions that use space as delimiter between the elements of a list, and it's impossible to use any built-in make functions.

Because of the space character problem, I gave up on make. I was looking for a build system that uses a more programatic approach, and so I found SCons. It's a nice build system built entirely in Python. I was about to write a build file for it when I found arscons, a small build file for Arduino-based projects. The funny thing is that the project was checked in Google Code only two weeks ago. Very timely indeed!

In classic open-source fashion - which I haven't done since I joined Google, back in 2003 - I made few improvements to the code and sent them to the author. I can now build my Arduino project on MacOS X, have it pushed to the GuruPlug and from there uploaded to the Arduino board connected to it.

Prerequisites

For this to work there are a few prerequisites for your development system. Make sure you have Python 2.5 or newer installed. Once you have Python installed, install SCons and the Python Serial module. On a Linux system you can install these by doing:

sudo apt-get install python2.5 python-serial scons

If you're on a Mac, you can get these slightly easier by installing MacPorts. If you do this, make sure you're running the python that comes with MacPorts, and not the default one in the MacOS X distribution. If you decide not to use MacPorts, download and install the packages manually, the process is pretty straightforward.

If you want to push the Arduino binary to a remote machine where the Arduino is connected, you also need to install the Python Fabric package on the host computer. On the remote computer (the GuruPlug in my case), you need to install Python Serial and avrdude. Again if your remote computer is a Linux box like the GuruPlug, you should be able to run apt-get to install the packages.

Once you have these packages installed, download the SConstruct file and put it in your Arduino project's directory. If you plan on remotely pushing the binary, download the pulsedtr.py file and place it in the same directory.

Building your Arduino project

After you install everything, you should be able to build your project using this command:

scons

This will only build the project. To build it and upload it to an Arduino connected to the same computer, run this:

scons upload

If you have a single Arduino board connected to your host computer, scons should be able to identify it automatically. If for some reason it doesn't, run scons like this:

scons ARDUINO_PORT=/dev/<arduino-tty-device> upload

If your Arduino board is not a Duemilanove, you might need to specify the board type using ARDUINO_BOARD. To see a list of possible values for this argument, run scons like this:

scons ARDUINO_BOARD=

Beside the parameters defined above, a few others can be defined in the command line:

• ARDUINO_HOME used to specify the location of your Arduino IDE package
• ARDUINO_VER the version of the Arduino IDE
• EXTRA_LIB if you have additional libraries that you'd like to use for the build, specify that directory using this argument.

For additional parameters that you can specify, look in the SConstruct file.

Deploying to an Arduino board connected to a remote computer

The most complex scenario is when you develop on a machine and push the binary to an Arduino connected to a remote computer (in my case that's a GuruPlug). This usually happens after the development phase, when you deploy your Arduino/GuruPlug project. At this point you don't want to take out the Arduino board from inside your project's box. This option allows you to reprogram the Arduino remotely, directly from your development machine.

To do this, you need to make sure you can login to the remote computer (GuruPlug in my case) using ssh without being asked for a password. You can do this by creating a new RSA key and placing the .pub key in the ~/.ssh/authorized_keys on the remote computer. Let's say your new key is in ~/.ssh/guruplug. You should be able to do this from your host computer, and be logged in to the remote machine automatically (obviously you need to replace user and remote-machine accordingly:

ssh -i ~/.ssh/guruplug user@remote-machine

You can now run scons and have it build, push the Arduino binary to the remote system, and upload the binary to the Arduino board connected to the remote system. Here's how:

scons REMOTE_HOST_STRING=user@remote-machine REMOTE_BOARD=atmega328 \
    REMOTE_TTY=/dev/ttyUSB0 SSH_KEY=~/.ssh/guruplug1 remote

Possible issues

I ran into a strange while trying to push the binary to the remote system using fabric. From the stack trace the error was inside paramiko, the library on which fabric is based. This problem was happening even after I created a new RSA key without a password. Each time I would run scons I got this error:

SSHException: invalid key

The error turned out to be inside ~/.ssh/known_hosts which apparently contained an invalid key, or a key paramiko couldn't parse properly. I removed all the suspect keys from it, and the error disappeared.

How to build a plastic box for a small project

I needed a small box to house some electronic boards for a remotely controlled projection screen project I'm building. The GuruPlug computer I'm using for the project has WiFi on board but no external antenna. A metal box would severely limit the radio waves, if not shield them completely, so I decided to go with a plastic box instead.

I couldn't find any plastic boxes that would fit my project, so I decided to build my own. Luckily there is a TAP Plastics store right in downtown Mountain View, and they carry a large selection of plastics. On their web site they have a lot of videos showing what can be done with plastics.

It turns out you can build a box using acrylic panels very easily. Acrylic is very easy to cut holes into and you can glue the panels of the box almost as easily as you can glue paper. Check out this video showing how.

Below is the finished plastic box. If you don't have the machines at home to cut the panels to length, you can have TAP Plastics cut the pieces for you. I just got a bunch of scrap plastic panels they had on sale for $1 a piece, cut them on my table saw, drilled the holes using a Dremel, and glued them together. The total cost for the box below was $2 (excluding the tools and the time used to build it :)

Pictures that document the building process of the box can be found in this Picasa album.

Here are few tips when building your own box:

• Draw a diagram of how everything fits together and make sure you take into account all dimensions, including the thickness of the panels.
• Measure everything using the metric system, the inches are too coarse grained to achieve a good precision.
• Before glueing make sure the pieces fit the way they're supposed to.
• Use eye and ear protection. Drilled plastic flies around and it's really hot, you don't want this nasty stuff to get into your eyes. The Dremel is loud and I prefer to maintain my hearing to listen to music.
• When cutting with a Dremel the right attachments will make your life a lot easier.

  	If you don't already own a Dremel, this Dremel kit has the basic tool plus a lot of goodies that you'll find useful later on.
  	The circle cutter attachment for cutting circles. If you have the Dremel kit above, this accessory is included in it.
  	The Dremel router attachement for cutting straight lines. The circle cutter attachment above has a little metal piece that allows you to cut straight lines with it as well. The router attachment however is much easier to move in a straight line while you push on it.

• Here are additional accessories I found very handy when building the box.

  	When gluing panels together at a 90° it's a lot easier to align and hold if you use an adjustable clamp like this.
  	If you cut the plastic yourself on a table saw, get a high quality blade for cutting plastic. With this blade I didn't need to route the edges of the plastic pieces I cut prior to gluing.