Ovidiu Predescu's Weblog

Rsync over ssh: the dreaded "writefd_unbuffered failed to write 4 bytes to socket" error

I use rsnapshot to implement a backup solution for the computers in my home. Rsnapshot runs as a cron job on a ReadyNAS Pro Business Edition system with 6 2TB drives inside (a total of 7.4TB in an X-RAID2 configuration). It backs up the data from a Mac Pro using rsync over ssh. I would have used TimeMachine but its inability to reliably run on non-Apple hardware and with volumes larger than 2TB drives me nuts.

Every once in a while, especially when I have some new fresh data to be backed up, I see rsync start up but it mysteriously dies after a short while (in in /var/log/rsnapshot.log). Running the same command in a terminal gives the following error:

rsync: writefd_unbuffered failed to write 4 bytes to socket [generator]: Broken pipe (32)
rsync error: timeout in data send/receive (code 30) at io.c(1530) [generator=3.0.7]
rsync error: received SIGUSR1 (code 19) at main.c(1306) [receiver=3.0.7]

Both the source and the destination machines were using rsync 3.0.7. The command line I was running on the source machine (readynas):

root@readynas:~> /usr/bin/rsync -a -v --iconv=UTF-8 --timeout=180 --archive \
    --compress --delete --numeric-ids --relative --delete-excluded --copy-unsafe-links \
    --rsync-path="/opt/local/bin/rsync" --rsh="/usr/bin/ssh -p 22 \
    -o 'ClearAllForwardings yes' -o 'ServerAliveInterval 60'" \
    root@monster:/Volumes/BigDisk /backup/hourly.0/monster/BigDisk/

I upgraded rsync both on the source machine (the ReadyNAS), as well as on my Mac Pro to 3.0.8, by manually compiling the latest version. However the error still persisted.

Googling around didn't reveal any solution to the problem, which apparently goes back all the way to at least 2008. On one forum a poster suggested removing the compression from rsync and let ssh handle it. This works for small files but it tends to break on the larger files (larger than 15GB in size).

What seems to work best however is remove compression altogether. Another thing I've done is to use rsh instead of ssh. Here is the new command line (note the new paths to rsync 3.0.8 on each machine).

root@readynas:~> /usr/local/bin/rsync -a -v --iconv=UTF-8 --timeout=180 --archive \
    --delete --numeric-ids --relative --delete-excluded --copy-unsafe-links \
    --rsync-path="/usr/local/bin/rsync" --rsh="/usr/bin/rsh" \
    root@monster:/Volumes/BigDisk /backup/hourly.0/monster/BigDisk/