ProFTPD: Sendfile Support


What is "sendfile"?
Many Unix kernels provide a function called sendfile(2), a system call which provides a "zero-copy" way of copying data from one file (or socket) descriptor to another. The phrase "zero-copy" refers to the fact that all of the copying of data between the two descriptors is done entirely by the kernel, with no copying of data into userspace buffers. The normal way of copying data between files involves using the read(2) system call to read data from the source descriptor into a userspace buffer, then calling the write(2) system call to write that buffer to the destination descriptor. This copying of the data twice (once into the userland buffer, and once out from that userland buffer) imposes some performance and resource penalties. The sendfile(2) avoids these penalties by avoiding any use of userland buffers; it also results in a single system call (and thus only one context switch), rather than the series of read(2)/write(2) system calls (each system call requiring a context switch) usually used for data copying.

Unix kernels handle a socket as just another file descriptor; this means that sendfile(2) can be used to efficiently copy data from a file on disk to a network socket, e.g. for downloading a file.

Use of sendfile(2) in ProFTPD
For the above reasons, ProFTPD will, by default, attempt to use the sendfile(2) function for all downloads. However, there are a few cases where ProFTPD will specifically avoid the use of sendfile(2):

The use of sendfile(2) can also be explicitly configured at run-time by using the following in your proftpd.conf file:
  UseSendfile off

Sendfile support in the compiled proftpd daemon can also be disabled at compile time, by using the --disable-sendfile configure option, e.g.:

  $ ./configure --disable-sendfile ...
This is not recommended unless necessary.

Known Issues with sendfile(2)
As useful as the sendfile(2) function can be, there are unfortunately cases where bad implementations of the function can cause problems. These problems manifest as broken or aborted downloads, or as downloaded data being corrupted, or even as downloaded files being larger than the original file.

The sendfile(2) function requires support from both the Unix kernel and from the filesystem drivers which handle the filesystems for the file descriptors being used. The following operating systems have had reports of issues when using sendfile(2):

There have also been some issues reported from users of virtualization technologies such as OpenVZ and VMware.

There have been cases where it was the filesystems, rather than the kernels, which appeared to have been the culprits in sendfile(2) problems:

Again, if you encounter issues with downloading files from ProFTPD when those files reside on a networked or virtualized filesystem, try using "UseSendfile off" in your proftpd.conf.

Bugs in certain network cards have been reported on Linux, where the use of sendfile(2) triggers TCP checksum offloading bugs on these cards when using IPv6.

Bug 3081 also demonstrates an interesting sendfile(2) issue. It is a special case where the FTP client and server are on the same machine, sendfile(2) is used when downloading a file, and the client is downloading the file to the exact same filesystem path from which the server is reading the file. When this happens, proftpd will "hang". The fix for this situation is relatively simple: "Don't do that."

If your Unix kernel and your filesystems work together to support the use of sendfile(2) properly, you may run into one last potential issue with ProFTPD's ftptop and ftpwho utilities when sendfile(2) is used (at least for versions of proftpd prior to 1.3.4rc1). These utilities rely on data in ProFTPD's ScoreboardFile for providing their information, including transfer rates. The progress of a file being downloaded is tracked in the ScoreboardFile as part of the download process; specifically, as part of the normal read(2)/write(2) loop. When the sendfile(2) function is used, the ScoreboardFile is not updated with the progress (since all of the data copying is being handled by the kernel, not by proftpd code at that point). Thus for downloads which use sendfile(2), ftptop and ftpwho often report the transfer rate as "Inf".

The transfer rates for uploads are still reported properly by ftptop and ftpwho, however.

In proftpd-1.3.4rc1 and later, the UseSendfile directive changed. Now you can use UseSendfile in <Directory> sections and .ftpaccess files. This allows sites to disable use of sendfile(2) just for specific directories/filesystems where it might be a problem, e.g.:

  <Directory /path/to/nfs/files>
    UseSendfile off
  </Directory>

Instead of the usual on/off parameters, the UseSendfile directive can also take a byte length, or a file size percentage, as parameters. These parameters can be used so that sendfile(2) is used for more optimal downloads while still letting ftptop and ftpwho display download progress rates. To use this, the best way to use specify a file size percentage such as 10% or 25%; this means that sendfile(2) will be used for each 10% or 25% percent of the size of the file being downloaded:

  <Directory />
    UseSendfile 10%
  </Directory>