The mod_redis module enables ProFTPD support for caching data in Redis servers, using the hiredis client library.

Directives

• RedisEngine
• RedisLog
• RedisLogFormatExtra
• RedisLogOnCommand
• RedisLogOnEvent
• RedisOptions
• RedisSentinel
• RedisServer
• RedisTimeouts

RedisEngine

The RedisEngine directive enables or disables the mod_redis module, and thus the configuration of Redis support for the proftpd daemon.

RedisLog

The RedisLog directive is used to specify a log file for mod_redis's reporting on a per-server basis. The file parameter given must be the full path to the file to use for logging.

Note that this path must not be to a world-writable directory and, unless AllowLogSymlinks is explicitly set to on (generally a bad idea), the path must not be a symbolic link.

RedisLogFormatExtra

The RedisLogFormatExtra directive configures "extra" data to be added to the JSON logging, performed by RedisLogOnCommand and RedisLogOnEvent. The format-name parameter indicates that the json-object extra data should be added to that payload. This allows adding of custom JSON object key/value pairs. Example:

  LogFormat file-transfers "%h %l %u %t \"%r\" %s %b"

  RedisLogOnCommand RETR,STOR file-transfers %m:ftpxfer:%u
  RedisLogFormatExtra file-transfers {"custom_property1":"%m","custom_property2":"custom_value"}

RedisLogOnCommand

The RedisLogOnCommand directive configures the use of Redis for logging. Whenever one of the comma-separated list of commands occurs, mod_redis will compose a JSON object, using the LogFormat named by format-name as a template for the fields to include in the JSON object. The JSON object of that event will then be appended to a list stored in Redis, using format-name as the default key name. Multiple RedisLogOnCommand directives can be used, for different log formats for different events.

The optional key parameter, if present, specifies the value to use as the Redis key. This key parameter supports all of the LogFormat variables, thus you can use e.g.:

  RedisLogOnCommand RETR,STOR xferlog %m:ftpxfer:%u

More on the use of Redis logging, including a table showing how LogFormat variables are mapped to JSON object keys can be found here.

Example:

  LogFormat file-transfers "%h %l %u %t \"%r\" %s %b"

  # Only log to Redis for uploads in this directory
  <Directory /path/to/inbox>
    RedisLogOnCommand APPE,STOR,STOU file-transfers
  </Directory>

  # Only log to Redis for downloads from this directory
  <Directory /path/to/pub>
    RedisLogOnCommand RETR file-transfers
  </Directory>

  # ...but prevent Redis logging in this subdirectory
  <Directory /path/to/pub/subdir>
    RedisLogOnCommand none
  </Directory>

Note that RedisLogOnCommand does not currently support the logging classes that the ExtendedLog directive supports.

RedisLogOnEvent

The RedisLogOnEvent directive configures the use of Redis for logging. Whenever one of the comma-separated list of events occurs, mod_redis will compose a JSON object, using the LogFormat named by format-name as a template for the fields to include in the JSON object. The JSON object of that event will then be appended to a list stored in Redis, using format-name as the default key name. Multiple RedisLogOnEvent directives can be used, for different log formats for different events.

The optional key parameter, if present, specifies the value to use as the Redis key. This key parameter supports all of the LogFormat variables, thus you can use e.g.:

  RedisLogOnEvent READ ftp.download.%u.%m:%f

More on the use of Redis logging, including a table showing how LogFormat variables are mapped to JSON object keys can be found here.

Example:

  LogFormat sessions "%{iso8601} %a"
  RedisLogOnEvent CONNECT,DISCONNECT sessions

In addition to specific FTP commands, the events list can specify "ALL", for logging on all commands. Or it can include the "CONNECT" and "DISCONNECT" events, which can be useful for logging the start and end times of a session. Note that RedisLogOnEvent does support the logging classes that the ExtendedLog directive supports.

RedisOptions

The RedisOptions directive is used to configure various optional behavior of mod_redis.

Example:

  RedisOptions NoReconnect

The currently implemented options are:

• NoReconnect
  

  If the connection to Redis breaks unexpectedly, mod_redis will attempt to reconnect automatically. Use this option to disable the automatic reconnection.

RedisSentinel

The RedisSentinel directive is used to configure a list of IP addresses/ports of Redis Sentinels that the mod_redis module is to use, for discovering the location of named master database. For example:

  # Configure two Sentinels; the first master discovered will be used
  RedisSentinel 1.2.3.4:26379 5.6.7.8:36379

  # Configure two Sentinels on the default Sentinel port, and
  # look for the location of the master named "proftpd".
  RedisSentinel 1.2.3.4 5.6.7.8 master proftpd

  # Configure three Sentinels including an IPv6 address, with explicit ports,
  # and look for the location of the "proftpd" master.
  RedisSentinel redis1.example.com:26379 1.2.3.4:26379 [::1]:26379 master proftpd

The RedisSentinel directive can be used instead of the RedisServer directive, for discovering the Redis server to use. However, if your Redis server requires authentication, or supports multiple databases, then you will need to use RedisServer as well:

  # Use this to configure our password and database
  RedisServer 127.0.0.1:6379 redis redisr0cks 2

  # And use Sentinels to discover the true address/port
  RedisSentinel 1.2.3.4 5.6.7.8 9.10.11.12

In ProFTPD 1.3.8rc1 and later, it is possible to configure SSL/TLS parameters when connecting to Redis Sentinel. Most of the time, all that is needed for the SSL/TLS session is the CA (Certificate Authority) to use, for verifying the certificate presented by the Redis sentine. Thus:

  RedisSentinel ... ssl:true ssl-ca:/path/to/cacert.pem

  RedisSentinel ... ssl:true \
    ssl-ca:/path/to/cacert.pem \
    ssl-cert:/path/to/client-cert.pem \
    ssl-key:/path/to/client-key.pem

RedisServer

The RedisServer directive is used to configure the IP address/port of the Redis server that the mod_redis module is to use. For example:

  RedisServer 1.2.3.4:6379

  RedisServer [::ffff:1.2.3.4]:6379

Alternatively, you can configure a Unix domain socket path using e.g.:

  RedisServer /var/run/redis.sock

Optional username and password parameters can be provided, for Redis servers which are password protected.

As of ProFTPD 1.3.7rc1, the optional db-index parameter can be provided, for selecting the server-side Redis database by index:

  RedisServer 1.2.3.4:6379 user passwd 2

  RedisServer 1.2.3.4:6379 "" "" 2

In ProFTPD 1.3.8rc1 and later, it is possible to configure SSL/TLS parameters when connecting to Redis. Most of the time, all that is needed for the SSL/TLS session is the CA (Certificate Authority) to use, for verifying the certificate presented by the Redis server. Thus:

  RedisServer ... ssl:true ssl-ca:/path/to/cacert.pem

  RedisServer ... ssl:true \
    ssl-ca:/path/to/cacert.pem \
    ssl-cert:/path/to/client-cert.pem \
    ssl-key:/path/to/client-key.pem

RedisTimeouts

The RedisTimeouts directive configures timeouts to be used when communicating with the Redis server. The connect-millis parameter specifies a timeout, in milliseconds, to use when first connecting to the Redis server. The io-millis parameter specifies a timeout, in milliseconds, to use both when sending commands to Redis, and when reading responses.

The default is 500 milliseconds for both timeouts:

  RedisTimeouts 500 500

Installation

  $ ./configure --enable-redis ...
  $ make
  $ make install

You may also need to tell configure how to find the hiredis header and library files:

  $ ./configure --enable-redis \
    --with-includes=/path/to/hiredis/include \
    --with-libraries=/path/to/hiredis/lib

Usage

Configuring Redis for use by other modules, e.g. mod_ban or mod_tls_redis:

  <IfModule mod_redis.c>
    RedisEngine on
    RedisLog /var/log/ftpd/redis.log
    RedisServer 127.0.0.1:6379
  </IfModule>

This example shows the use of Redis logging for all commands:

  <IfModule mod_redis.c>
    RedisEngine on
    RedisLog /var/log/ftpd/redis.log
    RedisServer 127.0.0.1:6379

    LogFormat redis "%h %l %u %t \"%r\" %s %b"
    RedisLogOnCommand ALL redis
  </IfModule>

Redis Logging
When using Redis logging, the following table shows how mod_redis converts a LogFormat variable into the key names in the JSON logging objects:

In addition to the standard LogFormat variables, the mod_redis module also adds a "connecting" key for events generated when a client first connects, and a "disconnecting" key for events generated when a client disconnects. These keys can be used for determining the start/finish events for a given session.

Here is an example of the JSON-formatted records generated, using the above example configuration:

  {"connecting":true,"timestamp":"2013-08-21 23:08:22,171"}
  {"command":"USER","timestamp":"2013-08-21 23:08:22,278"}
  {"user":"proftpd","command":"PASS","timestamp":"2013-08-21 23:08:22,305"}
  {"user":"proftpd","command":"PASV","timestamp":"2013-08-21 23:08:22,317"}
  {"user":"proftpd","command":"LIST","bytes_sent":432,"transfer_secs":4.211,"timestamp":"2013-08-21 23:08:22,329"}
  {"user":"proftpd","command":"QUIT","timestamp":"2013-08-21 23:08:22,336"}
  {"disconnecting":true,"user":"proftpd","timestamp":"2013-08-21 23:08:22,348"}

Another thing to notice is that the generated JSON object ignores the textual delimiters configured by the LogFormat directive; all that matters are the LogFormat variables which appear in the directive.

Frequently Asked Questions

Question: How can I convert this SQL logging into the equivalent Redis logging?

  SQLNamedQuery upload FREEFORM "INSERT INTO ftplogs ('userid', 'server_ip', 'transfer_date', 'operation', 'protocol', 'client_ip', 'transfer_time', 'bytes_transfer', 'file_hash_type', 'file_hash', 'file_path', 'transfer_status') VALUES ('%u', '%H', NOW(), '%r', '%{protocol}', '%a', '%T', '%b', '%{note:mod_digest.algo}', '%{note:mod_digest.digest}', '%f', '%{transfer-status}')"
  SQLLog STOR upload

  LogFormat upload "%u %H %{YYYY-MM-DD HH:MM:SS}t %r %{protocol} %a %T %b %{note:mod_digest.algo} %{note:mod_digest.digest} %f %{transfer-status}"
  RedisLogOnCommand STOR upload