ProFTPD module mod_redis



The mod_redis module enables ProFTPD support for caching data in Redis servers, using the hiredis client library.

Directives


RedisEngine

Syntax: RedisEngine on|off
Default: RedisEngine off
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.6rc5 and later

The RedisEngine directive enables or disables the mod_redis module, and thus the configuration of Redis support for the proftpd daemon.


RedisLog

Syntax: RedisLog path|"none"
Default: None
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.6rc5 and later

The RedisLog directive is used to specify a log file for mod_redis's reporting on a per-server basis. The file parameter given must be the full path to the file to use for logging.

Note that this path must not be to a world-writable directory and, unless AllowLogSymlinks is explicitly set to on (generally a bad idea), the path must not be a symbolic link.


RedisLogOnCommand

Syntax: RedisLogOnCommand commands format-name
Default: None
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.6rc5 and later

The RedisLogOnCommand directive configures the use of Redis for logging. Whenever one of the comma-separated list of commands occurs, mod_redis will compose a JSON object, using the LogFormat named by format-name as a template for the fields to include in the JSON object. The JSON object of that event will then be appended to a list stored in Redis, using format-name as the key name. Multiple RedisLogOnCommand directives can be used, for different log formats for different events.

More on the use of Redis logging, including a table showing how LogFormat variables are mapped to JSON object keys can be found here.

Example:

  LogFormat file-transfers "%h %l %u %t \"%r\" %s %b"
  RedisLogOnCommand APPE,RETR,STOR,STOU file-transfers

  LogFormat sessions "%{iso8601}"
  RedisLogOnCommand CONNECT,DISCONNECT sessions

In addition to specific FTP commands, the commands list can specify "ALL", for logging on all commands. Or it can include the "CONNECT" and "DISCONNECT" commands, which can be useful for logging the start and end times of a session. Note that RedisLogOnCommand does not currently support the logging classes that the ExtendedLog directive supports.


RedisServer

Syntax: RedisServer server [password]
Default: None
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.6rc5 and later

The RedisServer directive is used to configure the IP address/port of the Redis server that the mod_redis module is to use. For example:

  RedisServer 1.2.3.4:6379
or, for an IPv6 address, make sure the IPv6 address is enclosed in square brackets:
  RedisServer [::ffff:1.2.3.4]:6379

Alternatively, you can configure a Unix domain socket path using e.g.:

  RedisServer /var/run/redis.sock

An optional password parameter can be provided, for Redis servers which are password protected.


RedisTimeouts

Syntax: RedisTimeouts connect-millis io-millis
Default: RedisTimeouts 500 500
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.6rc5 and later

The RedisTimeouts directive configures timeouts to be used when communicating with the Redis server. The connect-millis parameter specifies a timeout, in milliseconds, to use when first connecting to the Redis server. The io-millis parameter specifies a timeout, in milliseconds, to use both when sending commands to Redis, and when reading responses.

The default is 500 milliseconds for both timeouts:

  RedisTimeouts 500 500


Installation

The mod_redis module is distributed with ProFTPD. To enable support and use of the Redis protocol in your proftpd daemon, use the --enable-redis configure option:
  $ ./configure --enable-redis ...
  $ make
  $ make install
This option causes the mod_redis module to be compiled into proftpd.

You may also need to tell configure how to find the hiredis header and library files:

  $ ./configure --enable-redis \
    --with-includes=/path/to/hiredis/include \
    --with-libraries=/path/to/hiredis/lib


Usage

Configuring Redis for use by other modules, e.g. mod_ban or mod_tls_redis:

  <IfModule mod_redis.c>
    RedisEngine on
    RedisLog /var/log/ftpd/redis.log
    RedisServer 127.0.0.1:6379
  </IfModule>

This example shows the use of Redis logging for all commands:

  <IfModule mod_redis.c>
    RedisEngine on
    RedisLog /var/log/ftpd/redis.log
    RedisServer 127.0.0.1:6379

    LogFormat redis "%h %l %u %t \"%r\" %s %b"
    RedisLogOnCommand ALL redis
  </IfModule>

Redis Logging
When using Redis logging, the following table shows how mod_redis converts a LogFormat variable into the key names in the JSON logging objects:

LogFormat Variable Key
 %A  anon_password
 %a  remote_ip
 %b  bytes_sent
 %c  connection_class
 %D  dir_path
 %d  dir_name
 %E  session_end_reason
 %{name}e  ENV:name
 %F  transfer_path
 %f  file
 %{file-modified}  file_modified
 %g  group
 %{gid}  gid
 %H  server_ip
 %h  remote_dns
 %I  session_bytes_rcvd
 %{iso8601}  timestamp
 %J  command_params
 %L  local_ip
 %l  identd_user
 %m  command
 %{microsecs}  microsecs
 %{millisecs}  millisecs
 %{note:name}  NOTE:name
 %O  session_bytes_sent
 %P  pid
 %p  local_port
 %{protocol}  protocol
 %r  raw_command
 %S  response_msg
 %s  response_code
 %T  transfer_secs
 %t  local_time
 %{transfer-failure}  transfer_failure
 %{transfer-status}  transfer_status
 %U  original_user
 %u  user
 %{uid}  uid
 %V  server_dns
 %v  server_name
 %{version}  server_version
 %w  rename_from

In addition to the standard LogFormat variables, the mod_redis module also adds a "connecting" key for events generated when a client first connects, and a "disconnecting" key for events generated when a client disconnects. These keys can be used for determining the start/finish events for a given session.

Here is an example of the JSON-formatted records generated, using the above example configuration:

  {"connecting":true,"timestamp":"2013-08-21 23:08:22,171"}
  {"command":"USER","timestamp":"2013-08-21 23:08:22,278"}
  {"user":"proftpd","command":"PASS","timestamp":"2013-08-21 23:08:22,305"}
  {"user":"proftpd","command":"PASV","timestamp":"2013-08-21 23:08:22,317"}
  {"user":"proftpd","command":"LIST","bytes_sent":432,"transfer_secs":4.211,"timestamp":"2013-08-21 23:08:22,329"}
  {"user":"proftpd","command":"QUIT","timestamp":"2013-08-21 23:08:22,336"}
  {"disconnecting":true,"user":"proftpd","timestamp":"2013-08-21 23:08:22,348"}
Notice that for a given event, not all of the LogFormat variables are filled in. If mod_redis determines that a given LogFormat variable has no value for the logged event, it will simply omit that variable from the JSON object.

Another thing to notice is that the generated JSON object ignores the textual delimiters configured by the LogFormat directive; all that matters are the LogFormat variables which appear in the directive.

Frequently Asked Questions

Question: How can I convert this SQL logging into the equivalent Redis logging?

  SQLNamedQuery upload FREEFORM "INSERT INTO ftplogs ('userid', 'server_ip', 'transfer_date', 'operation', 'protocol', 'client_ip', 'transfer_time', 'bytes_transfer', 'file_hash_type', 'file_hash', 'file_path', 'transfer_status') VALUES ('%u', '%H', NOW(), '%r', '%{protocol}', '%a', '%T', '%b', '%{note:mod_digest.algo}', '%{note:mod_digest.digest}', '%f', '%{transfer-status}')"
  SQLLog STOR upload
Answer: Since the JSON object key names are hardcoded in mod_redis, converting the above SQLNamedQuery into a suitable/matching LogFormat is the necessary step. Thus for example it might become:
  LogFormat upload "%u %H %{YYYY-MM-DD HH:MM:SS}t %r %{protocol} %a %T %b %{note:mod_digest.algo} %{note:mod_digest.digest} %f %{transfer-status}"
  RedisLogOnCommand STOR upload
Note that LogFormat does not provide a NOW() function, unlike many SQL databases, thus the %t variable is needed to provide/fill in that timestamp.


© Copyright 2017 The ProFTPD Project
All Rights Reserved