Overview

super_mediator is an IPFIX mediator for use with the yaf and SiLK tools. What is an IPFIX mediator? An IPFIX mediator is an intermediate entity between IPFIX Exporters and Collectors that can potentially provide aggregation, filtering, and modification of IPFIX records. It may provide conversion to or from IPFIX or a conversion of IPFIX transport protocols. super_mediator collects and processes yaf output (IPFIX files or via TCP, UDP, or Spread) and exports that data in IPFIX, JSON, or CSV text format to one or more IPFIX collectors such as rwflowpack, flowcap, or to text files that may be bulk uploaded to a database. MySQL support is provided for automatic import.

super_mediator can provide simple filtering upon collection or at export time. Any traditional flow field can be used in a filter, including IP address or IPset (requires SiLK IPset library).

super_mediator can be configured to pull the Deep Packet Inspection (DPI) data from yaf and export that information to another IPFIX collector, or simply export the data to a CSV file or JSON file for bulk upload into a database of your choice. Given MySQL credentials, super_mediator will import the files into the given database.

super_mediator can also be configured to perform de-duplication of DNS resource records, DPI data, and SSL/TLS certificate data exported by YAF. It will export the de-duplicated records in IPFIX, CSV, or JSON format. See the man pages and tutorials for more information.

Tools

super_mediator	super_mediator is an IPFIX mediator that provides filtering, de-duplication, and modification of IPFIX records from yaf or a previous invocation of super_mediator. It also provides conversion of transport protocols and configurable human-readable, text or JSON output.
super_table_creator	super_table_creator builds the MySQL database and tables used for auto insert with super_mediator.

Run-Time Configuration

super_mediator.conf

This document describes the syntax of the file used to configure an invocation of super_mediator.

While super_mediator may be run with only command-line options for a limited set of capabilities, using the super_mediator.conf file is required for advanced configuration, such as using multiple collectors and/or multiple exporters, filtering, and customized output.

SiLK Features

The SiLK features of super_mediator greatly increased in version 1.9.

Traditionally, super_mediator has been able to filter records based on whether the record's IP addresses appear in a SiLK IPSet. This feature required access to either the complete SiLK installation or the standalone SiLK IPSet library installation.

As of version 1.9, building with SiLK support gives super_mediator the capability of modifying a record's deep packet inspection subTemplateMultiList as follows:

Adding sub-records that include a silkFlowtypeId and silkSensorId similar to how SiLK's rwflowpack tool labels records. This is called Type and Sensor Labeling. The algorithm used to label records is similar to that of the "twoway" packing logic. For information on the SiLK tools, see https://tools.netsa.cert.org/silk/rwflowpack.html https://tools.netsa.cert.org/silk/packlogic-twoway.html
Adding sub-records to note whether the record's IP addresses were present in an IPSet (IPSet Labeling) The name of the IPSet or an alternate label is included in the sub-record.
Adding sub-records that record the Prefix Map (pmap) labels associated with either the IPFIX record's source and destination IP addresses or the record's protocolIdentifier paired with the sourceTransportPort and destinationTransportPort. This is Prefix Map Labeling. The sub-record includes the name of the Prefix Map file or an alternate label.

See the SiLK Support section below for details on including or excluding this feature when building super_mediator.

Build Dependencies

Required Dependencies

super_mediator requires glib 2.18.0 or later. Build and install glib before building super_mediator. Note that glib is also included in many operating environments or ports collections.

super_mediator requires libfixbuf 2.3.0 or any later 2.x release.

Optional Dependencies

Spread support requires Spread 4.1 or later. Build and install Spread before building super_mediator.

super_mediator can process compressed IPFIX files if the zlib library is installed.

If MySQL libraries are available, the super_table_creator program will also be built. Compiling with MySQL, will also provide additional capabilities for automatic upload into a MySQL database. To disable these capabilities and building of the super_table_creator configure super_mediator with --with-mysql=no.

X.509 MD5/SHA1 Hashing support requires super_mediator to be built with OpenSSL support. Build and install OpenSSL before building super_mediator. Use the --with-openssl option to ./configure to enable hashing support.

Optional SiLK Support

To enable the capabilities described in SiLK Features above, super_mediator must be built with either SiLK or the stand-alone SiLK IPSet support.

All features may be enabled when building with a complete SiLK installation. When building with a SiLK IPSet installation, only IPSet Labeling and IPSet filtering support is available.

The configure script always attempts to include the SiLK Features to the greatest extend possible. To disable the SiLK Features completely, specify the --without-skipset switch to configure.

If you want to include this support but discover that the configure script is unable to find the library you want it to use, specify --with-skipset=SILK_DIR where SILK_DIR is the directory where either SiLK or SiLK IPSet is installed. This switch may be included with any of the following options:

If you want to force super_mediator to use only the stand-alone SiLK IPSet library, specify --without-libsilk.

When the SiLK library is used, the new labeling features in version 1.9 may be independently disabled:

To disable IPSet filtering and IPSet labeling, specify --without-silk-ipset.

To disable Prefix Map labeling, specify --without-silk-pmap.

To disable Type and Sensor labeling, specify --without-silk-type-sensor.

Building super_mediator

super_mediator uses a reasonably standard autotools-based build system. The customary build procedure
(./configure && make && make install)
should work in most environments. Note that super_mediator finds libfixbuf using the pkg-config facility, so you may have to set the PKG_CONFIG_PATH variable on the configure command line if these libraries are installed in a nonstandard location, other than the prefix to which you are installing super_mediator itself.

super_mediator will look for MySQL and the SiLK library or SiLK IPSet library by default. However, if they are installed in a nonstandard location, providing the --with-mysql=MYSQL_CONFIG_PATH --with-skipset=SKIPSET_DIR options to ./configure will help in locating the libraries.

Tutorials

Quick and Easy super_mediator Setup (html)

A Quick and Easy guide for configuring super_mediator.

Running super_mediator as a Service (html)

Instructions for starting super_mediator as a service using systemctl (systemd support was added in v1.11.0) or using the traditional init.d-based service command.

super_mediator Tutorial (html)

Tutorial for installing, configuring, and working with super_mediator. This tutorial will show examples of various outputs and how to configure super_mediator accordingly.

Configuring YAF and super_mediator (html)

How-to guide on using yaf and super_mediator to collect DPI data and import that data into a MySQL database. SiLK Flow collection will also be described.

Configuring de-duplication (html)

Configuration guide for data de-duplication

Configuring SSL de-duplication (html)

Configuration guide for SSL certificate de-duplication

Known Issues

Similar to yaf, the destinationTransportPort information element contains ICMP type and code information for ICMP or ICMP6 flows; this is nonstandard and may not be interoperable with other IPFIX implementations.

Contact

Please send bug reports, feature requests, and questions to contact_email . We welcome bug fixes and patches.