NAME

super_mediator.conf - Configuration file for super_mediator

DESCRIPTION

super_mediator(1) is an IPFIX Mediator to be used as a manifold for yaf(1) IPFIX export data.

super_mediator expects IPFIX data created by yaf or a previous invocation of super_mediator. It will collect other valid IPFIX data, but it expects certain information elements to be present to use particular features.

As part of its collection and export of network flow data, super_mediator needs to know how to collect it (e.g. listen on 18000/udp), what data to keep (ignore flow data unless the source port is 80), how to manipulate the data (deduplicate DNS records), and where to write it (export to 18001/tcp). While a rudimentary mediator can be created via the command line, more complex mediators require use of a configuration file, and this manual page describes the syntax of that file.

This manual uses super_mediator.conf as the name of the configuration file, but the file may have any name, and it does not have to reside in a particular location. The location of the file is specified by the --config switch to super_mediator.

Some command line arguments to super_mediator override settings declared in the configuration file. Command line arguments for exporters are ignored if a configuration file is specified.

General Concepts

The configuration file for super_mediator defines the following concepts:

Collector

A collector represents an input source for flow data from yaf. The source may be a transport protocol such as TCP or UDP, a directory that is periodically polled for IPFIX files, or a single file. super_mediator supports multiple collectors, and at least one collector is required. When a configuration file is used, the collector(s) must be specified in the file. (This is a change from super_mediator 1.x.)

Additional details:

Exporter

An exporter represents output from super_mediator. The exporter names the location of the output: a transport protocol such as TCP or UDP, a single file, or a directory into which files are written and periodically rotated. The exporter specifies the format of the output: binary IPFIX data, a text file in the JSON key-value format, or delimited text (for example, CSV). The exporter may also specify operations to perform on the data before it is written; these operations are introduced within this subsection. super_mediator supports multiple exporters, and it exits with an error if no exporters are specified. Each exporter processes the flow records read by every collector when no filters are defined. When a configuration file is used, the exporter(s) must be specified in the file.

Additional details:

Filter

A filter limits the flow records are passed out of a collector or into an exporter. A filter is composed of multiple comparisons, where each comparison specifies a flow attribute, an operator, and a value, for example sourceTransportPort <= 1024. Filters may be specified within a collector or exporter. In addition, super_mediator allows a single FILTER block to be specified whose rules apply to all collectors.

Additional details:

DNS Resource Record Export

An exporter may have DNS resource record export enabled. When enabled, the exporter examines the DNS DPI data on the flow records, and for every DNS-containing flow record it sees, it exports a summary record containing DNS-specific fields (dnsRRType, dnsRRName, dnsTTL, dnsId, dnsQueryResponse, dnsResponseCode, dnsAuthoritative, dnsSection, and smDNSData), yafFlowKeyHash, and optionally the flow record's five-tuple (source and destination addresses, source and destination ports, and IP protocol). These records are not deduplicated.

Additional details:

DNS Deduplication Records

super_mediator performs DNS deduplication for each exporter that enables it. When DNS deduplication is enabled, the exporter examines DNS DPI data and caches the unique {dnsRRName, dnsRRType, and rrvalue} tuples it sees. (Examples of an rrvalue are an IPv4 address or an SOA name.) These "DNS dedup" records contain only the unique information elements, the time when first seen, and a limited number of other fields. The records may be exported when first seen, when last seen, or after a certain time out or number of matches.

Additional details:

TLS/SSL Deduplication Records

When an exporter enables TLS/SSL Deduplication, the exporter examines TLS DPI data and keeps track of the unique {serial-number, issuer} tuples it sees. Records with unique tuples are exported when first seen and then again after a time out or some number of matches. These "SSL dedup" records contain only the unique information elements, the time when first seen, and a limited number of other fields. The certificate is exported separately from the SSL dedup record. (The SSL_CONFIG block is used both to configure SSL deduplication and to customize which TLS/SSL fields are written during TEXT export; for the latter, see "Customizing TLS/SSL Certificate Text Export" below.)

Additional details:

General Deduplication Records

When enabled, an exporter can perform deduplication on nearly any DPI information element value. The term general deduplication is used to distinguish this deduplication from that for DNS and TLS/SSL records. To enable general deduplication, a DEDUP_CONFIG block must be present and reference the exporter's name. Specifying information element IDs in this block causes the exporter to cache a tuple consisting of the values of fields having those information element IDs and either the source IP address, the destination IP address, or a hash of the flow's five-tuple. These "dedup" records are flushed after a certain time period or when the count reaches a value. To limit an exporter's output to only these records, specify DEDUP_ONLY in the exporter block.

Additional details:

Additional Deduplication Key

One may configure super_mediator to add an additional key for DNS-, SSL-, or general-deduplication. This key is comprised of groups of either observation domain IDs (observationDomainId) or VLAN IDs (vlanId). To use this additional key, first create a named VLAN_MAP or OBID_MAP block, and within the block assign a name to groups of IDs. Finally, specify the name of the map as the argument to a MAP() statement within a DNS_DEDUP block, a SSL_CONFIG block, or a DEDUP_CONFIG block.

Additional details:

Customizing TLS/SSL Certificate Text Export

A TEXT or JSON exporter may be configured to export either a subset of the TLS/SSL X.509 fields available or additional fields that are not normally written (assuming their values are available in the incoming data). This customization is done with an SSL_CONFIG block that references the exporter's name. The SSL_CONFIG block may specify issuer fields, subject fields, extension fields, and other fields. If this block is present for an exporter, the only TLS/SSL fields that are written are those explicitly specified within one of the lists. (The SSL_CONFIG block is used both to customize the textual export of TLS/SSL records and to configure SSL deduplication; for the latter, see "TLS/SSL Deduplication Records" above.)

Additional details:

Advanced Configuration of Text Output

The DPI_CONFIG block is used for advanced configuration of how all the TEXT exporters in the configuration file write records' Deep Packet Inspection (DPI) data. Use of the DPI_CONFIG block is optional.

The block configures the exporters to insert custom labels into the output of the DPI metadata, or if an exporter has contains the MULTI_FILES command, the DPI_CONFIG block configures the names of the output files. The block also allows the user to choose which DPI fields (e.g., user agent strings, urls) are exported and to ignore others. The labels may assist a database loading process determine into which table a particular line should be inserted; for example, the label could be the name of the database table the line should be loaded into.

Additional details:

Log File Contents

The log file contains process statistics such as uptime, total flows received, DNS flows received, and deduplicated flows. It also contains information about each of the collectors and exporters. In addition, it records the statistics messages received from YAF. Below are example log messages. Lines that begin with "SM" are statistics about super_mediator. The "SM" log messages contain the uptime, total number of flows received by all Collectors, any flows that were filtered by all collectors, and the total number of statistics (IPFIX options records) received. If DNS deduplication is enabled, super_mediator will also report the number of flows that contain DNS data (dns), the number of individual resource records (RRrecords), and the number of DNS records "exported" after deduplication (dedup). yaf statistics contain "YAF" and the Collector name that received the stats record. The Exporter line contains the number of flows exported, statistics exported, and information about bandwidth. If DNS deduplication is enabled, the Exporter will also report the number of DNS records exported. The Collector log message reports the number of flows received by that collector, the total number of statistics (IPFIX options) records, the total number of flows filtered, and the number of connections accepted.

SM: Uptime: 0d:0h:4m:23s, Total Flows: 93, Filtered: 0, Stats: 0

SM: dns: 10, RRrecords: 41 dedup: 0

C2: YAF ID: 10 IP: 10.20.11.51 Uptime: 0d:0h:5m:1s

C2: YAF Flows: 104 Packets: 13230 Dropped: 0 Ignored: 188 Out of Sequence: 0 Expired Frags: 0 Assembled Frags: 0

Exporter E4: 96 flows, 0 stats, 0.0005 Mbps, 192.00 bytes per record

Collector C2: 93 flows, 0 stats, 0 filtered, 1 connection

CONFIGURATION COMMAND QUICK REFERENCE

This section provides a brief overview of the complete syntax of the super_mediator configuration file syntax. For more explanation, refer to the "DETAILED SYNTAX" section.

The newline character separates commands. Comments begin with # and continue to the end of the line. Empty lines and lines containing only a comment or whitespace are ignored.

The commands in this section are presented alphabetically as a quick reference, but that presentation may separate related concepts.

In the description of the syntax, reserved words are displayed in a strong front, e.g., COLLECTOR, and placeholders for arguments are displayed in an emphasized font, e.g, COLLECTOR-NAME. Optional arguments are surrounded by asterisks (*).

Top-Level Commands, Quick Reference

(See "Top-Level Commands" for the complete details.)

The following commands are accepted outside of any other block.

COLLECTOR COLLECTOR-TYPE *COLLECTOR-NAME*

Begins a COLLECTOR block, used to describe an input stream. Multiple collectors may be specified, at least one is required. See "COLLECTOR Block, Quick Reference".

DEDUP_CONFIG *EXPORTER-NAME*

Begins an optional DEDUP_CONFIG block, to enable or configure general deduplication for one exporter. The EXPORTER-NAME is required unless exactly one exporter has been defined at the time the DEDUP_CONFIG block is parsed. Multiple DEDUP_CONFIG blocks may be specified. See "DEDUP_CONFIG Block, Quick Reference".

DNS_DEDUP *EXPORTER-NAME*

Begins an optional DNS_DEDUP block, to configure DNS deduplication for one exporter. The EXPORTER-NAME is required unless exactly one exporter has been defined at the time the DNS_DEDUP block is parsed. Multiple DNS_DEDUP blocks may be specified. See "DNS_DEDUP Block, Quick Reference".

DPI_CONFIG *INDEX-NAME*

Begins an optional DPI_CONFIG block, to configure how and where text output is written. Only one DPI_CONFIG block may be defined. See "DPI_CONFIG Block, Quick Reference".

EXPORTER EXPORTER-FORMAT EXPORTER-TYPE *EXPORTER-NAME*

Begins an EXPORTER block, used to describe an output stream. Multiple exporters may be specified, at least one is required. See "EXPORTER Block, Quick Reference".

FILTER

Begins an optional FILTER block, used to limit which flow records are processed across all collectors. See "FILTER Block, Quick Reference".

LOGLEVEL LOG-LEVEL

Sets the minimum level for logged messages. In increasing levels of verbosity, the supported log levels are QUIET, ERROR, WARNING, MESSAGE, and DEBUG.

LOG_DIR LOG-DIRECTORY-PATH

Specifies the directory in which log files are created.

LOG_FILE LOG-SPECIFIER

Specifies either a single file where log messages are written, a syslog(3) specifier, or the string stderr to write log messages to the standard error.

NO_STATS

Disables logging of statistics about the number and types of records read and wriitten, and the export of tombstone records.

OBID_MAP MAPNAME

Begins a OBID_MAP block, to allow consideration of observationDomainId when performing general deduplication. Multiple OBID_MAP maps may be defined. See "MAP Blocks, Quick Reference".

PIDFILE PIDFILE-NAME

Specifies the complete path to the file where super_mediator writes its process ID (PID) when told to run as a daemon (via the --daemon switch).

PRESERVE_OBDOMAIN

Tells super_mediator not to overwrite the observationDomainId value in incoming IPFIX records. This applies to all incoming records.

REWRITE_SSL_CERTS

Tells super_mediator to modify the IPFIX templates used to represent TLS/SSL certificate information. This applies to all incoming records.

SSL_CONFIG EXPORTER-NAME

Begins an SSL_CONFIG block, to configure SSL deduplication or the export of SSL certificates for one exporter. Multiple SSL_CONFIG blocks may be specified. See "SSL_CONFIG Block, Quick Reference".

STATS_TIMEOUT TIMEOUT-SECONDS

Specifies how often super_mediator logs statistics about the number of records read and written. This also configures how often tombstone records are written to IPFIX exporters.

TOMBSTONE_CONFIG

Begins an optional TOMBSTONE_CONFIG block, to configure the tombstone records generated by super_mediator. See "TOMBSTONE_CONFIG Block, Quick Reference".

USER_IE INFO-ELEMENT-ID INFO-ELEMENT-NAME *APPLICATION-LABEL*

Defines a new information element.

VLAN_MAP MAPNAME

Begins an optional VLAN_MAP block, to allow consideration of vlanId when performing general deduplication. Multiple VLAN_MAP maps may be defined. See "MAP Blocks, Quick Reference".

COLLECTOR Block, Quick Reference

(See "Defining an Input: The COLLECTOR Block" for the complete details.)

A COLLECTOR block defines an input to super_mediator. At least one collector must be defined, and multiple collectors may be defined. (In a change from super_mediator v1.x, the collector(s) must be defined in the configuration file when --config is specified on the command line.)

The following command is used at top-level to begin a COLLECTOR block.

COLLECTOR COLLECTOR-TYPE *COLLECTOR-NAME*

Begins a COLLECTOR block. The COLLECTOR-TYPE specifies the source of the IPFIX records; it may be one of

  • DIRECTORY_POLL — a directory to be polled periodically for files. Must specify PATH and either DELETE or MOVE; may specify LOCK and POLL.

  • SINGLE_FILE — a single file. Must specify PATH.

  • TCP — a TCP port, defaults to 18000 on all available interfaces. Modify with PORT and HOSTNAME.

  • UDP — a UDP port, defaults to 18000 on all available interfaces. Modify with PORT and HOSTNAME.

The following commands are used in the context of that block.

AND_FILTER

Specifies that each record is tested against all the comparison statements defined in this collector and all statements must be true for the collector to pass the record to the exporter(s).

COLLECTOR END

Closes the current COLLECTOR block.

DECOMPRESS_DIRECTORY FILE-PATH

Names FILE-PATH as the directory where compressed files are decompressed prior to being read by super_mediator.

DELETE

Used for a DIRECTORY_POLL collector to delete files after they are processed.

HOSTNAME HOSTNAME

Specifies the hostname or IP address of the interface for a TCP or UDP collector to listen on. The default is all available interfaces.

LOCK

Does nothing as this is currently unimplemented.

MOVE DIRECTORY-PATH

Names the directory to move files to after they are processed by a DIRECTORY_POLL collector.

PATH PATH

For a SINGLE_FILE collector, PATH is either the name of the file to read or the string stdin to read from the standard input. For a DIRECTORY_POLL collector, PATH names the directory to poll for files. (Unlike super_mediator v1.x, PATH is a directory path, not a globbing pattern.)

POLL POLL-TIME

Specifies how often super_mediator polls PATH for a DIRECTORY_POLL collector. The default is 30 seconds. Causes a fatal error when specified for any other type of collector.

PORT PORT

Specifies the port number for a TCP or UDP collector to listen on. The default is 18000. Causes a fatal error when specified for any other type of collector.

IE-NAME COMPARISON-OPERATOR COMPARE-VALUE

Limits the records that are returned from this collector. Each comparison specifies a flow attribute, an operator, and a value, for example sourceTransportPort < 1024. If the block does not contain AND_FILTER, any comparison returning true causes the record to be passed to the exporters, and all comparisons must be false for the record to be ignored. If AND_FILTER is present, any comparison returning false causes the record to be ignored, and all comparisons must be true for the record to be passed. See "Filter Block" for details on the comparison syntax.

DEDUP_CONFIG Block, Quick Reference

(See "General Deduplication: DEDUP_CONFIG" for the complete details.)

The following command is used at top-level to begin a DEDUP_CONFIG block.

DEDUP_CONFIG *EXPORTER-NAME*

Begins a DEDUP_CONFIG block that applies to EXPORTER-NAME. EXPORTER-NAME is required unless exactly one exporter has been defined at the time the DEDUP_CONFIG block is parsed.

The following commands are used in the context of that block.

ADD_EXPORTER_NAME

Adds the name of the exporter to the output. The name is added to the observationDomainName element for IPFIX and JSON output or as a separate column for TEXT output.

DEDUP_CONFIG END

Closes the current DEDUP_CONFIG block.

PREFIX FILE-PREFIX [ FIELD-LIST ]

Specifies that the information elements in FIELD-LIST are to be used for deduplification, that the values should be paired with the source IP address, and that FILE-PREFIX is how those records will be identified. For a TEXT exporter, FILE-PREFIX is the basename of the file in which the records are stored, and the exporter's PATH value must be a directory. For a JSON exporter, FILE-PREFIX is used as the label for the flow and PATH is treated normally. For an IPFIX exporter, FILE-PREFIX is unused.

PREFIX FILE-PREFIX { SIP | DIP | FLOWKEYHASH } [ FIELD-LIST ]

Similar to the preceding with how to cache the data is explicitly specified.

PREFIX FILE-PREFIX { | SIP | DIP | FLOWKEYHASH } MAP( MAPNAME ) [ FIELD-LIST ]

Similar to the preceding with groups of observationDomainId or vlanId values included as part of the deduplification key.

FLUSH_TIME FLUSH-SECONDS

Causes the record to be exported and its key removed from the cache if its unique key is not seen for FLUSH-SECONDS seconds. If the key is seen again, a new entry is created.

MAX_HIT_COUNT COUNT

Causes the record to be exported and its key removed from the cache once the unique key has been seen COUNT times. If the key is seen again, a new entry is created.

MERGE_TRUNCATED

Allows super_mediator to merge truncated values into complete cached records.

DNS_DEDUP Block, Quick Reference

(See "DNS Deduplication: DNS_DEDUP" for the complete details.)

The following command is used at top-level to begin a DNS_DEDUP block.

DNS_DEDUP *EXPORTER-NAME*

Begins an DNS_DEDUP block to configure DNS deduplication for EXPORTER-NAME. If only one exporter exists, the EXPORTER-NAME argument is not required.

The following commands are used in the context of that block.

ADD_EXPORTER_NAME

Adds the name of the exporter to the output. The name is added to the observationDomainName element for IPFIX and JSON output or as a separate column for TEXT output. This setting is ignored when a MAP() statement is present in the DNS_DEDUP block.

BASE64_ENCODE

Causes a TEXT exporter to encode domain names using Base64.

DNS_DEDUP END

Closes the current DNS_DEDUP block.

FLUSH_TIME FLUSH-SECONDS

Causes the key removed from the cache if its unique key is not seen for FLUSH-SECONDS seconds. If the key is seen again, a new entry is created.

LAST_SEEN

Tells the exporter to emit the record when it is flushed instead of when it is first seen.

MAP( MAPNAME )

Adds groups of observationDomainId or vlanId values to the unique key. See "MAP Blocks, Quick Reference".

MAX_HIT_COUNT COUNT

Causes the key removed from the cache once the unique key has been seen COUNT times. If the key is seen again, a new entry is created.

RECORDS [ DNS-RESOURCE-RECORD-TYPE-LIST ]

Limits deduplication to the the resource record types specified in the list. Valid values are 0 (NXDomain), 1 (A), 2 (NS), 5 (CNAME), 6 (SOA), 12 (PTR), 15 (MX), 16 (TXT), 28 (AAAA), 33 (SRV).

DPI_CONFIG Block, Quick Reference

(See "Textual Output Configuration: DPI_CONFIG" for the complete details.)

The following command is used at top-level to begin a DPI_CONFIG block.

DPI_CONFIG *INDEX-NAME*

Begins a DPI_CONFIG block named INDEX-NAME.

The following commands are used in the context of that block.

DPI_CONFIG END

Closes the DPI_CONFIG block.

TABLE TABLE-NAME [ FIELD-LIST ]

Tells a TEXT exporter to use TABLE-NAME as the label for lines that contain the information element(s) in FILED-LIST.

If MULTI_FILES is present in the exporter, TABLE-NAME is instead used as the basename of the file that contains the element(s) in FILED-LIST.

EXPORTER Block, Quick Reference

(See "Defining an Output: The EXPORTER Block" for the complete details.)

An EXPORTER block defines an output from super_mediator. At least one exporter must be defined, and multiple exporter may be defined.

The following command is used at top-level to begin an EXPORTER block.

EXPORTER EXPORTER-FORMAT EXPORTER-TYPE *EXPORTER-NAME*

Begins an EXPORTER block. The EXPORTER-FORMAT specifies the format of the output:

  • IPFIX — The output is binary IPFIX files or IPFIX written to a socket.

  • JSON — The output is text in the JSON format; may not be written to a socket.

  • TEXT — The output is text, where a line generally represents a record and a delimiter separates the record's values. May not be written to a socket.

The EXPORTER-TYPE specifies where the output is written:

  • ROTATING_FILES — a directory where files are written and periodically rotated. Must specify PATH and ROTATE_INTERVAL. May specify MOVE or LOCK.

  • SINGLE_FILE — a single file, or a directory in some bizarre circumstances because this is all a big mess. Must specify PATH.

  • TCP — a TCP port, defaults to 18001. Must specify HOSTNAME and may change the default with PORT. Only valid for IPFIX exporters.

  • UDP — a UDP port, defaults to 18001. Must specify HOSTNAME and may change the default with PORT. Only valid for IPFIX exporters.

The following commands are used in the context of that block.

AND_FILTER

Specifies that each record is tested against all the comparison statements defined in this exporter and all statements must be true for the exporter to process the record.

DEDUP_ONLY

Causes the exporter to export only general deduplication records that are configured with a DEDUP_CONFIG block.

DEDUP_PER_FLOW

Does nothing as it is currently unimplemented.

DELIMITER DELIM

Only valid for a TEXT exporter, specifies the delimiter to use between top-level fields. DELIM must be a single character.

DNS_DEDUP

Enables DNS deduplication for this exporter in addition to the typical flow record export.

DNS_DEDUP_ONLY

Enables DNS deduplication for this exporter and causes the output to contain only DNS deduplicated records. No other flow data or DPI data is written by this exporter.

DNS_RESPONSE_ONLY

Causes DNS query records to be ignored when DNS_RR or DNS_RR_ONLY is active. This option is ignored if no DNS DPI data is present in the flow.

DNS_RR *FULL*

Valid only for IPFIX exporters, enables export of DNS-specific records for this exporter in addition to the typical flow record export. The DNS resource records are not deduplicated, and every (unfiltered) resource record seen by super_mediator is exported. When the FULL keyword is present, the output record includes the entire 5-tuple and vlanId.

DNS_RR_ONLY *FULL*

Valid only for IPFIX exporters, enables export of DNS-specific records for this exporter and causes the output to contain only the DNS-specific records. No other flow data or DPI data is written by this exporter. When the FULL keyword is present, the output record includes the entire 5-tuple and vlanId.

DPI_DELIMITER DPI-DELIM

Only valid for a TEXT exporter, specifies the delimiter to use between DPI fields. DPI-DELIM must be a single character.

DPI_FIELD_LIST [ DPI-IE-LIST ]

Only valid for a TEXT exporter, specifies the DPI information elements to be included in the output. DPI-IE-LIST is a comma-separated list of integers representing elements under the CERT IPFIX enterpriseId.

DPI_ONLY

Causes the exporter to ignore flow records that do not have DPI information. This is equivalent to a comparison filter of silkAppLabel != 0.

ESCAPE_CHARS

Only valid for a TEXT exporter, specifies that non-printable ASCII characters (characters outside decimal range 32 to 127) should be printed as \uXXXX where XXXX is the character's code point in hexadecimal. In addition, backslash (\), double quote ("), and the delimiter are printed with a preceding backslash.

EXPORTER END

Closes the current EXPORTER block.

FIELDS [ FIELD-LIST ]

Valid only for a TEXT exporter, specifies the information elements to be written for each record.

FLOW_ONLY

Causes the exporter to strip the DPI information from the flow records prior to export. This is useful when sending data to a tool that does not process the DPI data, such as SiLK's rwflowpack(8).

GZIP_FILES

Tells the exporter to compress the output files with gzip(1) after it has finished writing them.

HOSTNAME HOSTNAME

Only valid and required for a TCP or UDP exporter, specifies the hostname or IP address to connect to.

INVARIANT
LOCK

Causes super_mediator to prepend a single dot . to an output file that it is currently writing to. This is can be used with SiLK's rwsender(8) to prevent it from processing an incomplete file. Once the file is closed, the dot is removed from the filename.

METADATA_EXPORT
MOVE FILE-DIRECTORY

Tells the exporter to move the output file(s) to FILE-DIRECTORY after it has finished writing them.

MULTI_FILES

Only valid for a TEXT exporter, causes records to be segregated by application label and written to separate files.

MYSQL_DATABASE DATABASE-NAME

Only valid for a TEXT exporter and available only when super_mediator is compiled with MySQL support, causes super_mediator to load the generated text file into DATABASE-NAME. Requires MYSQL_PASS and MYSQL_USER, and may use MYSQL_HOST and MYSQL_TABLE.

MYSQL_HOST HOSTNAME

The host to contact when loading data into a MySQL database instead of localhost. See MYSQL_DATABASE above.

MYSQL_PASSWORD PASSWORD

The password to use when loading data into MySQL database. See MYSQL_DATABASE above.

MYSQL_TABLE TABLE-NAME

The MySQL table-name to used when loading data into MySQL database. Required either FLOW_ONLY or DNS_DEDUP_ONLY to also be present. See MYSQL_DATABASE above.

MYSQL_USER USER-NAME

The MySQL user-name to used when loading data into MySQL database. See MYSQL_DATABASE above.

NO_FLOW

Disables export of flow records.

NO_FLOW_STATS

Only valid for a TEXT exporter, disables the export of YAF flow stats records.

NO_INDEX
NO_STATS

Disables export of YAF flow stats records and tombstone records.

PATH PATH

Names a file, a filename pattern, or a directory where the exporter is to write its output. For a SINGLE_FILE exporter, may be - to denote the standard output.

PORT PORT

Specifies the port number for a TCP or UDP exporter to connect to. The default is 18001. Causes a fatal error when specified for any other type of collector.

Only valid for a TEXT exporter, causes each output file to begin with a header that names the information elements in the file. The element names are separated using the same delimiter as the values.

REMOVE_EMPTY_FILES

Does nothing as this is always enabled.

REMOVE_UPLOADED

Causes the output files to be removed after they have been loaded into the MySQL database.

ROTATE_INTERVAL ROTATE-SECONDS

Only valid and required for a ROTATING_FILES exporter, specifies how often output files are closed and a new file opened.

SSL_CERT_HASH_MD5

Tells the exporter to include the MD5 hash of the entire X.509 certificate in the output.

SSL_CERT_HASH_SHA1

Tells the exporter to include the SHA1 hash of the entire X.509 certificate in the output.

SSL_DEDUP

Enables TLS/SSL deduplication for this exporter in addition to the typical flow record export.

SSL_DEDUP_ONLY

Enables TLS/SSL deduplication for this exporter and causes the output to contain only TLS/SSL deduplicated records. No other flow data or DPI data is written by this exporter.

STATS_ONLY

Causes the exporter to write only YAF flow statistics records, or re-enables export of YAF flow stats when specified after FLOW_ONLY or DPI_ONLY.

UDP_TEMPLATE_TIMEOUT TIMEOUT-MINS

Ignored. The stated purpose is to control how often the IPFIX templates are resent for a UDP exporter.

IE-NAME COMPARISON-OPERATOR COMPARE-VALUE

Limits the records that are processed by this exporter. Each comparison specifies a flow attribute, an operator, and a value, for example sourceTransportPort < 1024. If the block does not contain AND_FILTER, any comparison returning true causes the record to be processed by the exporter, and all comparisons must be false for the record to be ignored. If AND_FILTER is present, any comparison returning false causes the record to be ignored, and all comparisons must be true for the record to be processed. See "Filter Block" for details on the comparison syntax.

FILTER Block, Quick Reference

(See "Filter Block" for the complete details.)

A single FILTER block may be specified in the configuration file. The comparisons in this block apply to records read by any collector.

The following command is used at top-level to begin a FILTER block.

FILTER

Begins a filter block.

The following commands are used in the context of that block.

AND_FILTER

Specifies that a record is tested against all the comparison statements defined in this filter block and all statements must be true for the exporter(s) to see the record.

FILTER END

Closes the current FILTER block.

IE-NAME COMPARISON-OPERATOR COMPARE-VALUE

Limits the records that are returned from this collector. Each comparison specifies a flow attribute, an operator, and a value, for example sourceTransportPort < 1024. If the block does not contain AND_FILTER, any comparison returning true causes the record to be passed to the exporters, and all comparisons must be false for the record to be ignored. If AND_FILTER is present, any comparison returning false causes the record to be ignored, and all comparisons must be true for the record to be passed. See "Filter Block" for details on the comparison syntax.

Limits the records that are passed to the exporter(s). Each comparison specifies a flow attribute, a value, and an operator, for example sourceTransportPort < 1024. If the comparison is FALSE, the record is not seen by any exporter. Otherwise, the record is passed to the exporter(s) if either the AND_FILTER keyword is not present or all other comparison statements in this block are also true.

MAP Blocks, Quick Reference

(See "MAP Blocks" for the complete details.)

The OBID_MAP and VLAN_MAP blocks may be used to add a key to the unique tuple used for DNS-, SSL-, or general-deduplication. This key is comprised of groups of either observation domain (observationDomainId) or VLAN ID (vlanId) values. These blocks map the IDs to names; multiple IDs may map to the same name. The block itself has a name which is used as the argument to a MAP() statement to reference the map within a DNS_DEDUP block, a SSL_CONFIG block, or a DEDUP_CONFIG block. During deduplication, the record's observationDomainId or vlanId is mapped to the name which is used as part of the key for deduplication. During export, the name is used as the value for observationDomainName.

The following commands are used at top-level to begin a OBID_MAP block or a VLAN_MAP block.

OBID_MAP MAPNAME

Creates a map named MAPNAME to hold observationDomainId values.

VLAN_MAP MAPNAME

Creates a map named MAPNAME to hold vlanId values.

The following commands are used in the context of that block.

DISCARD_OTHER

Causes the deduplication code to ignore a record if its observationDomainId or vlanId value is not explicitly listed in this map.

OBID_MAP END

Closes the current OBID_MAP block.

VLAN_MAP END

Closes the current VLAN_MAP block.

GROUP-NAME OTHER

Causes this map to use GROUP-NAME for any value not explicitly listed.

GROUP-NAME [ INTEGER-LIST ]

Maps each of the values in INTEGER-LIST to the name GROUP-NAME.

SSL_CONFIG Block, Quick Reference

(See "TLS/SSL Deduplication: SSL_CONFIG Part 1" and "TLS/SSL Certificate Export: SSL_CONFIG Part 2" for the complete details.)

The SSL_CONFIG block is used to configure TLS/SSL deduplication and to configure X.509 certificate export for a TEXT exporter.

The following command is used at top-level to begin a SSL_CONFIG block.

SSL_CONFIG EXPORTER-NAME

Begins an SSL_CONFIG block to figure TLS/SSL deduplication or the certificate fields for the exporter named EXPORTER-NAME.

The following commands are used in the context of that block.

ADD_EXPORTER_NAME

Adds the name of the exporter to the output for deduplicated TLS/SSL records. The name is added to the observationDomainName element for IPFIX and JSON output or as a separate column for TEXT output. This setting is ignored when a MAP() statement is present in the SSL_CONFIG block.

CERT_FILE FILENAME

Valid only for a TEXT exporter with SSL_DEDUP enabled, causes certificates to be written to the file named FILENAME.

EXTENSIONS [ FIELD-LIST ]

Specifies the names of certificate extension fields to include in the output of a TEXT or JSON exporter. See also ISSUER, OTHER, and SUBJECT.

FLUSH_TIME FLUSH-SECONDS

For TLS/SSL deduplication, causes the record to be exported and its key removed once its unique key has not been seen in FLUSH-SECONDS seconds. If the key is seen again, a new entry is created.

ISSUER [ FIELD-LIST ]

Specifies the names of certificate issuer fields to include in the output of a TEXT or JSON exporter. See also EXTENSIONS, OTHER, and SUBJECT.

MAP( MAPNAME )

Adds an additional field to the key used for TLS/SSL deduplication. See "MAP Blocks, Quick Reference".

MAX_HIT_COUNT COUNT

For TLS/SSL deduplication, once the unique key has been seen COUNT times, the record is exported and the key is removed from the cache. If the key is seen again, a new entry is created.

OTHER [ FIELD-LIST ]

Specifies the names of other certificate fields to include in the output of a TEXT or JSON exporter. See also EXTENSIONS, ISSUER, and SUBJECT.

SSL_CONFIG END

Closes the current SSL_CONFIG block.

SSL_DEDUP

Enables TLS/SSL deduplication for EXPORTER-NAME.

SUBJECT [ FIELD-LIST ]

Specifies the names of certificate subject fields to include in the output of a TEXT or JSON exporter. See also EXTENSIONS, ISSUER, and OTHER.

TOMBSTONE_CONFIG Block, Quick Reference

(See "Tombstone Block" for the complete details.)

This block is for configuring how super_mediator creates tombstone records.

The following command is used at top-level to begin a TOMBSTONE_CONFIG block.

TOMBSTONE_CONFIG

The following commands are used in the context of that block.

GEN_TOMBSTONE

Tells super_mediator to generate and output tombstone records. Tombstone records will be sent to any exporter that is configured to export yaf stats records. See other documentation for the configuration of stats.

TOMBSTONE_CONFIG END

Closes the current TOMBSTONE_CONFIG block.

TOMBSTONE_CONFIGURED_ID TOMBSTONE-CONFIGURED-ID

Sets the value of the certToolExporterConfiguredId element in tombstone records. If not specified, a value of 0 is used. If present, the TOMBSTONE_CONFIGURED_ID command will override the default id value of 0 with the given TOMBSTONE-CONFIGURED-ID. The valid range is 0 to 65535.

DETAILED SYNTAX

When parsing the super_mediator configuration file, blank lines (an empty line or a line containing only whitespace characters) are ignored. At any location in a line outside of a double quoted string, the character # indicates the beginning of a comment, which continues to the end of the line. These comments are ignored.

All other lines begin with optional whitespace and must contain a complete command. Commands have a command name and zero or more arguments to the command. The command and its arguments are separated by one or more space or tab characters. Command names are a sequence of characters. Arguments may either be double quoted strings ("an example") or textual atoms: a sequence of ASCII letters, numbers, and the symbols _, -, /, and @.

Integer numbers may be specified either as decimal or as hexadecimal when preceded by 0x or 0X. The syntax supports non-negative integers, with a typical maximum of 2147483647 (0x7fffffff), though the maximum is sometimes greater.

There are ten main contexts for commands:

The collector block, exporter block, filter block, dns-dedup block, ssl-config block, dpi-config block, and dedup-config block contexts are used to describe individual features of collectors, exporters, filters, DNS deduplication, SSL configuration and deduplication, DPI export, and general deduplication, respectively.

The valid commands for each context are described below.

Top-Level Commands

(See also "Top-Level Commands, Quick Reference".)

In addition to the commands to begin a specific block (COLLECTOR, EXPORTER, et cetera), the top-level context supports the following commands:

PRESERVE_OBDOMAIN

By default, super_mediator copies the observation domain identifier from the incoming IPFIX messages' headers to the records it exports, overwriting any previous observationDomainId value in the records. The PRESERVE_OBDOMAIN keyword tells super_mediator to preserve the incoming records' value of the observationDomainId. If the incoming records do not contain that element, the records exported by super_mediator will contain an observation domain of zero.

REWRITE_SSL_CERTS

If specified, super_mediator will, for all IPFIX exporters, rewrite the TLS/SSL certificate records that are represented in a subTemplateList on the SSL DPI record (template 51722 (0xca0a)). The rewriting changes the structure of template 51723 (0xca0b) and eliminates template 52756 (0xce14). The new SSL certificate record has explicit information elements for parts of the certificate's issuer and subject instead of having data stored in a list of key-value pairs. This setting must be used when super_mediator is reading IPFIX records from an upstream super_mediator instance that rewrote its SSL records.

NO_STATS

The top level NO_STATS keyword causes the collector(s) to ignore all yaf process statistics records they receive. These "yaf stats" records are not logged to the super_mediator log file (if specified) nor are they exported by any of the defined exporters.

super_mediator also keeps process statistics about how many flows it receives, filters, and exports, and it logs this data every 5 minutes (or the value given by STATS_TIMEOUT. If NO_STATS is present, these statistics messages are not logged.

STATS_TIMEOUT TIMEOUT-SECOND

If present, super_mediator logs process statistics every TIMEOUT-SECOND seconds instead of the default 600 seconds (5 minutes). If set to 0, super_mediator does not log statistics.

LOGLEVEL LOG-LEVEL

Specify minimum level for logged messages. In increasing levels of verbosity, the supported log levels are QUIET, ERROR, WARNING, MESSAGE, and DEBUG. The default logging level is WARNING. This level only logs critical errors or potential problems. The MESSAGE level will log all yaf process statistics it receives, along with periodic process statistics about itself. Setting the LOG-LEVEL to QUIET, ERROR, or WARNING will prevent these messages from being logged. The DEBUG level will report any I/O operations, such as opening, closing, moving, and deleting of input and output files in addition to all yaf and super_mediator process statistics. Specifying --quiet or --verbose on the command line overrides the value specified in this command.

LOG_FILE LOG-SPECIFIER

Specifies the destination for log messages. LOG_FILE can be a syslog(3) facility name, the special value stderr for standard error, or the absolute path to a file for file logging. The default log specifier is stderr. The log level can be specified by the LOGLEVEL keyword. See also LOG_DIR.

LOG_DIR LOG-DIRECTORY-PATH

If present, super_mediator writes log files to the directory LOG-DIRECTORY-PATH. LOG-DIRECTORY-PATH must be a complete directory path. The log files are named "LOG-DIRECTORY-PATH/sm-YYYYMMDD.log" where YYYYMMDD is the current date. The log files are rotated at midnight local time. When the log files are rotated a new log is opened, the previous file is closed, and gzip(1) is invoked on the previous day's log file. (Old log files are not removed by super_mediator.)

PIDFILE PIDFILE-NAME

If present and super_mediator is run in daemon mode (by specifying --daemon on the command line), super_mediator writes the process ID (pid) to the file specified by PIDFILE-NAME. PIDFILE-NAME should be the complete path to the file.

USER_IE INFO-ELEMENT-ID INFO-ELEMENT-NAME *APPLICATION-LABEL*

If present, add the Information Element with ID INFO-ELEMENT-ID and INFO-ELEMENT-NAME name to the Information Model. The Information Element will have the CERT Private Enterprise Number. If user-defined Information Elements are not added prior to collection, super_mediator will ignore the information element and emit a warning similar to:

BasicList Decode Error: No Information Element with ID 254 defined

Any user-defined information element defined in the yafDPIRules.conf file, should also be added to the super_mediator.conf in order for super_mediator to collect the element. If APPLICATION-LABEL is present, then this element will be added to the appropriate file if using MULTI_FILES. For example, if APPLICATION-LABEL is set to 80, then the information element will be added to the http.txt file by default. If you are using custom lists or a DPI_FIELD_LIST, setting APPLICATION-LABEL is not necessary. If DPI_CONFIG is set for custom tables, do not set APPLICATION-LABEL.

USER_IE 999 my_info_element
USER_IE 1002 http_other_field 80

Defining an Input: The COLLECTOR Block

(See also "COLLECTOR Block, Quick Reference".)

A collector block defines an input to super_mediator. A collector has a type that indicates whether the data is coming from a single file, from a directory of files, or a network socket. The collector may be given a name that is used in log messages and as value of observationDomainName in some outputs. If no name is specified, one is generated.

The statements within the block specify details about the source of the data (for example, the name of the file or directory). When polling a directory, the collector block must specify the disposal of files after they are processed.

The collector block may also contain filtering comparisons that limit which records are passed out of the collector. It is also possible to specify comparisons that apply to all collectors; see "Filter Block".

Multiple collectors may be defined in the configuration file, and at least one collector must be defined. super_mediator v2.0, unlike previous releases does not allow collectors to be defined on the command line when the --config switch is given.

COLLECTOR Block Commands

The COLLECTOR command is specified in the top-level context to begin a collector definition. All other commands in this section are accepted in the context of the collector block. The COLLECTOR END statement ends the collector block and sets the context back to top-level.

COLLECTOR COLLECTOR-TYPE *COLLECTOR-NAME*

The COLLECTOR command begins a new collector definition. The arguments to the COLLECTOR command are the type of collector to be defined and an optional COLLECTOR-NAME. If COLLECTOR-NAME is not provided, super_mediator uses C1, C2, C3, et cetera. The COLLECTOR-TYPE must be one of the following:

TCP

This collector listens for connections from yaf on a TCP port. The default is to listen on port 18000 on all available network interfaces. Modify the default settings with the PORT and/or HOSTNAME settings.

UDP

This collector listens for connections from yaf on a UDP port. The default is to listen on port 18000 on all available network interfaces. Modify the default settings with the PORT and/or HOSTNAME settings.

Using UDP is not recommended as it is not a reliable transport protocol, and thus cannot guarantee delivery of messages. Also note that unless super_mediator is started prior to starting yaf, super_mediator may not have the received the templates necessary to decode the IPFIX messages. super_mediator ignores the messages and logs warnings until yaf periodically retransmits the templates. (Use the --udp-temp-timeout switch on yaf to specify how often templates are retransmitted.)

SINGLE_FILE

This collector reads from a single IPFIX file or from the standard input. The PATH statement is required.

DIRECTORY_POLL

This collector periodically polls a given directory for files. The PATH statement is required. After processing a file, super_mediator either deletes it or moves it to another directory. Either the DELETE or MOVE command is also required.

COLLECTOR END

The COLLECTOR END command ends the definition of a collector.

For a TCP or UDP collector, the following optional commands may be used change the default listening port and interfaces:

PORT PORT

This optional command specifies the network port on which the collector should listen for flow data. If not specified, the default is 18000.

HOSTNAME HOSTNAME

This optional command specifies the IP address or name of the interface on which the collector should listen. The default is to listen on all available interfaces.

The following commands are to configure a collector reading from file(s). The first is required. All are allow for a DIRECTORY_POLL collector, but only the first and last are valid SINGLE_FILE collector.

PATH PATH

For a SINGLE_FILE collector, this command specifies the file to read or, if PATH is -, that the collector reads the standard input. For a DIRECTORY_POLL collector, PATH specifies the directory path to poll for files, and super_mediator attempts to read all files in the directory. (In super_mediator v1.x, PATH was a file glob; that is no longer the case.)

POLL POLL-TIME

The POLL-TIME specifies how often, in seconds, a DIRECTORY_POLL collector checks PATH for new files. If not specified, the default is 30 seconds. This command is only valid for a DIRECTORY_POLL collector.

DELETE

This command tells a DIRECTORY_POLL collector to delete a file after it has been processed (either successfully or unsuccessfully). To avoid deleting the files, use MOVE. One of DELETE or MOVE is required.

MOVE DIRECTORY-PATH

This command provides the directory in which a DIRECTORY_POLL collector moves files after they are processed. Either MOVE or DELETE must be specified for a collector that is polling a directory.

LOCK

NOTE: This is currently unimplemented.

When this optional command is given, super_mediator does not read files that end with .lock. This keyword is only valid if the collector is polling a directory.

This may be useful to coordinate reading a directory where yaf is creating files. Using this command prevents super_mediator from reading and deleting or moving a file that yaf is still writing. Note this is different from how super_mediator locks export files; for that, see LOCK under in the "EXPORTER Block, Quick Reference" section.

DECOMPRESS_DIRECTORY DIRECTORY-PATH

When an input file is compressed with gzip and super_mediator has been compiled with zlib support, this command specifies the directory in which a temporary file is created to hold the decompressed content during processing. If this command is not specified, super_mediator uses the value in the TMPDIR environment variable if set, otherwise it uses /tmp. This applies to both SINGLE_FILE and DIRECTORY_POLL collectors.

The following commands may be use to limit the records that a collector passes to the exporters. They are valid for all types of collectors. See also the FILTER block.

IE-NAME COMPARISON-OPERATOR COMPARE-VALUE

Each collector may contain one or more filtering comparison statements to limit what data should be collected. The syntax of these statements is detailed below in the "Filter Block" section; a summary is that the record's value for element IE-NAME is compared to COMPARE-VALUE using COMPARISON-OPERATOR. If the block contains AND_FILTER, this comparison and every other comparison in the block must be true for the record to be passed to the exporter(s). If the block does not contain AND_FILTER, any comparison being true causes the record to be passed to the exporter(s).

AND_FILTER

If present, a record is passed to the exporter(s) only when all the filtering comparison statements in the block return true. In effect, it joins the comparisons with a logical AND.

Filter Block

(See also "FILTER Block, Quick Reference".)

A filter block is used to ignore certain flow records on collection across all collectors. That is, it limits which record are passed to the exporter(s). Use of a filter block is optional. Only one filter block may be specified in the configuration file.

Instead of using a filter block, filtering statements may be added to any collector and any exporter to limit what a particular collector passes or what a particular exporter processes. No matter where they appear, all filtering statements have the same syntax, described later in this section.

When filter statements appear in multiple locations, they are processed in the following order. A record that passes one level is handed to the next. An AND_FILTER statement only applies to the filtering statements in the containing block.

  1. Filtering statements in the COLLECTOR block reading the records are processed first.

  2. The statements in the FILTER block are processed. Records that pass this level are sent to all exporters.

  3. Finally, filtering statements in the EXPORTER block are checked. Records ignored by one exporter may be processed by a different exporter.

Filters are composed of one or more filtering comparison statements, with each comparison appearing on a line by itself. If the filter block does not contain the AND_FILTER command, the record is passes the filter if any of the comparisons return true, meaning all comparisons must be false for the record to be ignored. When AND_FILTER is present, the record passes the filter only when all comparison statements return true; thus any false comparison causes the record to be ignored.

FILTER Block Commands

FILTER

A filter block starts with the FILTER keyword on a single line, and it continues to the FILTER END command. There is no argument to the FILTER command.

FILTER END

The FILTER END command ends the definition of the filter block and resets the context to top level.

AND_FILTER

If present, a record must match all comparisons in the filter block for it to be forwarded to the exporter(s), and any false comparison causes the record to be ignored. That is, the comparisons are joined by a logical AND. If this command is not present, the comparisons are joined by a logical OR.

IE-NAME COMPARISON-OPERATOR COMPARE-VALUE

Each comparison contains three parts: an IE-NAME, a COMPARISON-OPERATOR, and either a compare value or a comma-separated list of compare values surrounded by square brackets.

A comparison is considered true for a record if the expression created by replacing IE-NAME with the field's value is true. A field that is not present on a record has a value equivalent to setting all bits to 0 (e.g., 0 for numbers, a zero-length string or octetArray, 0.0.0.0 for IPv4 addresses, :: for IPv6 addresses, and the UNIX epoch [1970-01-01T00:00:00 UTC] for times).

The IE-NAME may be any known IPFIX information element, e.g., silkAppLabel, sourceTransportPort, flowStartMilliseconds. In addition, the special keyword collector is allowed (except within a COLLECTOR block), and its value is the name of the COLLECTOR which read the record.

The COMPARE-VALUE may be:

  • an unsigned 63-bit integer expressed in either decimal or hexadecimal; e.g., 34, 0xffff

  • a non-negative floating point value with an optional exponent; e.g., 3.14159, .5, 8., 1.E6, 1.0e-6

  • an IPv4 address an IPv6 address in the canonical format; e.g., 10.1.2.3, ::1, 2011::db8::, ::ffff:10.1.2.3

  • a double-quoted string; e.g., "a string". Unquoted strings are not allowed as the compare value.

  • a comma separated list of any of the above as long as all are the same type, with the list surrounded by square brackets; e.g., [ 3, 4, 5 ]. The list may not be empty.

The supported COMPARISON-OPERATORs are:

==

Succeeds when the value from the record is equal to the compare value. May be used with any type of COMPARE-VALUE except a list of values.

!=

Succeeds when the value from the record is not equal to the compare value. May be used with any type of COMPARE-VALUE except a list of values.

<

Succeeds when the value from the record is strictly less than the compare value. May be used only only when the COMPARE-VALUE is an integer or a floating point value.

<=

Succeeds when the value from the record is less than or equal to the compare value. May be used only only when the COMPARE-VALUE is an integer or a floating point value.

>

Succeeds when the value from the record is strictly greater than the compare value. May be used only only when the COMPARE-VALUE is an integer or a floating point value.

>=

Succeeds when the value from the record is greater than or equal to the compare value. May be used only only when the COMPARE-VALUE is an integer or a floating point value.

IN_LIST

Succeeds when the value from the record is equal to any value in the list. May only be used when the COMPARE-VALUE is a list; the list may hold any type of value.

NOT_IN_LIST

Succeeds when the value from the record does not match any value in the list. May only be used when the COMPARE-VALUE is a list; the list may hold any type of value.

The data type of the IE-NAME information element determines the expected type for COMPARE-VALUE, as shown in this table. The third column shows whether the comparisons other than == and != are allowed, and the fourth whether a list of values may be used with the IN_LIST and NOT_IN_LIST operators.

IE-NAME Type       COMPARE-VALUE Type  <,> Allowed  List Allowed
=================  ==================  ===========  ============
 collector          string              no           no
 unsigned integer   integer             yes          yes
 signed integer     integer             yes          yes
 boolean            integer             yes?         yes
 date-time          integer             yes          yes
 float              float               yes          yes
 string             string              no           yes
 octetArray         string              no           yes
 macAddress         string, len == 6    no           yes
 IPv4               ip                  no           yes
 IPv6               ip                  no           yes
 list               N/A                 N/A          N/A

When the data type of IE-NAME is a macAddress, the string must contain exactly six characters. If the type of IE-NAME is a list (e.g., basicList), super_mediator prints an error and exits.

If the COMPARE-VALUE is not valid for the IE-NAME, super_mediator prints an error and exits.

Filter Example: Filter on DNS Traffic:

FILTER
  silkAppLabel == 53
FILTER END

Filter Example: Filter on Ports 80 OR 53:

FILTER
  sourceTransportPort IN_LIST [ 53, 80 ]
  destinationTransportPort IN_LIST [ 53, 80 ]
FILTER END

Filter Example: Filter for IPv6 Address(es) in IPset:

FILTER
  sourceIPv6Address IN_LIST "/data/sets/mysamplev6.set"
FILTER END

Filter Example: Filter for DNS labeled traffic not on port 53:

FILTER
  silkAppLabel == 53
  destinationTransportPort != 53
  AND_FILTER
FILTER END

Defining an Output: The EXPORTER Block

(See also "EXPORTER Block, Quick Reference".)

Taken from the intro:

super_mediator will rotate output files every so many seconds. A TEXT exporter can be configured to import CSV files to a MySQL database if MySQL libraries are available. Exporters may have filters associated with them. When super_mediator loses connection with one of the configured exporters, it will report an initial warning message to the log and retry the connection immediately. If the retry is unsuccessful, it will retry the connection every 15 seconds until successful. To verify the connection attempts, use the DEBUG loglevel. Flows will be lost while the connection is down.

An exporter block defines an output from super_mediator. Like a collector, the exporter has a type that indicates whether the data is going to a single file, a directory of files that are periodically rotated (closed and reopened), or a network socket. An exporter also has a format that specifies the type of data being written: binary IPFIX, text records in the JSON format, or delimited text similar to CSV (comma-separated value). A name may be given to the exporter; the name is used in other blocks in the configuration file that further describe the output and in log messages. super_mediator generates a name for the exporter if no name is given.

For a JSON exporter, the standard root element for a flow is "flows". The root elements will be different for DNS_DEDUP, SSL_DEDUP, SSL CERTIFICATE, DEDUP, or DNS_RR records.

The contents of the exporter block specify where to write the data and enable various features, such as deduplication.

The exporter block may contain filtering comparisons that limit which records are processed by the exporter. These statements have the same syntax as those in the COLLECTOR and FILTER blocks.

The configuration file must contain at least one exporter, and it may contain multiple exporters.

EXPORTER Block Commands

In the top-level context, the EXPORTER command begins an exporter definition. All other commands in this section are accepted in the context of the exporter block. The EXPORTER END command closes the exporter definition and sets the context to top-level.

EXPORTER EXPORTER-FORMAT EXPORTER-TYPE *EXPORTER-NAME*

The EXPORTER command begins a new exporter definition. Its arguments are the format of the output, the type of exporter to be defined, and an optional EXPORTER-NAME. If EXPORTER-NAME is not provided, super_mediator generates names E1, E2, E3, et cetera.

The EXPORTER-FORMAT specifies the format of the output, and must be one of the following:

IPFIX

The output is binary IPFIX files or IPFIX written to a socket.

JSON

The output is text in the JSON format; the output may not be written to a socket.

TEXT

The output is text, where a line generally represents a record and a delimiter separates the record's values. The output may not be written to a socket.

The EXPORTER-TYPE must be one of the following:

TCP

The exporter sends IPFIX via TCP to the hostname or IP address specified in the HOSTNAME command. By default, data is written to port 18001, but that may be changed with the PORT command.

UDP

The exporter sends IPFIX via UDP to the hostname or IP address specified in the HOSTNAME command. By default, data is written to port 18001, but that may be changed with the PORT command. Using UDP is not recommended, as it is not a reliable transport protocol, and thus cannot guarantee delivery of messages.

SINGLE_FILE

The exporter writes IPFIX, JSON, or TEXT to a file or to the standard output. The PATH statement is required.

ROTATING_FILES

The exporter writes IPFIX, JSON, or TEXT to a file; after a period of time that file is closed and a new file is opened. The PATH and ROTATE_INTERVAL statements are required. The argument to PATH must include a filename-prefix; super_mediator appends a timestamp and the file extension .med when opening the file.

EXPORTER END

The EXPORTER END command ends the definition of an exporter. Following this command, top-level commands are accepted.

PORT PORT

This optional command specifies the port on which the exporter writes IPFIX when the EXPORTER-TYPE is TCP or UDP. If not specified, the default is 18001.

HOSTNAME HOSTNAME

The command must be given when the EXPORTER-TYPE is TCP or UDP to specify the hostname or IP address the exporter contacts for writing IPFIX.

PATH PATH

For a SINGLE_FILE exporter, specifies the path of the file to write to or may be - to write to the standard output. For a ROTATING_FILES exporter, specifies the path and filename-prefix of the output file; a date-time and the suffix .med will be appended to the filename-prefix. When MULTI_FILES is present in the exporter or a DEDUP_CONFIG block is associated with the exporter, PATH must be a directory even if the EXPORTER-TYPE is SINGLE_FILE. In all cases, PATH may be relative the current directory or a complete path. When super_mediator is invoked with the --daemon switch, PATH should be a complete path.

ROTATE_INTERVAL ROTATE-SECONDS

Specifies the interval, in seconds, for rotating the output file(s) when the EXPORTER-TYPE is ROTATING_FILES.

If super_mediator is not receiving any flow data, files will not be rotated. super_mediator uses flow end time in the incoming flow records to determine the current time and when to rotate files. Text filenames use the flow end time when rotating files for indexing purposes. If MULTI_FILES is present, by default super_mediator will rotate files using a serial number, not a timestamp in the filename. If TIMESTAMP_FILES is present, it will timestamp the files instead of using the serial number. IPFIX Exporters use system time when rotating files.

LOCK

Causes the exporter to prepend a dot (.) to filenames that it is currently writing to. This may be used with SiLK's rwsender(8) tool to prevent it from removing files that super_mediator is writing. When super_mediator closes the file it removes the leading dot from the filename. May not be used for network-based exporters.

MOVE FILE-DIRECTORY

Moves the output file(s) created by this exporter to FILE-DIRECTORY after it has finished writing them. FILE-DIRECTORY must be a valid file directory. May not be used for network-based exporters.

GZIP_FILES

Causes the exporter to invoke gzip(1) on the output files(s) it creates after it has finished writing them. May not be used for network-based exporters.

REMOVE_EMPTY_FILES

If present for TEXT or SINGLE_FILE Exporters, super_mediator will remove output files that have a file size of 0. This is enabled by default.

DISABLE METADATA_EXPORT

Causes this exporter not to include information element and template metadata in the IPFIX output. This command is only valid for IPFIX exporters.

UDP_TEMPLATE_TIMEOUT TIMEOUT-MIN

FIXME: WHAT ARE THE UNITS HERE????

If specified, send templates out 3 times in TIMEOUT-MINS. By default, as per the recommendations in RFC 5101, super_mediator will retransmit templates three times within 10 minutes.

The following commands may be used to limit which records are processed by an exporter.

DPI_ONLY

Causes the exporter to export only those flow records that (potentially) have deep packet inspection data. Specifically, it ignores flow records where the value of the silkAppLabel element is 0.

If specified, only export flows that have some Deep Packet Inspection data associated with it. If the exporter has EXPORTER-TYPE of TEXT, super_mediator will write a flow index line and associated DPI data to the output file. (See above TEXT for format and examples). super_mediator will not write stats messages. For advanced configuration of the DPI to CSV export use the DPI_CONFIG block.

FLOW_ONLY

Tells the exporter to strip DPI data from the flow records and export only the basic flow record. This also prevents the export of YAF statistics records

If specified, only forward basic flow information to the exporter. This should be used with SiLK collectors, such as rwflowpack or flowcap, as they do not collect Deep Packet Inspection data. If present, super_mediator will not forward or write stats messages.

NO_FLOW

If present, do not forward basic flow records to this EXPORTER. This is typically used if other deduplication features are enabled. Flow export is enabled by default unless SSL_DEDUP_ONLY, DEDUP_ONLY, STATS_ONLY, DNS_DEDUP_ONLY, DNS_RR_ONLY or NO_FLOW is present.

DNS_RESPONSE_ONLY

If present, only export DNS responses. This will ignore all DNS queries. This option is ignored if no DNS DPI data is present in the flow.

NO_STATS

If preset, the exporter will not write or forward stats messages from yaf.

STATS_ONLY

If present, the exporter will write or forward only stats messages from yaf. If this is present after the DPI_ONLY or FLOW_ONLY keywords, it will turn on stats messages in the exporter. The stats messages will not be prefaced "stats|". Instead they will be prefaced with "\N|" so that when they are imported into a database, the timestamp field will be updated to the current time.

IE-NAME COMPARISON-OPERATOR COMPARE-VALUE

Each exporter may contain one or more filtering comparison statements to limit what records that exporter processes. The syntax of these statements is detailed in the "Filter Block" section; a summary is that the record's value for element IE-NAME is compared to COMPARE-VALUE using COMPARISON-OPERATOR. If the block contains AND_FILTER, this comparison and every other comparison in the block must be true for the record to be processed by the exporter. If the block does not contain AND_FILTER, any comparison being true causes the record to be processed by the exporter.

AND_FILTER

If present, a record is processed by the exporter only when all the filtering comparison statements in the block return true. In effect, it joins the comparisons with a logical AND.

The following commands apply only to TEXT exporters:

DELIMITER DELIMITER

Tells the exporter to print DELIMITER between field values in the TEXT output. DELIMITER must be a single character. The default is |. See also DPI_DELIMITER.

DPI_DELIMITER DPI-DELIM

Tells the exporter to print DPI-DELIM between DPI field values in the TEXT output. DPI-DELIM must be a single character. If not specified, this delimiter is the same as the value of DELIMITER, whose default is |. Using a different delimiter than above potentially causes two different delimiters to be used on the same line, which may be useful when uploading text files to a database with the desire to use one column for DPI fields.

ESCAPE_CHARS

If present, super_mediator will escape any control characters by placing a \ (backslash) in front of the character. Octal codes will be used for control characters. Backslash characters that are present in the string will also be escaped by using a double backslash. super_mediator will also escape the delimiter character that is used for the EXPORTER, "|" by default. This option is only available for TEXT EXPORTERS.

Causes the first line of every output file to be a header giving the field names that are present in the file; the names are separated by the DELIMITER.

MULTI_FILES

Only valid if DPI_ONLY is also present. Only valid for TEXT Exporters. If present, super_mediator will separate DPI data based on application protocol into separate files in the file directory given to PATH, which must exist prior to starting super_mediator. For advanced configuration of the filenames, edit the DPI_CONFIG block. This is useful if the mysqlimport(1) tool will be loading the CSV output from super_mediator. The mysqlimport tool loads tables from text files. The base name of the text file must be the name of the table that should be used. For a list of the default table names and information elements they contain, see below. If MULTI_FILES is present, the CSV does not contain the table name. The EXPORTER will write flow index lines in the following form to a separate file flow.txt0:

flow_key_hash | start_time_ms | srcip | dstip | \
protocol | srcport | dstport | vlan | obid

DPI data (all protocols except DNS, SSL, and DNP) will be written in the following form:

flow_key_hash | start_time_ms | ob-id | elem_id | data

If DEDUP_PER_FLOW is also present, the format will be:

flow_key_hash | start_time_ms | ob-id | elem_id | count | data

Exceptions:

DNS will be written in the following form (different from DNS_DEDUP form. DNS_DEDUP is not permitted if MULTI_FILES is present). See above under TEXT for a description of each field:

flow_key_hash | start_time_ms | obid | [Q|R] | dnsId | \
section | nxdomain | authoritative | response_type | \
ttl | name | value

SSL/TLS will be written in the following form (See above under TEXT for a description of each field):

elem_id | flow_key_hash | start_time_ms | obid | [I|S|E] | \
cert_no_seq | data

DNP3.0 will be written in the following form:

elem_id | flow_key_hash | start_time_ms | obid | \
dnpsrcaddress | dnpdstaddress | dnpfunction | dnpdata

RTP will be written in the following form:

elem_id | flow_key_hash | start_time_ms | obid |
payloadType | reversePayloadType

Flow-stats will be written as described above, except the line will not include the table name [flowstats].

Example EXPORTER MULTI_FILES Configuration:

EXPORTER TEXT
    PATH "/data/dpi"
    DPI_ONLY
    MULTI_FILES
    ROTATE 600
    LOCK
EXPORTER END

Example Data in /data/dpi/flow.txt0:

109074684|1207197775606|10.10.1.102|10.10.34.130|6|2898|7000|0|0

Example Data in /data/dpi/irc.txt0:

109074684|1207197775606|0|125|NICK OmegaT
109074684|1207197775606|0|125|USERHOST OmegaT
109074684|1207197775606|0|125|VERSION mIRC v6

In the above example, the three lines were written to irc.txt0 because information element id 125 has the default label irc. The default labels are listed below. They can be modified in the DPI_CONFIG block. The files will rotate if ROTATE is present in the EXPORTER Block. The files will "lock" if LOCK is present in the EXPORTER Block. Enclosed with the super_mediator distribution is super_table_creator, a program that creates the default MySQL tables for default super_mediator CSV output. The following is an example of using the super_table_creator and mysqlimport tool with super_mediator output:

super_table_creator -n username -p password -d super_db

mysqlimport -u user -p --fields-terminated-by="|" super_db irc.txt0

Alternatively, you can use the MySQL LOAD DATA INFILE command to load the first three columns from a file into the "super" database:

mysql -u user -p -e "LOAD DATA INFILE '/data/dpi/flow.txt0' into table super_flows FIELDS TERMINATED by '|' (column1, column2, column3);" super

super_mediator can also be configured to do the importing if the MySQL client libraries are installed and if the MySQL credentials are listed in the EXPORTER block. The EXPORTER block must have MYSQL_USER, MYSQL_PASSWORD, and MYSQL_DATABASE all present for super_mediator to attempt the import. super_mediator uses the default database schemas used by the super_table_creator. super_table_creator contains schemas for all DPI protocols, DNS-deduplication, and flow.

If NO_INDEX is also present in the EXPORTER BLOCK, super_mediator will not write to a flow index file. For each line in the DPI files, the flow key hash will be expanded into its components and the timestamp will be printed in human-readable format (text wrapped for readability):

start-time | srcip | dstip | protocol | srcport | \
dstport | vlan | obid | elem_id | data

Example:

2008-04-10 04:41:36.583|1.2.3.4|5.6.7.8|6|1360|80|0|0|114|HTTP/1.1

The follow example shows an example configuration file and sample data using the NO_INDEX and TIMESTAMP_FILES options:

Example NO_INDEX Exporter Configuration:

EXPORTER TEXT
    PATH "/data/dpi"
    DPI_ONLY
    MULTI_FILES
    ROTATE 600
    LOCK
    TIMESTAMP_FILES
    NO_INDEX
EXPORTER END

Example Data in /data/dpi/http.txt20080410044142:

2008-04-10 04:41:36.582|1.2.3.4|5.6.7.8|6|1359|80|0|0|111|Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
2008-04-10 04:41:36.582|1.2.3.4|5.6.7.8|6|1359|80|0|0|112|/w/index.php?
2008-04-10 04:41:36.582|1.2.3.4|5.6.7.8|6|1359|80|0|0|114|HTTP/1.1

Example Data in /data/dpi/dns.txt20110128215727:

2011-01-28 21:52:23.473|1.2.3.4|6.7.8.9|17|49664|53|905|0|Q|525|0|0|0|1|0|www.dropbox.com
TIMESTAMP_FILES

By default, super_mediator includes the timestamp (flow end time) of the first flow in the filename of the TEXT file, except if MULTI_FILES is present. If, TIMESTAMP_FILES is present, super_mediator will include the timestamp in the DPI files after the file extension (e.g. http.txt20120606123430). To search for a particular flow, use the flow's end time to determine which file contains the flow.

FIELDS [ FIELD-LIST ]

Causes a TEXT or JSON exporter to write only the fields (information elements) specified in FIELD-LIST, where the list contains the names of the elements. Any information element name known to super_mediator may be included in FIELD-LIST, each name must be a string (surrounded by double quotes (")), and the comparison is case-sensitive. The elements are exported in the order they appear in FIELD-LIST. A name may be repeated.

Specifying FIELDS disables the printing of DPI elements unless "DPI" is included in FIELD-LIST. By default, the printing of YAF process statistics is disabled when FIELDS is specified. Use the STATS_ONLY command in the block to re-enable them.

In addition to any information element name, the following special values are supported (they must also be specified as strings):

SIP_ANY

Either sourceIPv4Address or sourceIPv6Address.

DIP_ANY

Either destinationIPv4Address or destinationIPv6Address.

COLLECTOR

The name of the collector that read the flow record.

DPI

Deep Packet Inspection information. For all protocols, except DNS and TLS/SSL, adding DPI to the field list will add the information element id and data value to the end of the line regardless of what order DPI is in the list. DPI information will always be at the end of the line. For each DPI field captured by YAF, there will be one line in the output text file. To configure the DPI fields super_mediator exports, use the DPI_FIELD_LIST or the DPI_CONFIG block options. If both DPI_FIELD_LIST and DPI_CONFIG are present in the configuration file, the elements listed in the DPI_FIELD_LIST will take priority for that particular exporter. All protocols except DNS and SSL will add the following columns to the end of the line:

elem_id | data

For DNS, the following fields will be added to the end of the line:

QR | dnsId | section | nxdomain | authoritative | \
response_type | ttl | name | value

For SSL, the following fields will be added to the end of the line:

elem_id | IS | cert seq no. | data

See above (under TEXT) for explanations of the fields.

DPI_FIELD_LIST [ DPI-IE-LIST ]

If present for TEXT exporters, super_mediator will only export DPI information elements contained in DPI-IE-LIST. The DPI-IE-LIST is a list of information element ids from the below list (see DPI_CONFIG), separated by a comma, and surrounded by square brackets, [ and ]. For example, the following line will direct super_mediator to only export HTTP user agent strings and get requests.

DPI_FIELD_LIST [111, 112]
NO_INDEX

If present for TEXT Exporters, super_mediator will not write separate lines with flow information. It will include the flow information in the following form at the beginning of each DPI data line. The start time will be in human-readable format. The flow key hash will not be printed. This will make the DPI files substantially larger in size. See the above sample configuration and example (under MULTI_FILES).

start-time | srcip | dstip | protocol | srcport | \
dstport | vlan | obid
NO_FLOW_STATS

If present for TEXT Exporters, super_mediator will not write yaf flow-stats. Only valid for TEXT exporters. Ignored for any other exporter type. See yaf Flow Statistics Template for more information.

NO_FLOW

If present, do not forward basic flow records to this EXPORTER. This is typically used if other deduplication features are enabled. Flow export is enabled by default unless SSL_DEDUP_ONLY, DEDUP_ONLY, STATS_ONLY, DNS_DEDUP_ONLY, DNS_RR_ONLY or NO_FLOW is present.

FLOW_ONLY

If specified, only forward basic flow information to the exporter. This should be used with SiLK collectors, such as rwflowpack or flowcap, as they do not collect Deep Packet Inspection data. If present, super_mediator will not forward or write stats messages.

DPI_ONLY

If specified, only export flows that have some Deep Packet Inspection data associated with it. If the exporter has EXPORTER-TYPE of TEXT, super_mediator will write a flow index line and associated DPI data to the output file. (See above TEXT for format and examples). super_mediator will not write stats messages. For advanced configuration of the DPI to CSV export use the DPI_CONFIG block.

DNS_RESPONSE_ONLY

If present, only export DNS responses. This will ignore all DNS queries. This option is ignored if no DNS DPI data is present in the flow.

NO_STATS

If preset, the exporter will not write or forward stats messages from yaf.

STATS_ONLY

If present, the exporter will write or forward only stats messages from yaf. If this is present after the DPI_ONLY or FLOW_ONLY keywords, it will turn on stats messages in the exporter. The stats messages will not be prefaced "stats|". Instead they will be prefaced with "\N|" so that when they are imported into a database, the timestamp field will be updated to the current time.

The following commands cause the exporter to create new types of records that are written to the output stream(s). Most commands have two forms, one creates the new records and writes them along with the normal flow records (e.g., SSL_DEDUP), and another that creates the records and writes only those new records (SSL_DEDUP_ONLY). Multiple new record types may be enabled within a single exporter as long as none of the _ONLY variants are used.

When super_mediator is processing IPFIX records from a previous invocation of super_mediator, by default it reads and writes these generated records as it does any other flow record. If the _ONLY variant of one of these commands is given, super_mediator ignores all incoming generated record types except those specified by that command. For example, an exporter that specifies DNS_DEDUP_ONLY generates new DNS dedup records from the incoming flow records and passes through any DNS dedup records that appear in the input.

DNS_DEDUP

Enables DNS deduplication (see "DNS Deduplication: DNS_DEDUP") of yaf IPFIX flow records with DNS DPI data and writes the aggregated DNS records to the output. DNS_DEDUP may not be specified when MULTI_FILES is present. It is recommended to separate the DNS_DEDUP and MULTI_FILES into separate exporters. See super_mediator(1) for a complete description of the information elements exported for IPFIX Exporters.

DNS_DEDUP_ONLY

Enables DNS deduplication as in the previous command but causes the exporter to emit only the DNS deduplicated records.

DNS_RR *FULL*

Enables DNS resource record export ("DNS Resource Record Export") on this exporter, which must be an IPFIX exporter. Every DNS resource record is exported; they are not deduplicated. For the format of the exported records, see "DNS Resource Record Export". If FULL is specified, the exported records include the 5-tuple of the containing flow record.

DNS_RR_ONLY *FULL*

Similar to the previous command, but causes the exporter to emit only DNS resource records. The exporter does not write any flow or other deep packet inspection data that it processes.

SSL_DEDUP

Enables TLS/SSL deduplication of yaf IPFIX flow records with TLS DPI data and writes the aggregated records to the output.

SSL_DEDUP_ONLY

Enables TLS/SSL deduplication as in the previous command and causes the exporter to emit only the TLS deduplicated records.

DEDUP_ONLY

Causes the exporter to emit only records created by general deduplication as configured by a DEDUP_CONFIG block. There is no command within the exporter block to enable general deduplication.

DEDUP_PER_FLOW

If present, super_mediator will deduplicate DPI fields within a flow. Often, yaf will export multiple values for an information element that are the same. With this option, super_mediator will only export unique values for an information element along with a count of how many times the value was present with the flow. This only affects certain protocols such as HTTP, FTP, IMAP, RTSP, SIP, SMTP, SSH, IRC, POP3, MODBUS, ENIP, SLP. The hit count for each value will be written in the column before the value. For JSON exporters, DPI values will be deduplicated, but super_mediator will not export a hit count.

EXPORTER TEXT
    PATH "/data/flow.txt"
    DEDUP_PER_FLOW
EXPORTER END

Example DPI Output with above config (third column is hit count):

http|115|1|http://en.wikipedia.org/wiki/Http
http|114|2|HTTP/1.0
http|117|1|en.wikipedia.org
DEDUP_ONLY

If present, super_mediator will only write deduplicated DPI data. See the DEDUP_CONFIG block for more information on configuring data deduplication.

SSL_CERT_HASH_SHA1

If present, super_mediator computes the SHA1 hash of the entire X.509 certificate, if available. yaf does not export the entire certificate by default; enable this in yafDPIRules.conf, and this option is only available if super_mediator was built with OpenSSL support.

SHA1 hashing may also be enabled by listing 298 in the OTHER FIELD-LIST in the SSL_CONFIG block. If using the MULTI_FILES option, add 298 to the TABLE-LIST in the DPI_CONFIG block.

SSL_CERT_HASH_MD5

If present, super_mediator computes the MD5 hash of the entire X.509 certificate, if available. yaf does not export the entire certificate by default; enable this in yafDPIRules.conf, and this option is only available if super_mediator was built with OpenSSL support.

MD5 hashing may also be enabled by listing 299 in the OTHER FIELD-LIST in the SSL_CONFIG block. If using the MULTI_FILES option, add 299 to the TABLE-LIST in the DPI_CONFIG block

The following commands tell the exporter to load its textual output files into a MySQL database as the files are closed. The exporter must be a TEXT exporter, and the commands MYSQL_DATABASE, MYSQL_USER, and MYSQL_PASSWORD must all be present. MYSQL_HOST may be specified if the database is not running on the local machine. The commands are only available when super_mediator has been built with MySQL support; to determine whether it has, specify the --version switch to super_mediator and examine the MySQL support line.

MYSQL_USER USER-NAME

Sets the login name to use when connecting to the MySQL database server.

MYSQL_PASSWORD PASSWORD

Sets the password to use when connecting to the MySQL database server.

MYSQL_DATABASE DATABASE-NAME

Sets the database name to use when connecting to the MySQL database server.

MYSQL_HOST HOSTNAME

Sets the hostname where the the MySQL database server is running. If not specified, super_mediator attempts to connect to a database server running locally.

MYSQL_TABLE TABLE-NAME

If present for TEXT Exporters, super_mediator will import the CSV file(s) to the table TABLE-NAME. This is only used if FLOW_ONLY or DNS_DEDUP_ONLY is also present. The default table name for FLOW_ONLY data is flow. The default table name for the dns-dedup data is dns. In order to set the table names for the DPI files, use the DPI_CONFIG block. Use the super_table_creator to create the database schemas for the DNS deduplication and flow tables.

Example of DPI Import:

EXPORTER TEXT
    PATH "/data/dpi"
    DPI_ONLY
    MULTI_FILES
    ROTATE 600
    LOCK
    TIMESTAMP_FILES
    NO_INDEX
    MYSQL_DATABASE super_db
    MYSQL_USER root
    MYSQL_PASSWORD password
EXPORTER END

Example of DNS_DEDUP Import:

EXPORTER TEXT
    PATH "/data/dns/dns_dedup"
    DNS_DEDUP_ONLY
    ROTATE 600
    LOCK
    MYSQL_DATABASE dns_dedup_db
    MYSQL_USER root
    MYSQL_PASSWORD password
    MYSQL_TABLE dns
EXPORTER END

Example of User-defined table names:

EXPORTER TEXT
    PATH "/data/dpi"
    DPI_ONLY
    MULTI_FILES
    ROTATE 600
    LOCK
    MYSQL_DATABASE my_super_db
    MYSQL_USER bob
    MYSQL_PASSWORD password
EXPORTER END
DPI_CONFIG
    TABLE my_http_table [111, 112, 113, 114, 115]
    TABLE my_dns_table [1, 2, 6, 28]
    TABLE my_dhcp_table [242, 243]
DPI_CONFIG END
REMOVE_UPLOADED

If present and loading the data into the MySQL was successful, super_mediator removes the file it loaded into the database. This only removes files that were successfully loaded. super_mediator does not consider MySQL warnings as unsuccessful. Please be certain that the database is set up correctly, and the data is imported successfully before using this command.

Exporter Textual Output

The exporter will write delimited text to a file or JSON if the keyword JSON is present within the EXPORTER block. Options present within the EXPORTER block will vary the format of the output. If the ROTATE_INTERVAL keyword is present, the exporter will rotate output files every ROTATE-SECONDS seconds. Text flow data files, by default, will be in the form (text wrapped for readability):

start-time | end-time | dur | rtt | protocol | srcip | \
srcport | pkt | oct | attributes | mac | dstip | dstport | \
rpkt | roct | rev-attributes | dstmac | iflags | uflags | \
riflags | ruflags | tcpseq | revtcpseq | ingress| egress |
vlan | app | tos | end-reason | collector | payload | revpayload

start-time and end-time are in the form 2012-01-28 13:12:32.786. Using the included program, super_table_creator(1), which is available if mysql is installed, you can create a MySQL table for the full flow, by running super_table_creator with --flow-only.

Unless FLOW_ONLY is present, a TEXT Exporter will write every IPFIX field it decodes into the file given to PATH, as well as yaf process statistics records. Statistics records will be labeled with the word stats and are in the following form (text wrapped for readability):

stats | total_flows | total_packets | dropped_packets | \
ignored_packets | expired_fragments | assembled_fragments |\
flush_events | flow_table_peak_count | sensor_IP | \
process_id | mean_flow_rate | mean_pkt_rate

Example:

stats|2|56|0|0|0|0|2|1|127.0.0.1|0|131|3685

If DPI data exists with the flow, super_mediator will write one line of flow data in the above format followed by a flow index line and one or more lines containing the DPI meta data.

Example EXPORTER TEXT configuration:

EXPORTER TEXT
   PATH "/data/flow.txt"
EXPORTER END

Example output with above configuration:

2012-04-03 04:42:55.606|2012-04-03 04:45:13.738| 138.132|   0.088|  6| 10.10.1.102| 2592| 30| 1591|00|00:00:00:00:00| 10.10.34.130| 7000| 24| 8001|00|00:00:00:00:00|  S| APRF|  AS| AP|b3332bea|ead9dce8|000| 194|000|000|
irc|125|NICK nickname
irc|125|USERHOST Omega
irc|125|VERSION mIRC v6
irc|125|NAMESX
stats|2|56|0|0|0|0|2|1|10.20.11.51|0|131|3685

If DPI_ONLY is present, the line of flow data will condensed to the following format and labeled with the word flow by default (text wrapped for readability):

flow | flow_key_hash | start-time-ms | srcip | dstip | \
protocol | srcport | dstport | vlan | obid

Example:

flow|109074684|1207197775606|10.10.1.102|10.10.34.130|6|2592|7000|0|0
flow_key_hash is a 32-bit hash of the 5-tuple + vlan.
start-time-ms is the milliseconds since Epoch time.

For all protocols except DNS, SSL/TLS, and DNP3 the output follows the following format. Each line contains the default table name, information element id, flow key hash, flow start time in milliseconds, observation domain id, and the data. There will be one line for each data field associated with the flow (text wrapped for readability):.

table_name | element_id | data

If DEDUP_PER_FLOW is present for a TEXT EXPORTER, the format for each DPI line will be (see below for a description of DEDUP_PER_FLOW):

table_name | element_id | hitcount | data

If DPI_ONLY is present for a TEXT EXPORTER, any DPI data associated with the flow will have the following form (note that DPI_ONLY adds the flow_key_hash, start_time_ms, and observation domain id to the DPI data):

table-name | flow_key_hash | start_time_ms | obid | elem_id | data

If DPI_ONLY and DEDUP_PER_FLOW are both present, the format will have the following form:

table-name | flow_key_hash | start_time_ms | obid | elem_id | hitcount | data

Example EXPORTER TEXT config with DPI_ONLY:

EXPORTER TEXT
  PATH "/data/flow.txt"
  DPI_ONLY
EXPORTER END

Example Output with above config:

http|1441601726|1207802496583|115|http://en.wikipedia.org/wiki/Http
http|1441601726|1207802496583|114|HTTP/1.0
http|1441601726|1207802496583|114|HTTP/1.0
http|1441601726|1207802496583|117|en.wikipedia.org
flow|1441601726|1207802496583|128.237.224.172|208.80.152.2|6|1360|80|0

Example EXPORTER TEXT config with DPI_ONLY and DEDUP_PER_FLOW:

EXPORTER TEXT
    PATH "/data/flow.txt"
    DPI_ONLY
    DEDUP_PER_FLOW
EXPORTER END

Example Output with above config:

http|1441601726|1207802496583|115|1|http://en.wikipedia.org/wiki/Http
http|1441601726|1207802496583|114|2|HTTP/1.0
http|1441601726|1207802496583|117|1|en.wikipedia.org
flow|1441601726|1207802496583|128.237.224.172|208.80.152.2|6|1360|80|0

See below for a list of information element ids and the default table names. See yafdpi(1) for descriptions of each of the information elements. The default flow index and table names can be configured in the DPI_CONFIG block. The DPI_CONFIG block will also configure super_mediator to write only particular information elements.

The format of the DNS CSV output is as follows (Note: This is different from the deduplicated DNS output):

table_name | QR | dnsId | section | nxdomain | authoritative | \
response_type | ttl | name | value
QR denotes if the record is a Query(Q) or a Response(R).
The dnsId is the transaction ID from the DNS record.
section is the section of the packet the resource record was extracted from (0-Query,1-Answer,2-Name Server,3-Additional).
nxdomain denotes if the record was an NXDomain(1) or not(0).
authoritative denotes if the response is from an authoritative name server (1) or not (0).
The response_type is the TYPE field of the DNS resource record.
ttl is the time to live from the resource record.
name is the Query or Response Name.
value is the RDATA field from the resource record.

DNS Example with DPI_ONLY:

flow|114422227|1207802496560|128.237.224.172|128.2.1.10|17|1599|53|0
dns|114422227|1207802496560|0|Q|14728|0|0|0|1|0|meta.wikimedia.org.

yaf version 2.3.0 changed the format of X.509 Certificate export. If using version 2.3.0 or later, the format of TLS/SSL CSV will be as follows:

table_name | elem_id | [I|S|E] | cert seq no. | data

Note that SSL Certificate Extension fields are only exported if specifically set in the SSL_CONFIG block.

elem_id

is the object identifier as given in the X.509 ASN.1 RelativeDistinguishedName sequence. A list of common identifiers are listed below with the element ID numbers.

ISE

denotes if the data came from an Issuer Field(I), Subject Field(S), or Extension Field (E). For fields that are not associated with the issuer, subject, or extension but describe other characteristics of the certificate, an I will be used (for example, not-before or not-after timestamps).

cert seq no

signifies which certificate the data came from in the certificate chain. Usually, this field will contain a 0, 1, or 2.

DNP3.0 also has a different format. DNP3.0 will be written in the following form:

table_name | elem_id [284] | dnp src addr | dnp dst addr | \
dnp function | dnp data
dnp elem_id will always be 284.
dnp src addr is the source address found in the packet payload.
dnp dst addr is the destination address found in the packet payload.
dnp function is the function code describing the function of the following dnp data.
dnp data is the bytes captured by the regular expression executed by yaf written in hexadecimal.

modbus and ethernet/IP data will also be written in hexadecimal.

As of yaf version 2.3.0, yaf can export enhanced flow metrics when running yaf with --flow-stats. By default, super_mediator will print the flow-stats to the TEXT file given to PATH. flow-stats will be written in the following form (text wrapped for readability) and will directly follow the flow they refer to:

flowstats | tcpUrgTotalCount | smallPacketCount | nonEmptyPacketCount | \
dataByteCount | averageInterarrivalTime | \
firstNonEmptyPacketSize | largePacketCount | maxPacketSize |\
firstEightNonEmptyPacketDirections | \
standardDeviationPayloadLength | \
standardDeviationInterarrivalTime | \
averagePacketSize | reverseTcpUrgTotalCount | \
reverseSmallPacketCount | reverseNonEmptyPacketCount | \
reverseDataByteCount | reverseAverageInterarrivalTime | \
reverseFirstNonEmptyPacketSize | reverseLargePacketCount | \
reverseMaxPacketSize | reverseStandardDeviationPayloadLength |\
reverseStandardDeviationInterarrivalTime | reverseAveragePayloadLength

For descriptions of these information elements, see the yaf man page.

FlowStats Example:

flowstats|0|1|1|49|0|49|0|49|00|0|0|49|0|0|0|0|0|0|0|0|0|0|0|0

For a more custom TEXT output, use the FIELDS keyword. See the documentation below.

DNS Resource Record Export

An IPFIX exporter may have DNS resource record export enabled, which tells the exporter to emit a "DNS RR" record for every DNS-containing flow record it sees; the records are not deduplicated.

Use the exporter's DNS_RR or DNS_RR_ONLY command, with an optional FULL argument, to enable DNS resource record export. Unlike most other exporter features, DNS resource record export does not have a separate customization block.

The DNS RR records contain DNS-specific fields, the starting time, and yafFlowKeyHash. If the FULL argument is specified, the records also contain the flow record 5-tuple (The IP address and port of the source and destination and the IP protocol). The fields are listed here, in order. The exporter only emits the elements labeled FULL when the FULL argument is specified.

DNS_RR IPFIX Record Format

flowStartMilliseconds IE 152, 8 octets, unsigned

Flow start time in milliseconds since 1970-01-01 00:00:00 UTC.

sourceIPv6Address IE 27, 16 octets, unsigned, FULL

IPv6 address of flow source or biflow initiator. Present for IPv6 flows or IPv6-mapped IPv4 flows only.

destinationIPv6Address IE 28, 16 octets, unsigned, FULL

IPv6 address of flow source or biflow responder. Present for IPv6 flows or IPv6-mapped IPv4 flows only.

sourceIPv4Address IE 8, 4 octets, unsigned, FULL

IPv4 address of flow source or biflow initiator. Present for IPv4 flows without IPv6-mapped addresses only.

destinationIPv4Address IE 12, 4 octets, unsigned, FULL

IPv4 address of flow source or biflow responder. Present for IPv4 flows without IPv6-mapped addresses only.

dnsTTL CERT (PEN 6871) IE 199, 4 octets, unsigned

DNS Time To Live. This is an unsigned integer that specifies the time interval, in seconds, that the resource record may be cached for. This contains a value of zero for DNS Queries

observationDomainId IE 149, 4 octets, unsigned

An identifier of an Observation Domain that is locally unique to an Exporting Process. This is typically set on the yaf command line.

yafFlowKeyHash CERT (PEN 6871) IE 106, 4 octets, unsigned

The hash of the 5-tuple (sourceIPAddress, destinationIPAddress, sourcePort, destinationPort, protocol) and vlanId.

dnsRRType CERT (PEN 6871) IE 175, 2 octets, unsigned

DNS Query/Response Type. This corresponds with the QTYPE field in the DNS Question Section or the TYPE field in the DNS Resource Record Section. This field determines the type of data found in the smDNSData field.

sourceTransportPort IE 7, 2 octets, unsigned, FULL

TCP or UDP port on the flow source or biflow initiator endpoint.

destinationTransportPort IE 11, 2 octets, unsigned, FULL

TCP or UDP port on the flow destination or biflow responder endpoint.

vlanId IE 58, 2 octets, unsigned, FULL

802.1q VLAN tag of the first packet in the forward direction of the flow.

dnsId CERT (PEN 6871) IE 226, 2 octets, unsigned

DNS Transaction ID. This identifier is used by the requester to match up replies to outstanding queries

protocolIdentifier IE 4, 1 octet, unsigned, FULL

IP protocol of the flow.

dnsQueryResponse CERT (PEN 6871) IE 174, 1 octet, unsigned

DNS Query/Response header field. This corresponds with the DNS header one bit field, QR. If the message is a query (0), or a response (1).

dnsAuthoritative CERT (PEN 6871) IE 176, 1 octet, unsigned

DNS Authoritative header field. This corresponds with the DNS header one bit field, AA. This bit is only valid in responses (when dnsQueryResponse is 1), and specifies that the responding name server is an authority for the domain name in the question section.

dnsResponseCode CERT (PEN 6871) IE 177, 1 octet, unsigned

DNS Response Code (RCODE). This corresponds with the DNS RCODE header field. This field is 3 for a Name Error, 2 for a Server Failure, 1 for a Format Error, and 0 for No Error. See http://www.iana.org/assignments/dns-parameters for other valid values.

dnsSection CERT (PEN 6871) IE 178, 1 octet, unsigned

DNS Resource Record Section Field. This field is 0 if the information is from the Question Section, 1 for the Answer Section, 2 for the Name Server Section, and 3 for the Additional Section.

dnsName CERT (PEN 6871) IE 179, variable length

A DNS Query or Response Name. This field corresponds with the QNAME field in the DNS Question Section or the NAME field in the DNS Resource Record Section.

smDNSData CERT (PEN 6871) IE 927, variable length

The DNS Resource Record Data field. The information contained in this field depends on the type of resource record. For an A record, it is the resolving IPv4 address. For an AAAA record, the resolving IPv6 address. For an NS record, the NSDNAME. For a CNAME Record, a CNAME. For a SOA Record, the SOA MNAME field. For a PTR Record, the PTRDNAME. For an MX Record, the MX Exchange field. For a TXT Record, the TXT-DATA field. For a SRV Record, the Target field.

DNS Deduplication: DNS_DEDUP

(See also "DNS_DEDUP Block, Quick Reference".)

When DNS deduplication is enabled, the exporter examines DNS DPI data and caches tuples that have unique values for the fields dnsRRName, dnsRRType, and rrvalue (where rrvalue may be an IPv4 address, a SOA name, et cetera). These "DNS dedup" records are written to the exporter, and the optional DNS_DEDUP block configures when that occurs, the contents of the records, and limits which DNS record types are considered for deduplication.

Use the exporter's DNS_DEDUP or DNS_DEDUP_ONLY command to enable DNS deduplication.

The DNS records always contain the time when first seen and the unique information elements. It is possible to add the time of the most recent sighting, the number of times the key was seen, and the exporter name.

DNS_DEDUP Block Commands

The first command below is used at in top-level context to begin a DNS deduplication block, and the remaining commands are accepted within the block's context.

DNS_DEDUP *EXPORTER-NAME*

The DNS_DEDUP command begins a new DNS deduplication customization block which continues to the DNS_DEDUP END command. The block customizes the deduplication for EXPORTER-NAME, which must have DNS deduplication enabled. When only one exporter has been defined, the EXPORTER-NAME argument may be omitted.

DNS_DEDUP END

The DNS_DEDUP END command ends the definition of a DNS deduplication block and sets the context back to top-level.

LAST_SEEN

If present, super_mediator writes records when they are removed from the cache rather than when they are first seen. The records also contain the time of the most recent sighting and the hit count.

MAX_HIT_COUNT COUNT

Removes a DNS deduplication record from the cache (and writes it if LAST_SEEN is present) when the internal hit count reaches COUNT. The default COUNT is 5000. The maximum COUNT is 65535.

FLUSH_TIME FLUSH-SECONDS

Removes a DNS deduplication record from the cache (and writes it if LAST_SEEN is present) when the cache entry has not been seen for over FLUSH-SECONDS seconds. The default FLUSH-SECONDS is 300 seconds (5 minutes). The maximum FLUSH-SECONDS is 65535 (18 hours 12 minutes).

BASE64_ENCODE

Tells super_mediator to Base64 encode the domain names in the DNS records if EXPORTER-NAME is a TEXT or JSON exporter.

RECORDS [ DNS-RESOURCE-RECORD-TYPE-LIST ]

Limits deduplication to the DNS resource record types contained in DNS-RESOURCE-RECORD-TYPE-LIST, a comma-separated list of integers. Resource record types available are 0 (NXDomain), 1 (A), 2 (NS), 5 (CNAME), 6 (SOA), 12 (PTR), 15 (MX), 16 (TXT), 28 (AAAA), and 33 (SRV).

MAP( MAPNAME )

By default, super_mediator hashes the DNS resource record name to store and aggregate DNS resource records. In addition to the name, super_mediator can aggregate the data per vlanId, per observationDomainId, or per a group of either value with a VLAN_MAP or OBID_MAP. To use a map in addition to the resource record name key, define the map ("MAP Blocks") and specify MAP(MAPNAME) in the DNS_DEDUP block. When a MAP is used, a TEXT record will include an extra column at the end of the line with the MAPNAME and an IPFIX record will contain an additional information element observationDomainName with the name of the MAP.

EXPORTER test
  PATH ...
  DNS_DEDUP
EXPORTER END

VLAN_MAP x
  S1 [1, 2, 3]
  S2 [6]
  DISCARD_OTHER
VLAN_MAP END

DNS_DEDUP test
  MAP(x)
DNS_DEDUP END
ADD_EXPORTER_NAME

Adds the name of the exporter to the DNS dedup record. The name appears in the final column of TEXT output and in the observationDomainName element for JSON and IPFIX output. If a MAP command is also present, it takes precedence and this command is ignored.

DNS Deduplication IPFIX Record Format

The "DNS dedup" IPFIX records contain the following information elements. Note that some elements are present only when the DNS_DEDUP block contains the LAST_SEEN command.

flowStartMilliseconds, IE 152, 8 octets, unsigned

The time in milliseconds when the DNS resource record was first seen.

flowEndMilliseconds, IE 153, 8 octets, unsigned

The time in milliseconds of the last record seen by the exporter. LAST_SEEN only.

sourceIPv4Address, IE 8, 4 octets, unsigned

The IPv4 Address found in a DNS A Record.

dnsTTL, CERT (PEN 6871) IE 199, 4 octets, unsigned

The maximum TTL seen for the aggregated DNS records. LAST_SEEN only.

dnsRRType, CERT (PEN 6871) IE 175, 2 octets, unsigned

The type of Resource Record. This corresponds with the QTYPE field in the DNS Question Section or the TYP field in the DNS Resource Record Section.

smDedupHitCount, CERT (PEN 6871) IE 929, 2 octets, unsigned

The number of times the exporter saw this record since it was previously flushed from the cache. LAST_SEEN only.

dnsName, CERT (PEN 6871) IE 179, variable length

A DNS Response Name. This field corresponds with the NAME field in the DNS Resource Record Section.

smDNSData, CERT (PEN 6871) IE 927, variable length

A DNS Resource Record Data Element. This field corresponds with the RDATA in the DNS Resource Record Section. For CNAME records, this will be the canonical name. For NS Records, this will be the name server name. For AAAA records, this will be an IPv6 Address.

observationDomainName IE 300, variable length

This field may not be present. If the DNS_DEDUP block was configured with a VLAN_MAP or OBID_MAP ("MAP Blocks"), this field contains the name of the map corresponding to the vlanId or observationDomainId. If the DEDUP_CONFIG block contains ADD_EXPORTER_NAME, this the name of the exporter.

DNS Deduplication Text Format

DNS deduplicated records for TEXT exporters have the format

first_seen | rrtype | rrname | rrval

If LAST_SEEN is present, the output is in the following format:

first_seen | last_seen | rrtype | rrname | hitcount | rrval

The first_seen and last_seen columns are timestamps in the form Y-M-D H:M:S.s, for example 2012-01-23 04:45:13.897.

TLS/SSL Deduplication: SSL_CONFIG Part 1

(See also "SSL_CONFIG Block, Quick Reference".)

TLS/SSL deduplication causes the exporter to examine TLS DPI data and create a key comprised of the X.509 certificate serial number and certificate issuer. This key is compared to a cache (a hash table), and either it is inserted with a count of one if it does not exist or its count is incremented if it does exist. When the key is inserted, a flow record containing the certificate is exported. Once customizable conditions are met, the key is removed from the cache and an "SSL dedup" record (described later) is exported. If the same serial number and issuer pair is seen again after the flush, a new cache entry is created. Use the SSL_CONFIG block to customize the conditions for exporting a TLS/SSL deduplication record.

To enable TLS/SSL deduplication on an exporter, specify SSL_DEDUP or SSL_DEDUP_ONLY in the exporter block, or specify SSL_DEDUP in this SSL_CONFIG block.

Note: Using the SSL_CONFIG block to customize which TLS/SSL-related elements are exported by TEXT exporters is described in "TLS/SSL Certificate Export: SSL_CONFIG Part 2".

SSL_CONFIG Block Commands (for Deduplication)

The SSL_CONFIG command is used in the top-level context to begin the block, and the SSL_CONFIG END command closes the block. The commands below may be used in the context of that block when configuring TLS/SSL deduplication.

SSL_CONFIG EXPORTER-NAME

This command is used at top-level to begin a TLS/SSL configuration block related to the existing exporter named EXPORTER-NAME.

SSL_CONFIG END

Closes the TLS/SSL configuration block.

SSL_DEDUP

Enables TLS/SSL certificate deduplication on the records processed by the associated exporter.

MAX_HIT_COUNT COUNT

Causes the deduplication record to be removed from the cache when the internal count reaches COUNT. The default COUNT is 5000.

FLUSH_TIME FLUSH-SECONDS

Causes the deduplication record to be removed from the cache when the cache entry has not been updated for over FLUSH-SECONDS seconds. The default FLUSH-SECONDS is 300 seconds (5 minutes).

MAP( MAPNAME )

Extends the key used when inserting entries into the cache to include a value derived from either the observationDomainId or the vlanId. Specifically, MAPNAME is used to find an OBID_MAP or VLAN_MAP (see "MAP Blocks"), the map is used to find the name for current flow record's observationDomainId or vlanId, and that name is added to the unique cache key. When a MAP is used, the name that was part of the key is included in output. For an IPFIX or JSON exporter, the name appears in the observationDomainName element; for a TEXT exporter it appears in an extra column. A MAP named MAPNAME must exist at the time this command is seen.

ADD_EXPORTER_NAME

Adds the name of the exporter to the output. For IPFIX and JSON output, the value is in the observationDomainName element. For a TEXT exporter, it appears in an extra column. This option is ignored if the MAP() command appears in this block.

TLS/SSL Deduplication IPFIX Record Format

The TLS/SSL deduplication record has the following layout for an IPFIX exporter.

flowStartMilliseconds IE 152, 8 octets, unsigned

The first time this certificate was seen (start time of the first flow that contained this certificate).

flowEndMilliseconds IE 153, 8 octets, unsigned

The last time this certificate was seen (start time of the most recent flow that contained this certificate).

smDedupHitCount CERT (PEN 6871) IE 929, 8 octets, unsigned

The number of times this certificate was seen in the time period.

sslCertSerialNumber CERT (PEN 6871) IE 244, variable length

The serial number of the X.509 Certificate.

sslCertIssuerCommonName CERT (PEN 6871) IE 196, variable length

The common name of the Issuer (Certificate Authority) in the X.509 Certificate.

observationDomainName

The name of either the exporter or the MAP() entry that contained the original flow record's observationDomainId or vlanId.

TLS/SSL Deduplication Text Record Format

The CSV format of the SSL dedup record is as follows:

fseen | lseen | serial | hitcount | issuer

The SSL certificates exported will follow the same IPFIX format as described in yafdpi(1). The text format for SSL certificates is as follows:

serial | issuer | stime | id | ISE | cert_no | data

Serial is the serial number in hexadecimal. Issuer is the common name (id-at 3) of the X.509 Issuer (Certificate Authority). If no common name is present, the organizational unit name is used (id-at 11). stime is the time in milliseconds that the certificate was first seen. id is the object/member ID for the X.509 RelativeDistinguishedName Sequence. A list of common objects can be found above. If ISSUER, SUBJECT, OTHER, or EXTENSION field lists are present within the SSL_CONFIG block, super_mediator will only print objects that are present within the field lists. ISE denotes if the data came from an Issuer Field(I), Subject Field(S), or Extension Field (E). For fields that are not associated with the issuer, subject, or extension but describe other characteristics of the certificate, an I will be used (for example, not-before or not-after timestamps). cert seq no signifies which certificate the data came from in the certificate chain. Usually, this field will contain a 0, 1, or 2.

General Deduplication: DEDUP_CONFIG

(See also "DEDUP_CONFIG Block, Quick Reference".)

An exporter may be configured to deduplicate records on nearly any DPI information element value. The DEDUP_CONFIG block enables and configures general deduplicatation for an exporter. The block contains the elements used for deduplicatation and controls when the deduplicated record is emitted.

For each record with DPI, the exporter creates a key containing the values of:

The key is inserted into a cache with a count of 1 or, if that key is already present, the counter for that key is incremented.

Records in the cache are removed and written to the exporter when of the following is true: the counter reaches a value (default 5000), the unique tuple has not been seen within a certain time period (default 300 seconds [5 minutes]), or super_mediator exits.

A record may be deduplicated on multiple elements, but each key contains only one DPI element. For example, if you deduplicate on the source IP address and both httpUserAgent and httpGet, two cache entries are created: one using the source IP and httpUserAgent and another using source IP and httpGet.

To have the exporter write only these deduplicated records, set DEDUP_ONLY in the block that defines the exporter.

Note: In super_mediator-1.x and earlier, general deduplication recognized DNS and SSL records and performed special handling for those records. As of v2.0.0, those record types no longer receive special treatment.

DEDUP_CONFIG Block Commands

The DEDUP_CONFIG command is used in the top-level context to begin a general deduplication block. The DEDUP_CONFIG END command closes the block and sets context back to top-level. At least one PREFIX command must be given within the block.

DEDUP_CONFIG EXPORTER-NAME

Enables general deduplication for the exporter named EXPORTER-NAME and begins the configuration block. EXPORTER-NAME must name a previously defined exporter.

DEDUP_CONFIG END

Closes the block and sets the context to top-level.

PREFIX FILE-PREFIX RECORD-KEY MAP( MAPNAME ) [ FIELD-LIST ]

The PREFIX command may take four forms, with this form showing a fully specified command. The three others are described next.

A PREFIX command defines which information element(s) are to deduplicated, what additional parts of the record make up the key, and the label used to identify those elements. The FILE-PREFIX and FIELD-LIST are required; the other values are optional.

FILE-PREFIX is the label for the records; its exact meaning depends on the format of the exporter:

  • For a JSON exporter, the record is labeled with that prefix, and all records are written to the same (possibly rotating) file.

  • For an IPFIX exporter, the FILE-PREFIX is not used; all records are written to the same file.

  • For a TEXT exporter, FILE-PREFIX is the prefix of the name of the file in which the deduplicated records are written. In this case, the PATH value must name a directory in which the files are to be created. The files will be rotated and a timestamp appended to the name.

The RECORD-KEY must be SIP, DIP, or FLOWKEYHASH. It determines what, in addition to the DPI value itself, is part of the deduplicating tuple. Typically SIP (source IP address) is better for values associated with the forward part of the flow record (e.g., httpUserAgent, httpGet) and DIP (destination IP address) is better for values from the reverse direction (httpHost, httpResponse). FLOWKEYHASH is a 32-bit value representing the entire 5-tuple (source IP address and port, destination IP address and port, and IP protocol).

MAP() adds a vlanId, an observationDomainId, or a group of those identifiers to the deduplicating tuple in addition to the RECORD-KEY. The VLAN_MAP or OBID_MAP named MAPNAME must be defined before the DEDUP_CONFIG block in the configuration file. The name of the map is added to the output, in either the final column of a TEXT record or in the observationDomainName field of a JSON or IPFIX exporter.

FIELD-LIST is one or more information element names or information element IDs, separated by commas and surrounded by square brackets. The records are deduplicated on each element in the list independently.

PREFIX FILE-PREFIX MAP( MAPNAME ) [ FIELD-LIST ]

When RECORD-KEY is not specified to the PREFIX command, the source IP address is used.

PREFIX FILE-PREFIX RECORD-KEY [ FIELD-LIST ]

The use of a MAP in the PREFIX command is optional.

PREFIX FILE-PREFIX [ FIELD-LIST ]

Using the default RECORD-KEY of SIP and not using a MAP.

MERGE_TRUNCATED

Tells super_mediator to be less strict in deduplicating values by merging truncated strings. For example, without MERGE_TRUNCATED, four records are exported:

2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|1|Mozilla/5.0 (X11; U; CrOS i686
2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|1|Mozilla/5.0 (X11; U; CrOS i686 0.10.
2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|4|Mozilla/5.0 (X11; U; CrOS i686 0.10.146;
2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|18|Mozilla/5.0 (X11; U; CrOS i686 0.10.146; en-US)

With MERGE_TRUNCATED, those become one record:

1454017979940|1454018281649|10.10.1.44|1454017980035|978282914|18|Mozilla/5.0 (X11; U; CrOS i686 0.10.146; en-US)
FLUSH_TIME FLUSH-SECONDS

Causes the exporter to write the dedup record and remove it from the cache when the record's key has not been seen for over FLUSH-SECONDS seconds. The default value for FLUSH-SECONDS is 300 seconds (5 minutes).

MAX_HIT_COUNT COUNT

Causes the exporter to write the dedup record and remove it from the cache when it has been seen COUNT times. The default value of COUNT is 5000.

ADD_EXPORTER_NAME

Adds the name of the exporter to the dedup record. The name appears in the final column of TEXT output and in the observationDomainName element for JSON and IPFIX output. If a MAP command is also present, it takes precedence and this command is ignored.

DEDUP_CONFIG Output

deduplication can be used with any type of EXPORTER. This block can also be used to configure incoming deduplicated records from another super_mediator. The pipe-delimited format for any TEXT file (except SSL) is as follows (line wrapped for readability):

first_seen | last_seen | sourceIP or dstIP | flow stime | \
flowkeyhash | hitcount | value | map or exporter name

where first_seen is the time of the first record, last_seen is the time of the last record seen with this tuple. SourceIP or dstIP is the IP address for the flow which contained this value. By default, super_mediator stores values with the source IP address, but this behavior can be changed with the PREFIX command. flow stime is the start time of the last flow to have this data present within the payload of the flow. flowkeyhash is the hash of the last flow's 5-tuple to have this data present within the payload of the flow. The flow stime and the flowkeyhash can provide a unique key for tracing the record back to flow data. The hitcount is the number of times the tuple was seen within first_seen and last_seen. The value is the value of the information element. The map or exporter name is optional and will be present if ADD_EXPORTER_NAME is present in the DEDUP_CONFIG block or a PREFIX line is configured with a MAP.

For SSL, the format is slightly different (line wrapped for readability):

first_seen | last_seen | sourceIP or dstIP | flow stime | \
flowkeyhash | hitcount | serial1 | issuer1 | serial2 | issuer2 | \
map or exporter name

where serial1 is the serial number (in hex) of the first certificate in the SSL certificate chain. issuer1 is the issuer's common name (id 3) of the first certificate in the SSL certificate chain. serial2 is the serial number (in hex) of the second certificate in the SSL certificate chain and issuer2 is the the issuer's common name (id 3) of the second certificate in the chain.

A list of information element IDs can be found above in DPI_CONFIG. The only valid DNS and SSL/TLS information element IDs are 179 and 244, respectively. 179 will enable deduplication of DNS queries only. See the DNS_DEDUP block for more information on deduplicating on DNS responses. 244 will deduplicate SSL certificate chains used by a particular IP. To deduplicate on all SSL certificates, see the SSL_DEDUP option in the SSL_CONFIG block.

DEDUP_CONFIG "exporter1"
    PREFIX "useragent" [111]
    PREFIX "host" DIP [120]
    PREFIX "p0f" [36, 37, 107]
    PREFIX "dns" [179]
    PREFIX "ssl" [244]
DEDUP_CONFIG END

For IPFIX exporters, the following information elements will be exported:

monitoringIntervalStartMilliSeconds IE 359, 8 octets, unsigned

The first time this data was seen (start time of the flow that contained this data).

monitoringIntervalEndMilliSeconds IE 360, 8 octets, unsigned

The last time this data was seen (start time of the flow that contained this data).

flowStartMilliseconds IE 152, 8 octets, unsigned

The start time of the last flow that contained the data being aggregated. The flowStartMilliseconds, with the yafFlowKeyHash, can create a unique key that can be used to correlate with a SiLK flow or PCAP repository.

smDedupHitCount CERT (PEN 6871) IE 929, 8 octets, unsigned

The number of times this data was seen in the time period.

sourceIPv6Address IE 27, 16 octets, unsigned, FULL

IPv6 address of flow source or biflow initiator. Present for IPv6 flows or IPv6-mapped IPv4 flows only.

sourceIPv4Address IE 8, 4 octets, unsigned, FULL

IPv4 address of flow source or biflow initiator. Present for IPv4 flows without IPv6-mapped addresses only.

yafFlowKeyHash CERT (PEN 6871) IE 106, 4 octets, unsigned

The hash of the 5-tuple (sourceIPAddress, destinationIPAddress, sourcePort, Destinationport, protocol) and vlanId.

observationDomainName IE 300, variable length

This field is populated if the DEDUP_CONFIG block was configured with ADD_EXPORTER_NAME or the PREFIX line contained a VLAN_MAP or OBID_MAP.

Information element depends on FIELD_LIST defined in DEDUP_CONFIG block.

This information element will depend on how super_mediator is configured and which information elements are present in the FIELD_LIST provided to PREFIX.

sslCertSerialNumber CERT (PEN 6871) IE 244, variable length SSL_ONLY

The serial number of the first X.509 Certificate in the chain (local). Only present for SSL deduplicated data (FIELD_LIST = [244]).

sslCertIssuerCommonName CERT (PEN 6871) IE 196, variable length SSL_ONLY

The common name of the Issuer (Certificate Authority) in the X.509 Certificate (first certificate in the chain). Only present for SSL deduplicated data.

sslCertSerialNumber CERT (PEN 6871) IE 244, variable length SSL_ONLY

The serial number of the second X.509 Certificate in the chain. Only present for SSL deduplicated data.

sslCertIssuerCommonName CERT (PEN 6871) IE 196, variable length SSL_ONLY

The common name of the Issuer (Certificate Authority) in the X.509 Certificate. Only present for SSL deduplicated data.

DEDUP_CONFIG EXPORTER-NAME

The DEDUP_CONFIG command begins a new DEDUP_CONFIG block and it continues to the DEDUP_CONFIG END command. The only required argument to the DEDUP_CONFIG command is the EXPORTER-NAME for which the deduplication configuration applies. This should match the name of one and only one TEXT EXPORTER from the configuration file. The DEDUP_CONFIG block should be placed after the EXPORTER block for which it refers to.

DEDUP_CONFIG END

The DEDUP_CONFIG END command ends the definition of a DEDUP_CONFIG block. Following a DEDUP_CONFIG END command, top-level commands are again accepted.

PREFIX FILE-PREFIX SIP|DIP|FLOWKEYHASH MAP( MAPNAME ) [ FIELD-LIST ]

At least one PREFIX command must be present within a DEDUP_CONFIG block. The PREFIX command defines which information elements will be deduplicated and the filename prefix in which the records will be written to. If the EXPORTER for which this DEDUP_CONFIG applies is a TEXT exporter, it should have specified a PATH that is a file directory. FILE-PREFIX will be the prefix of the filename that deduplicated records will be written to. The timestamp will be appended to the FILE-PREFIX. If the EXPORTER is a JSON EXPORTER or an IPFIX File EXPORTER, all deduplicated information will be written to the same file. The PATH keyword for the associated EXPORTER should be a valid file path with file prefix. For JSON exporters, the deduplication information will be identified by the key FILE-PREFIX.

Optionally, you can specify if the values should be cached with the source or destination IP address. By default, super_mediator uses the source IP address (SIP). You may decide to use the destination IP address for fields that are traditionally found in the reverse direction of the flow, such as "httpResponse" or "httpHost". Or you can use FLOWKEYHASH to deduplicate flows that have the same 5-tuple and data field.

The FIELD-LIST is a list of information element IDs, separated by a comma, and surrounded by square brackets, [ and ].

To use a VLAN_MAP or OBID_MAP in addition to the SIP, DIP, FLOWKEYHASH key use MAP(MAPNAME). The VLAN_MAP or OBID_MAP with MAPNAME must be defined before the DEDUP_CONFIG block in the configuration file. If a MAP is used, then a TEXT record will include an extra column at the end of the line with the MAPNAME and an IPFIX record will contain an additional information element observationDomainName with the name of the MAP.

A list of information element IDs can be found above in DPI_CONFIG. The only valid DNS and SSL/TLS information element IDs are 179 and 244, respectively. 179 will enable deduplication of DNS queries only. See the DNS_DEDUP block for more information on deduplicating on DNS responses. 244 will deduplicate SSL certificate chains used by a particular IP. To deduplicate on all SSL certificates, see the SSL_DEDUP option in the SSL_CONFIG block.

DEDUP_CONFIG "exporter1"
    PREFIX "useragent" [111]
    PREFIX "host" DIP [120]
    PREFIX "p0f" [36, 37, 107]
    PREFIX "dns" [179]
    PREFIX "ssl" [244]
DEDUP_CONFIG END
MERGE_TRUNCATED

If present, super_mediator will be less strict in deduplicating values by merging truncated values into complete cached records. For example, the following records:

2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|1|Mozilla/5.0 (X11; U; CrOS i686
2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|1|Mozilla/5.0 (X11; U; CrOS i686 0.10.
2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|4|Mozilla/5.0 (X11; U; CrOS i686 0.10.146;
2016-01-28 21:52:59.940|2016-01-28 21:58:01.649|10.10.1.44|2016-01-28 21:53:00.035|978282914|18|Mozilla/5.0 (X11; U; CrOS i686 0.10.146; en-US)

will be collapsed into one record:

1296251215151|1296252025761|10.10.1.6|24|Mozilla/5.0 (X11; U; CrOS i686 0.10.146; en-US)
FLUSH_TIME FLUSH-SECONDS

If present, super_mediator will flush a dedup record from the cache when a new record has not been seen for over FLUSH-SECONDS. super_mediator will write the record when either the FLUSH_TIME or MAX_HIT_COUNT condition has been met. The default FLUSH-SECONDS is 300 seconds, or 5 minutes.

MAX_HIT_COUNT COUNT

If present, super_mediator will flush a dedup record from the cache when the internal hit count reaches COUNT. The record will be written at this time. The default COUNT is 5000.

ADD_EXPORTER_NAME

Adds the name of the exporter to the deduplicated record. The name appears in the final column of TEXT output and in the observationDomainName element for JSON and IPFIX output. If a MAP command is also present, it takes precedence and this command is ignored.

MAP Blocks

(See also "MAP Blocks, Quick Reference".)

Using of MAP blocks is optional; they may be used when performing DNS-, TLS/SSL-, or general deduplication.

The OBID_MAP and VLAN_MAP blocks provide a way to create a named dictionary that assigns names to groups of observationDomainId or vlanId values. The dictionary may then be referenced (via the MAP(MAPNAME) command) in the DNS_DEDUP, SSL_CONFIG, or DEDUP_CONFIG blocks. Multiple deduplicatation blocks may reference the same map. During deduplication, the record's observationDomainId or vlanId is mapped to an entry whose name becomes part of the key for deduplication. During export, the entry's name is used as the value for observationDomainName.

Each dictionary has a name that is used to reference the dictionary within super_mediator.conf, and the name must be unique across all dictionaries. The map must be defined before it can be used.

The map block must specify how values not explicitly specified in the map are handled by either specifying DISCARD_OTHER or assigning a name to OTHER.

OBID_MAP and VLAN_MAP Block Commands

The first two commands are used in the top-level context to begin a OBID_MAP block or a VLAN_MAP block. The remaining commands are accepted within that context.

OBID_MAP MAPNAME

Creates a dictionary named MAPNAME to hold observationDomainId values. The block continues to the OBID_MAP END command.

VLAN_MAP MAPNAME

Creates a dictionary named MAPNAME to hold vlanId values. The block continues to the VLAN_MAP END command.

OBID_MAP END

Closes the OBID_MAP block and sets the context back to top-level.

VLAN_MAP END

Closes the VLAN_MAP block and sets the context back to top-level.

DISCARD_OTHER

Tells the deduplication code to ignore a flow record if its observationDomainId or vlanId value is not explicitly listed in this block. Either this command or the next is required.

GROUP-NAME OTHER

Causes GROUP-NAME to be used for an observationDomainId or vlanId value that is not explicitly listed elsewhere in the block. Either this command or DISCARD_OTHER is required.

GROUP-NAME [ INTEGER-LIST ]

Creates a new entry in the dictionary that maps the values in INTEGER-LIST to GROUP-NAME. INTEGER-LIST is a comma-separated list of non-negative integer values; the list may not be empty. The maximum supported value is 0xffffffff (4294967295) for OBID_MAP and 0xfff (4095) for VLAN_MAP. This command may appear multiple times within the block.

MAP Block Examples

EXPORTER IPFIX SINGLE_FILE "dns-dedup"
  PATH "-"
  DNS_DEDUP_ONLY
EXPORTER END
VLAN_MAP networks
  "S1" [1,2,3]
  "S2" [4,5,6]
  "S3" [7,8,9]
  "other" OTHER
VLAN_MAP END
DNS_DEDUP "dns-dedup"
  ADD_EXPORTER_NAME
  MAP("networks")
DNS_DEDUP END

EXPORTER IPFIX SINGLE_FILE "ssl-dedup"
  PATH "-"
  SSL_DEDUP_ONLY
EXPORTER END
OBID_MAP "mymap"
   "S1" [1,2,3]
   "S2" [4,5,6]
   "S3" [7,8,9]
   DISCARD_OTHER
OBID_MAP END
SSL_CONFIG "ssl-dedup"
  ADD_EXPORTER_NAME
  MAP("mymap")
SSL_CONFIG END

TLS/SSL Certificate Export: SSL_CONFIG Part 2

(See also "SSL_CONFIG Block, Quick Reference".)

To customize which TLS/SSL information elements are exported by a TEXT or JSON exporter, the SSL_CONFIG block is used. This block may be used in conjunction with an exporter's DPI_FIELD_LIST setting. When this block is present and linked to an exporter, the exporter only writes the TLS/SSL fields mentioned in this block.

Note: Using the SSL_CONFIG block to customize TLS/SSL record deduplication is described in "TLS/SSL Deduplication: SSL_CONFIG Part 1".

SSL_CONFIG Block Commands (for TEXT Output Customization)

In the top-level context, the TLS/SSL configuration block begins with SSL_CONFIG and closes with SSL_CONFIG END. The commands below are used for configuring which information elements are written by a TEXT exporter.

SSL_CONFIG EXPORTER-NAME

The top-level command to begin an TLS/SSL configuration block. The EXPORTER-NAME argument is required and must name an existing exporter.

SSL_CONFIG END

Closes the configuration block and sets the context to top-level.

ISSUER [ FIELD-LIST ]

If present, super_mediator will only write certain X.509 object identifier values specified in FIELD-LIST from the ISSUER x.509 RelativeDistinguishedName Sequence. The FIELD-LIST is a list of X.509 RelativeDistinguishedName Sequence object IDs, separated by a comma, and surrounded by square brackets, [ and ]. A list of common object IDs is listed above under the DPI_CONFIG block. The default behavior is to print all issuer fields, however if any list (ISSUER, SUBJECT, OTHER, EXTENSIONS) is present in the SSL_CONFIG block, super_mediator will only print the elements specifically denoted in the FIELD-LIST. To print all elements use [*] to denote all elements.

SUBJECT [ FIELD-LIST ]

If present, super_mediator will only write certain X.509 object identifier values specified in FIELD-LIST from the SUBJECT x.509 RelativeDistinguishedName Sequence. The FIELD-LIST is a list of X.509 RelativeDistinguishedName Sequence object IDs, separated by a comma, and surrounded by square brackets, [ and ]. A list of common object IDs is listed above under the DPI_CONFIG block. The default behavior is to print all subject fields, however if any list (ISSUER, SUBJECT, OTHER, EXTENSIONS) is present in the SSL_CONFIG block, super_mediator will only print the elements specifically denoted in the FIELD-LIST. To print all elements use [*] to denote all elements.

OTHER [ FIELD-LIST ]

If present, super_mediator will only write certain SSL/TLS information elements specified in FIELD-LIST. The FIELD-LIST is a list of information element IDs, separated by a comma, and surrounded by square brackets, [ and ]. A list of SSL/TLS Information Element IDs is listed above under the DPI_CONFIG block (valid elements have a label of tls). The default behavior is to print all "other" fields, however if any list (ISSUER, SUBJECT, OTHER, EXTENSIONS) is present in the SSL_CONFIG block, super_mediator will only print the elements specifically denoted in the FIELD-LIST. To print all elements use [*] to denote all elements. If the full X.509 certificate is available and if openSSL is available, super_mediator has the ability to perform a MD5 or SHA1 hash of the certificate. To print these values, list 299 (MD5) or 298 (SHA1) in the FIELD-LIST. super_mediator can also Base64 encode the full X.509 certificate, if available. To export the Base64 encoded certificate, add 296 to the OTHER FIELD-LIST.

EXTENSIONS [ FIELD-LIST ]

If present, super_mediator will write particular extension types from the SSL/TLS certificate specified in FIELD-LIST. The FIELD-LIST is a list of X.509 RelativeDistinguishedName Sequence object IDs, separated by a comma, and surrounded by square brackets, [ and ]. A list of acceptable object IDs is listed below. super_mediator will only export extension types if they are specifically listed within the SSL_CONFIG block. Some extension types contain a sequence of values. Each element in the sequence will be exported on a separate line. The default behavior is to print all extension fields, however if any list (ISSUER, SUBJECT, OTHER, EXTENSIONS) is present in the SSL_CONFIG block, super_mediator will only print the elements specifically denoted in the FIELD-LIST. To print all elements use [*] to denote all elements. The accepted extension types are:

extension type         | id

subjectKeyIdentifier   | 14*

keyUsage               | 15*

privateKeyUsagePeriod  | 16*

subjectAltName         | 17

issuerAltName          | 18

certificateIssuer      | 29

cRLDistributionPoints  | 31

certificatePolicies    | 32*

* denotes when B<super_mediator> writes the value in hex.

For rewritten SSL certificates, the following EXTENSION values are supported:

sslCertExtSubjectKeyIdent      | 316

sslCertExtKeyUsage             | 317

sslCertExtPrivKeyUsagePeriod   | 318

sslCertExtSubjectAltName       | 319

sslCertExtIssuerAltName        | 320

sslCertExtCertIssuer           | 321

sslCertExtCrlDistribution      | 322

sslCertExtCertPolicies         | 323

sslCertExtAuthorityKeyIdent    | 324

sslCertExtExtendedKeyUsage     | 325
CERT_FILE FILENAME

If present, super_mediator will write SSL certificates in the above format to the file path provided by FILENAME. This option is only valid for TEXT Exporters (not JSON). The file will be rotated or locked if those options are configured in the EXPORTER block associated with the SSL_CONFIG block.

Textual Output Configuration: DPI_CONFIG

(See also "DPI_CONFIG Block, Quick Reference".)

The first command below is used at the top-level to begin a DPI configuration block, and the remaining commands are accepted within the DPI config context. The information from the dpi config block is used by super_mediator to determine how to write the DPI data in the text files, or if MULTI_FILES is present in the EXPORTER block, it determines the filenames of the CSV files that contain the DPI data. If a DPI_CONFIG block is present, it will be used for all exporters that are exporting DPI Information (it does not effect DNS deduplication exporters). For example, if an exporter is using a custom field list FIELDS, and DPI is an item in that list, it will only write flows that contain a DPI element that is listed in the DPI_CONFIG block. However, if the DPI_FIELD_LIST keyword is also present in the exporter, the DPI_FIELD_LIST takes precedence over the DPI_CONFIG items.

DPI_CONFIG INDEX-NAME

The DPI_CONFIG command begins a new DPI config block and it continues to the DPI_CONFIG END command. The INDEX-NAME is an optional argument to the DPI_CONFIG command. If INDEX-NAME is present, the table name or filename for the flow index information will have the name INDEX-NAME. If not present, the default flow will be used.

DPI_CONFIG END

The DPI_CONFIG END command ends the definition of DPI_CONFIG block. Following the DPI_CONFIG END command, top-level commands are accepted. Only one DPI_CONFIG block is permitted in a configuration file.

TABLE TABLE-NAME [ TABLE-LIST ]

If TABLE is present, label the lines that contain information element ids in the TABLE-LIST with the word TABLE-NAME. TABLE-NAME should not contain spaces. The TABLE-LIST is a list of information element ids from the below list, separated by a comma, and surrounded by square brackets, [ and ]. The following example will label the lines that contain user agent strings (id 111), with "http_ua":

TABLE http_ua [111]

The following list contains the default information element ids and their respective table name label (see yafdpi(1) for a description of each information element):

information element name  | id     | table label

osName                    | 36     | p0f

osVersion                 | 37     | p0f

osFingerprint             | 107    | p0f

httpServerString          | 110    | http

httpUserAgent             | 111    | http

httpGet                   | 112    | http

httpConnection            | 113    | http

httpVersion               | 114    | http

httpReferer               | 115    | http

httpLocation              | 116    | http

httpHost                  | 117    | http

httpContentLength         | 118    | http

httpAge                   | 119    | http

httpAccept                | 120    | http

httpAcceptLanguage        | 121    | http

httpContentType           | 122    | http

httpResponse              | 123    | http

httpCookie                | 220    | http

httpSetCookie             | 221    | http

pop3TextMessage           | 124    | pop3

ircTextMessage            | 125    | irc

tftpFilename              | 126    | tftp

tftpMode                  | 127    | tftp

slpVersion                | 128    | slp

slpMessageType            | 129    | slp

slpString                 | 130    | slp

ftpReturn                 | 131    | ftp

ftpUser                   | 132    | ftp

ftpPass                   | 133    | ftp

ftpType                   | 134    | ftp

ftpRespCode               | 135    | ftp

imapCapability            | 136    | imap

imapLogin                 | 137    | imap

imapStartTLS              | 138    | imap

imapAuthenticate          | 139    | imap

imapCommand               | 140    | imap

imapExists                | 141    | imap

imapRecent                | 142    | imap

rtspURL                   | 143    | rtsp

rtspVersion               | 144    | rtsp

rtspReturnCode            | 145    | rtsp

rtspContentLength         | 146    | rtsp

rtspCommand               | 147    | rtsp

rtspContentType           | 148    | rtsp

rtspTransport             | 149    | rtsp

rtspCSeq                  | 150    | rtsp

rtspLocation              | 151    | rtsp

rtspPacketsReceived       | 152    | rtsp

rtspUserAgent             | 153    | rtsp

rtspJitter                | 154    | rtsp

sipInvite                 | 155    | sip

sipCommand                | 156    | sip

sipVia                    | 157    | sip

sipMaxForwards            | 158    | sip

sipAddress                | 159    | sip

sipContentLength          | 160    | sip

sipUserAgent              | 161    | sip

smtpHello                 | 162    | smtp

smtpFrom                  | 163    | smtp

smtpTo                    | 164    | smtp

smtpContentType           | 165    | smtp

smtpSubject               | 166    | smtp

smtpFilename              | 167    | smtp

smtpContentDisposition    | 168    | smtp

smtpResponse              | 169    | smtp

smtpEnhanced              | 170    | smtp

smtpSize                  | 222    | smtp

sshVersion                | 171    | ssh

nntpResponse              | 172    | nntp

nntpCommand               | 173    | nntp

sslCipher                 | 185    | tls

sslClientVersion          | 186    | tls

sslServerCipher           | 187    | tls

sslCompressionMethod      | 188    | tls

sslCertVersion            | 189    | tls

sslCertSignature          | 190    | tls

sslCertIssuerCountryName* | 191    | tls

sslCertIssuerOrgName*     | 192    | tls

sslCertIssuerOrgUnitName* | 193    | tls

sslCertIssuerZipCode*     | 194    | tls

sslCertIssuerState*       | 195    | tls

sslCertIssuerCommonName*  | 196    | tls

sslCertIssuerLocalityName*| 197    | tls

sslCertIssuerStreetAddress*| 198    | tls

sslCertSubCountryName*    | 200    | tls

sslCertSubOrgName*        | 201    | tls

sslCertSubOrgUnitName*    | 202    | tls

sslCertSubZipCode*        | 203    | tls

sslCertSubState*          | 204    | tls

sslCertSubCommonName*     | 205    | tls

sslCertSubLocalityName*   | 206    | tls

sslCertSubStreetAddress*  | 207    | tls

sslCertSerialNumber       | 244    | tls (in hexadecimal)

sslCertValidityNotBefore  | 247    | tls

sslCertValidityNotAfter   | 248    | tls

sslPublicKeyAlgorithm     | 249    | tls

sslPublicKeyLength        | 250    | tls

sslRecordVersion          | 288    | tls

sslServerName             | 294    | tls

sslCertificateHash        | 295    | tls (in hexadecimal)

sslCertificate            | 296    | tls (Base64 encoded)

sslCertificateMD5         | 299    | tls (in hexadecimal)

sslCertificateSHA1        | 298    | tls (in hexadecimal)

mysqlUsername             | 223    | mysql

mysqlCommandText          | 225    | mysql

dhcpFingerprint           | 242    | dhcp

dhcpVendorCode            | 243    | dhcp

httpAuthorization         | 252    | http

httpVia                   | 253    | http

httpXForwardedFor         | 254    | http

httpExpires               | 255    | http

httpRefresh               | 256    | http

httpIMEI                  | 257    | http

httpIMSI                  | 258    | http

httpMSISDN                | 259    | http

httpSubscriber            | 260    | http

httpAcceptCharset         | 261    | http

httpAcceptEncoding        | 262    | http

httpAllow                 | 263    | http

httpDate                  | 264    | http

httpExpect                | 265    | http

httpFrom                  | 266    | http

httpProxyAuthentication   | 267    | http

httpUpgrade               | 268    | http

httpWarning               | 269    | http

httpDNT                   | 270    | http

httpXForwardedProto       | 271    | http

httpXForwardedHost        | 272    | http

httpXForwardedServer      | 273    | http

httpXDeviceId             | 274    | http

httpXProfile              | 275    | http

httpLastModified          | 276    | http

httpContentEncoding       | 277    | http

httpContentLanguage       | 278    | http

httpContentLocation       | 279    | http

httpXUaCompatible         | 280    | http

dnp3ObjectData            | 284    | dnp (in hexadecimal)

modbusData                | 285    | modbus (in hexadecimal)

enipData                  | 286    | enip (in hexadecimal)

rtpPayloadData            | 287    | rtp

Not all fields are turned on by default in YAF. See the yafDPIRules.conf to turn on/off certain fields.

*These items were removed as of YAF 2.3.0. SSL Certificate information elements were replaced by the X.509 object identifier value. Use id 443 in TABLE-LIST or DPI_FIELD_LIST to include all of the following elements. To specify individual fields, use the SSL_CONFIG block. Below is a list of common objects in an X.509 RelativeDistinguishedName Sequence (not all possibilities are listed):

object identifier name  | id     | table label

common name             | 3      | tls

countryName             | 6      | tls

localityName            | 7      | tls

stateOrProvinceName     | 8      | tls

streetAddress           | 9      | tls

organization            | 10     | tls

organizational unit     | 11     | tls

title                   | 12     | tls

postalCode              | 17     | tls

name                    | 41     | tls

DNS has a different format. TABLE-LIST should contain the DNS Resource Record Types in the below list:

Q/R Record Type           | id      | default table name

A Record                  | 1       | dns

NS Record                 | 2       | dns

CNAME Record              | 5       | dns

SOA Record                | 6       | dns

MX Record                 | 12      | dns

PTR Record                | 15      | dns

TXT Record                | 16      | dns

AAAA Record               | 28      | dns

SRV Record                | 33      | dns

DS Record*                | 43      | dns

RRSIG Record**            | 46      | dns

NSEC Record**             | 47      | dns

DNSKEY Record*            | 48      | dns

NSEC3 Record*             | 50      | dns

NSEC3PARAM Record*        | 51      | dns

All Others***             | 53      | dns

*Records that do not contain data for the rdata field.

**RRSIG Type records contain the signer name in the rdata field. NSEC Type records contains the next domain name in the rdata field.

***Any nonstandard query response type, such as 251 - Incremental Transfers, can be filtered by using 53.

Tombstone Block

(See also "TOMBSTONE_CONFIG Block, Quick Reference".)

The first command below is used at the top-level to begin a TOMBSTONE_CONFIG block, and the remaining commands are accepted within the TOMBSTONE_CONFIG context. The information from the TOMBSTONE_CONFIG block is used by super_mediator to decide if and how to send tombstone records to exporters. Tombstone records are IPFIX options template records that are to be used to extract information about the delay and operation of a chain of tools. They carry a creation timestamp and a timestamp from certain supported programs that forward the record.

TOMBSTONE_CONFIG Block Commands

TOMBSTONE_CONFIG

The TOMBSTONE_CONFIG command begins a tombstone configuration block and it continues until the TOMBSTONE_CONFIG END command. There should be at most one tombstone configuration block per configuration file.

TOMBSTONE_CONFIG END

The TOMBSTONE_CONFIG END command ends the definition of a tombstone configuration block. Following a TOMBSTONE_CONFIG END command, top-level commands are again accepted.

GEN_TOMBSTONE

Tells super mediator to generate and output tombstone records. Tombstone records will be sent to any exporter that is configured to receive stats. See other documentation for the configuration of stats.

TOMBSTONE_CONFIGURED_ID TOMBSTONE-CONFIGURED-ID

If present, the TOMBSTONE_CONFIGURED_ID command will override the default id value of 0 with the given TOMBSTONE-CONFIGURED-ID. This value must be specified in decimal and must be less than 65535.

EXAMPLES

TCP Collector

COLLECTOR TCP
    HOSTNAME "127.0.0.1"
    PORT 18000
COLLECTOR END

Poll-Directory Collector with Port 53 Filter

COLLECTOR DIRECTORY_POLL
    sourceTransportPort == 53
    destinationTransportPort == 53
    PATH "/tmp/flow/yaf-*"
    POLL 30
    MOVE "/data/flow/done"
    LOCK
COLLECTOR END

Filter on Collection (applied to all Collectors)

FILTER
    silkAppLabel == 80
    sourceTransportPort == 80
    destinationTransportPort == 80
FILTER END

TCP Exporter (rwflowpack) with name "SiLK"

EXPORTER IPFIX TCP SiLK
    HOSTNAME "localhost"
    PORT 18001
    FLOW_ONLY
EXPORTER END

TCP Exporter for flows that were collected by COLLECTOR "C1"

EXPORTER IPFIX TCP
    COLLECTOR == C1
    HOSTNAME "localhost"
    PORT 18001
    FLOW_ONLY
EXPORTER END

deduplicated DNS to CSV files

EXPORTER TEXT ROTATING_FILES
    PATH "/data/dns/yaf2dns"
    ROTATE 120
    LOCK
    DNS_DEDUP_ONLY
    DELIMITER ","
EXPORTER END

Example Data in "/data/dns/yaf2dns-20120504-0001.txt":

2012-04-10 04:41:54.194,2,wikimedia.org.,ns2.wikimedia.org.

Export to TEXT files with filter

EXPORTER TEXT SINGLE_FILE
    PATH "/data/http-only.txt"
    silkAppLabel == 80
    DPI_ONLY
EXPORTER END

Example Data in /data/http-only.txt:

flow|1441601726|1207802496583|1.2.3.4|208.80.152.2|6|1360|80|0
http|115|1441929406|1207802496582|http://en.wikipedia.org/wiki/Http
http|114|1441601726|1207802496583|HTTP/1.0
http|117|1441601726|1207802496583|en.wikipedia.org

deduplication options

DNS_DEDUP
    FLUSH_TIME 600
    BASE64_ENCODE
DNS_DEDUP END

Example Using BASE_64 Encode:

2010-07-21 11:51:15.166|6|bmVtby3kbGEubTlsLg==|bnMxLmTsYS5taWwu
2010-07-21 11:51:15.175|2|bm5zYy58kbGEuLWlsLg==|bnMnLmRsYS9taWwu

DPI Configuration options - MULTI_FILE Example

EXPORTER TEXT ROTATING_FILES
    PATH "/data/dpi"
    ROTATE 120
    LOCK
    DPI_ONLY
    MULTI_FILES
EXPORTER END
DPI_CONFIG
    TABLE myhttp [110, 111, 112]
    TABLE mydns [1, 28]
    TABLE myp0f [36, 37]
    TABLE myssl [244,247,248,443]
DPI_CONFIG END

Example Data for MULTI_FILE Config Example:

In /data/dpi/flow.txt0:
    1441601726|1207802496583|10.10.1.172|10.10.152.2|6|1360|80|0|0
    114422227|1207802496560|10.10.1.172|10.10.1.10|17|1599|53|0|0
    4144722023|1296251240788|10.10.0.204|10.10.4.3|6|54489|443|900|0
In /data/dpi/myp0f.txt0:
    36|1441601726|1207802496583|0|Windows
    37|1441601726|1207802496583|0|2000 SP2+, XP SP1+ (seldom 98)
In /data/dpi/myhttp.txt0:
    112|1441601726|1207802496583|0|/skins-1.5/monobook/headbg.jpg
    112|1441601726|1207802496583|0|/skins-1.5/common/images/poweredby_mediawiki_88x31.png
In /data/dpi/mydns.txt0:
    114422227|1207802496560|0|Q|14728|0|0|0|1|0|meta.wikimedia.org.
    114422227|1207802496560|0|R|14728|1|0|0|1|120|rr.pmtpa.wikimedia.org.|10.10.15.2
In /data/dpi/myssl.txt0:
    10|4144722023|1296251240788|0|S|0|Apple Inc

DPI Configuration with alternative index name

EXPORTER TEXT SINGLE_FILE DPIExporter
    PATH "/data/dpi/dpi_data.txt"
    DPI_ONLY
EXPORTER END
DPI_CONFIG flow_index
    TABLE kitchen_sink [110,111,112,113,2,15,186,200,201]
DPI_CONFIG END

Example Data with above configuration:

flow_index|1441208511|1207802506600|128.237.224.172|208.80.152.3|6|1370|80|0|0
kitchen_sink|111|1441208511|1207802506600|0|Mozilla/4.0(compatible; MSIE 7.0; Windows NT 5.1;)
kitchen_sink|112|1441208511|1207802506600|0|/wikipedia/commons/thumb/d/de/Www.wikipedia.org_screenshot.png/300px-Www.wikipedia.org_screenshot.png

IPset filter in Exporter

EXPORTER TEXT SINGLE_FILE
    SIP_ANY IN_LIST "mywatchlist.set"
    DIP_ANY IN_LIST "mywatchlist.set"
    PATH "/data/dpi/dpi_data.txt"
    DPI_ONLY
EXPORTER END

Custom Field List Example

EXPORTER TEXT SINGLE_FILE
    PATH "/data/flow/custom.txt"
    FIELDS stime,etime,sip,dip,sport,dport,protocol,vlan,pkts,bytes,dpi
    DPI_FIELD_LIST [111,112,110,1,2,5,6,12,244,248,247,443]
EXPORTER END

Custom SSL Field Example

EXPORTER TEXT SINGLE_FILE sslcerts
   PATH "/data/flow/sslcerts.txt"
   FIELDS stime, sip, dip, sport, dport, dpi
EXPORTER END
SSL_CONFIG sslcerts
   ISSUER [*]
   SUBJECT [7, 8]
   OTHER  [247, 248]
   EXTENSIONS [14, 15]
SSL_CONFIG END

Example Data with above configuration:

2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|8|S|0|Washington
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|7|S|0|Seattle
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|247|I|0|140601000000Z
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|248|I|0|150521235959Z
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|6|I|1|US
2015-04-08 19:14:32.251|1.2.3.4|3.4.5.6|49878|443|11|I|1|VeriSign Trust Network

SSL Certificate deduplication Example

EXPORTER TEXT ROTATING_FILES ssldedup
    PATH "/data/ssl/ssldedup"
    SSL_DEDUP_ONLY
    ROTATE 600
    LOCK
EXPORTER END

SSL_CONFIG ssldedup
    ISSUER [*]
    SUBJECT [*]
    OTHER [*]
    EXTENSIONS [*]
    CERT_FILE "/data/ssl/certs"
    FLUSH_TIME 1200
    MAX_HIT_COUNT 10000
SSL_CONFIG END

Example Data with above configuration:

$ head -n 5 /data/ssl/certs.txt
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|6|I|0|US
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|10|I|0|VeriSign, Inc.
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|11|I|0|VeriSign Trust Network
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|11|I|0|Terms of use at https://www.verisign.com/rpa (c)10
0x5294e23f0bfa5bb98c0|VeriSign Class 3 International Server CA - G3|\
2015-04-08 19:14:14.618|3|I|0|VeriSign Class 3 International Server CA - G3

$ head -n 5 /data/ssl/ssldedup.txt
2015-04-08 19:14:14.618|2015-04-08 19:14:30.117|0x5294e23f0bfa5bb98c0|2|VeriSign Class 3 International Server CA - G3
2015-04-08 19:14:39.902|2015-04-08 19:14:39.915|0x009ddde63d7dc9573067e|2|EssentialSSL CA
2015-04-08 19:14:39.902|2015-04-08 19:14:39.915|0x18b2cbbfc1f2f326462a4a|2|COMODO Certification Authority
2015-04-08 19:14:39.902|2015-04-08 19:14:39.915|0x2e79832eef31a6ee67a44|2|UTN - DATACorp SGC

HTTP Deduplication Example

EXPORTER TEXT ROTATING_FILES "httpdedup"
    PATH "/data/dedup"
    ROTATE 120
    LOCK
EXPORTER END

DEDUP_CONFIG "httpdedup"
    PREFIX "useragent" SIP [111]
    PREFIX "referer" [115]
    PREFIX "host" DIP [117]
    MAX_HIT_COUNT 65535
    FLUSH_TIME 600
    MERGE_TRUNCATED
DEDUP_CONFIG END

Example data with the above configuration:

$ head -n 4 /data/useragent_20110128220025.txt
2015-04-08 19:15:24.645|2015-04-08 19:15:24.645|10.10.1.60|2015-04-08 19:15:24.645|2221618956|2|Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8)
2015-04-08 19:15:28.809|2015-04-08 19:15:28.809|10.10.0.205|2015-04-08 19:15:24.645|546421315|1|TwitterAndroid/1.0.5 (109) Nexus One/8 (HTC;passion)
2015-04-08 19:15:11.544|2015-04-08 19:16:18.351|10.13.0.63|2015-04-08 19:15:24.645|213547784|2|urlgrabber/3.9.1 yum/3.2.28
2015-04-08 19:16:21.632|2015-04-08 19:16:21.668|10.10.1.5|2015-04-08 19:15:24.645|1315645613|4|OpenTable/3.2 CFNetwork/485.12.7 Darwin/10.4.0

$ head -n 4 /data/referer_20110128220025.txt
2015-04-08 19:15:24.645|2015-04-08 19:15:24.645|10.27.33.66|2015-04-08 19:15:24.645|532889529|10|http://www.google.com/search?hl=en&biw=1274
2015-04-08 19:15:28.809|2015-04-08 19:15:28.809|10.10.1.45|2015-04-08 19:15:24.645|654654123|4|http://www.ustream.tv/socialstream/6951299
2015-04-08 19:15:11.544|2015-04-08 19:16:18.351|10.11.0.139|2015-04-08 19:15:24.645|212754153|2|http://reviews.opentable.com/0938/33364/reviews.htm
2015-04-08 19:16:21.632|2015-04-08 19:16:21.668|10.10.1.31|2015-04-08 19:15:24.645|3264312556|1|http://www.northerntool.com/shop/tools/product.htm

$ head -n 4 /data/host_20110128220025.txt
2015-04-08 19:15:24.645|2015-04-08 19:15:24.645|10.10.0.196|2015-04-08 19:15:24.645|3251463421|1|www.funtrivia.com
2015-04-08 19:15:28.809|2015-04-08 19:15:28.809|172.16.0.163|2015-04-08 19:15:24.645|134313131|1|twitter.com
2015-04-08 19:15:11.544|2015-04-08 19:16:18.351|10.10.0.247|2015-04-08 19:15:24.645|313546131319|16|reviews.opentable.com
2015-04-08 19:16:21.632|2015-04-08 19:16:21.668|10.10.1.45|2015-04-08 19:15:24.645|210564613203|7|a2.twimg.com

Deduplication with a OBID_MAP

OBID_MAP "sensors"
    S1 [1, 2, 3]
    S2 [4, 5, 6]
    S3 [7, 8, 9]
    "S_OTHER" OTHER
OBID_MAP END

EXPORTER TEXT SINGLE_FILE "dedup"
    PATH "/data/dedup/"
EXPORTER END

DEDUP_CONFIG "dedup"
    PREFIX "useragent" SIP MAP(sensors) [111]
    MAX_HIT_COUNT 65535
    FLUSH_TIME 600
    MERGE_TRUNCATED
DEDUP_CONFIG END

Example with the above configuration:

$ head -n 4 /data/dedup/useragent.txt
2011-01-28 21:45:53.130|2011-01-28 21:49:33.630|10.10.1.13|2011-01-28 21:49:33.518|327929330|8|Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B367 Safari/531.21.10|S_OTHER
2011-01-28 21:48:45.775|2011-01-28 21:48:45.775|10.10.0.209|2011-01-28 21:48:43.713|2118954148|1|Google Update/1.2.183.39;winhttp|S2
2011-01-28 21:48:47.163|2011-01-28 21:48:47.163|10.10.0.209|2011-01-28 21:48:46.905|1985639067|1|Microsoft NCSI|S1

IPFIX Exporter with multiple protocol deduplication

EXPORTER IPFIX SINGLE_FILE "multi-dedup"
   PATH "/data/sm/super"
   ROTATE 120
   GZIP_FILES
   MOVE "/data/sm/done"
   DNS_DEDUP
   SSL_DEDUP
   NO_FLOW
EXPORTER END

DNS_DEDUP "multi-dedup"
   MAX_HIT_COUNT 5000
   FLUSH_TIME 600
   LAST_SEEN
DNS_DEDUP END

DEDUP_CONFIG "multi-dedup"
   PREFIX "httpua" [111]
   PREFIX "httplang" [121]
   PREFIX "httpserver" DIP [110]
   PREFIX "httphost" DIP [117]
   PREFIX "https" DIP [244]
   MERGE_TRUNCATED
   MAX_HIT_COUNT 10000
   FLUSH_TIME 600
DEDUP_CONFIG END

Logging options

LOG "/var/log/mediator.log"
LOGLEVEL DEBUG

KNOWN ISSUES

Bug reports and feature requests may be sent via email to <netsa-help@cert.org>.

yaf presently encodes the ICMP type and code information into the destinationTransportPort information element for ICMP and ICMP6 flows. super_mediator running in TEXT output mode writes the type in the sourceTransportPort field and the ICMP code in the destinationTransportPort field.

SEE ALSO

super_mediator(1), super_table_creator(1), yaf(1), applabel(1), yafdpi(1), rwflowpack(8), flowcap(8), gzip(1), mysqlimport(1), syslog(3)

AUTHORS

Emily Sarneso and the CERT Network Situational Awareness Group Engineering Team, <http://www.cert.org/netsa>.