NAME

rwsetbuild - Create a binary IPset file from list of IPs

SYNOPSIS

rwsetbuild [{--ip-ranges | --ip-ranges=DELIM}]
      [--record-version=VERSION] [--invocation-strip]
      [--note-add=TEXT] [--note-file-add=FILENAME]
      [--compression-method=COMP_METHOD]
      [{INPUT_TEXT_FILE | -} [{OUTPUT_SET_FILE | -}]]

rwsetbuild --help

rwsetbuild --version

DESCRIPTION

rwsetbuild creates a binary IPset file from textual input. The IPset is written to the second command line argument if it has been specified; otherwise the IPset is written to the standard output if the standard output is not a terminal. rwsetbuild will not overwrite an existing file unless the SILK_CLOBBER environment variable is set. The textual input is read from the first command line argument if it has been specified; otherwise the text is read from the standard input if the standard input is not a terminal. A input file name of stdin or - means the standard input; an output file name of stdout or - means the standard output. rwsetbuild will read textual IPs from the terminal if the standard input is explicitly given as the input. rwsetbuild exits with an error if the input file cannot be read or the output file cannot be written.

Comments are ignored in the input file; they begin with the '#' symbol and continue to the end of the line. Whitespace and blank lines are also ignored. Otherwise, a line should contain a single IP addresses unless the --ip-ranges switch is specified, in which case a line may contain two IP addresses separated by the user-specified delimiter, which defaults to hyphen (-).

rwsetbuild supports IPv4 addresses and, when SiLK has been built with IPv6 support, IPv6 addresses. When the input contains a mixture of IPv4 and IPv6 addresses, the IPv4 addresses are mapped into the ::ffff:0:0/96 block of IPv6. When writing the IPset, rwsetbuild converts the output to IPv4 if all IPv6 addresses are in the ::ffff:0:0/96 block. rwsetbuild does not allow the input to contain both integer values and IPv6 addresses.

Each IP address must be expressed in one of these formats:

If an IP address cannot be parsed, rwsetbuild exits with an error.

Use rwsetcat(1) to see the contents of an IPset file. To check for a specific IP address in an IPset, use rwsetmember(1). rwsettool(1) manipulates IPset files. To build an IPset file from SiLK Flow data, use rwset(1).

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--ip-ranges
--ip-ranges=DELIM

Allow lines of the the input file to contain a pair of IP addresses, separated by DELIM, that create an IP address range, and do not allow the IP Wildcard syntax. A line may also contain a single IP address or a 32-bit integer; these lines may have a CIDR designation. CIDR designations are not supported on lines that contain a pair of IP addresses. If DELIM is not specified, hyphen ('-') is used as the delimiter. When DELIM is a whitespace character, any amount of whitespace may surround and separate the two IP addresses. Since '#' is used to denote comments and newline is used to denote records, neither is a valid delimiter character.

--record-version=VERSION

Specify the format of the IPset records that are written to the output. VERSION may be 2, 3, 4, 5 or the special value 0. When the switch is not provided, the SILK_IPSET_RECORD_VERSION environment variable is checked for a version. The default version is 0.

0

Use the default version for an IPv4 IPset and an IPv6 IPset. Use the --help switch to see the versions used for your SiLK installation.

2

Create a file that may hold only IPv4 addresses and is readable by all versions of SiLK.

3

Create a file that may hold IPv4 or IPv6 addresses and is readable by SiLK 3.0 and later.

4

Create a file that may hold IPv4 or IPv6 addresses and is readable by SiLK 3.7 and later. These files are more compact that version 3 and often more compact than version 2.

5

Create a file that may hold only IPv6 addresses and is readable by SiLK 3.14 and later. When this version is specified, IPsets containing only IPv4 addresses are written in version 4. These files are usually more compact that version 4.

--invocation-strip

Do not record any command line history; that is, do not record the current command line invocation in the output file.

--note-add=TEXT

Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME

Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD

Specify the compression library to use when writing output files. If this switch is not given, the value in the SILK_COMPRESSION_METHOD environment variable is used if the value names an available compression method. When no compression method is specified, output to the standard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

none

Do not compress the output using an external library.

zlib

Use the zlib(3) library for compressing the output, and always compress the output regardless of the destination. Using zlib produces the smallest output files at the cost of speed.

lzo1x

Use the lzo1x algorithm from the LZO real time compression library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead.

snappy

Use the snappy library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead. Since SiLK 3.13.0.

best

Use lzo1x if available, otherwise use snappy if available, otherwise use zlib if available. Only compress the output when writing to a file.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLE

In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line.

Reading from a file:

$ echo 10.x.x.x > ten.txt
$ rwsetbuild ten.txt ten.set

$ echo 10.0.0.0/8 > ten.txt
$ rwsetbuild ten.txt ten.set

$ echo 10.0.0.0-10.255.255.255 > ten.txt
$ rwsetbuild --ip-ranges ten.txt ten.set

$ echo '167772160,184549375' > ten.txt
$ rwsetbuild --ip-ranges=, ten.txt ten.set

Reading from the standard input:

$ echo 192.168.x.x | rwsetbuild stdin private.set

Example input to rwsetbuild:

# A single address
10.1.2.4
# Two addresses in the same subnet
10.1.2.4,5
# The same two addresses
10.1.2.4/31
# The same two addresses
167838212/31
# A whole subnet
10.1.2.0-255
# The same whole subnet
10.1.2.x
# The same whole subnet yet again
10.1.2.0/24
# All RFC1918 space
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
# All RFC1918 space
10.x.x.x
172.16-20,21,22-31.x.x
192.168.x.x
# All RFC1918 space
167772160/8
2886729728/12
3232235520/16
# Everything ending in 255
x.x.x.255
# All addresses that end in 1-10
x.x.x.1-10

ENVIRONMENT

SILK_IPSET_RECORD_VERSION

This environment variable is used as the value for the --record-version when that switch is not provided. Since SiLK 3.7.0.

SILK_CLOBBER

The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SILK_COMPRESSION_METHOD

This environment variable is used as the value for --compression-method when that switch is not provided. Since SiLK 3.13.0.

SEE ALSO

rwset(1), rwsetcat(1), rwsetmember(1), rwsettool(1), rwfileinfo(1), silk(7), zlib(3)

NOTES

Prior to SiLK 3.0, an IPset file could not contain IPv6 addresses and the record version was 2. The --record-version switch was added in SiLK 3.0 and its default was 3. In SiLK 3.6, an argument of 0 was allowed and made the default. Version 4 was added in SiLK 3.7 as was support for the SILK_IPSET_RECORD_VERSION environment variable. Version 5 was added in SiLK 3.14.