super_mediator

super_mediator SSL Certificate De-duplication Configuration Tutorial

• Overview
• Configuration File
• Example TEXT Exporter
• De-duplicating IPs and Certificates
• Using MySQL

Overview

SSL/TLS are cryptographic protocols that add encryption and entity authentication to Internet communications. These protocols are commonly used with HTTP, aka HTTPS, to secure web traffic. Servers send certificates to clients to authenticate themselves for TLS sessions. Certificates are issued to administrators by Certificate Authorities (CAs). The role of the CA is to verify that the certificate holder is in control of the domain name in question and then mathematically bind particular encryption parameters to that domain name.

TLS connections start with an unencrypted handshake. The server presents its authentication credentials in the form of a certificate. The certificate contains public information about the server, such as its advertised domain name, public key, company, and information about the CA that issued the certificate. The certificate also contains certain characteristics that should prove the validity and authenticity of the certificate.

Computer Network Defense (CND) analysts often want to identify certificates used by malware command and control servers, certificates with weak cryptographic parameters to determine at-risk connections, and forged certificates. Collecting certificate information traversing the network can assist analysts in comparing collected certificate attributes to known forged or compromised certificate attributes.

Collecting SSL/TLS certificates can be very cumbersome, as they are rather large and you can end up with a lot of duplicate data, putting a strain on storage resources. super_mediator can help with de-duplicating SSL certificates by writing certificate data captured by yaf once and caching the certificate's serial number and issuer name in memory until either the MAX_HIT_COUNT has been met or the unique pair has not been seen in the FLUSH_TIMEOUT period. This tutorial will provide examples of super_mediator SSL de-duplication configurations.

Configuration File

The most important part of the configuration file for SSL de-duplication is the SSL_CONFIG block. An SSL_CONFIG block must be associated with a single EXPORTER. SSL de-duplication can be configured for any type of EXPORTER (TEXT, JSON, IPFIX, TCP, etc.). If SSL de-duplication is enabled, super_mediator exports two unique types of records: "certificate" records, and "dedup" records. "Certificate" records contain all of the data that yaf captures for an X.509 Certificate. The full list can be found in the yafdpi man page. The fields that super_mediator exports in "Certificate" records is configurable in the SSL_CONFIG block with the ISSUER, SUBJECT, OTHER, and EXTENSION keywords. The argument provided with each one of these keywords is a bracketed-list of object identifier values. Some common object IDs for certificate ISSUER and SUBJECT are listed in the following table. By default, super_mediator will export all issuer and subject fields.

The OTHER list can contain any one of the following information element IDs:

The EXTENSION list can contain any of the following object identfier values. By default, super_mediator will not write any EXTENSION objects. These must be explicitly identified in the SSL_CONFIG block.

To force super_mediator to write all SSL certificate characteristics captured by yaf, use the following configuration:

SSL_CONFIG "exportername"
   ISSUER [*]
   SUBJECT [*]
   OTHER [*]
   EXTENSIONS [*]
SSL_CONFIG END

The "certificate" record will have the following CSV format:

serial number | issuer name | first_seen | obj id | ISE | cert_no | data

There may be more than one of the same object IDs present for a ssl certificate if the object is a list (e.g. issuerAltName).

The IPFIX template for the "certificate" record is as follows:

--- template record ---
header:
    tid: 51723 (0xca0b) field count:    11  scope:     0
fields:
     ent:     0  id:   292  type: stl   length: 65535  subTemplateList
     ent:     0  id:   292  type: stl   length: 65535  subTemplateList
     ent:     0  id:   292  type: stl   length: 65535  subTemplateList
     ent:  6871  id:   190  type: octet length: 65535  sslCertSignature
     ent:  6871  id:   244  type: string    length: 65535  sslCertSerialNumber
     ent:  6871  id:   247  type: string    length: 65535  sslCertValidityNotBefore
     ent:  6871  id:   248  type: string    length: 65535  sslCertValidityNotAfter
     ent:  6871  id:   249  type: octet length: 65535  sslPublicKeyAlgorithm
 ent:  6871  id:   250  type: uint16    length:     2  sslPublicKeyLength
     ent:  6871  id:   189  type: uint8 length:     1  sslCertVersion
 ent:     0  id:   210  type: octet length:     5  paddingOctets

The other type of record super_mediator will export is a "dedup" record. A "dedup" record is a short record that simply provides the first and last time a certificate was seen, the unique identifier for a certificate (serial number, issuer name), and the number of times it was seen within that time period. The CSV format is as follows:

first_seen | last_seen | serial number | count | issuer name

The "dedup" IPFIX template is as follows:

--- template record ---
header:
    tid: 55983 (0xdaaf) field count:     5  scope:     0
fields:
     ent:     0  id:   152  type: milsec    length:     8  flowStartMilliseconds
     ent:     0  id:   153  type: milsec    length:     8  flowEndMilliseconds
     ent:  6871  id:   929  type: octet length:     8  observedDataTotalCount
     ent:  6871  id:   244  type: string    length: 65535  sslCertSerialNumber
     ent:  6871  id:   196  type: octet length: 65535  sslCertIssuerCommonName

To enable SSL Certificate de-duplication either the SSL_DEDUP_ONLY keyword must be present in the EXPORTER block OR the SSL_DEDUP keyword must be present in the SSL_CONFIG block:

EXPORTER TEXT "name"
    PATH "/data/ssl/sslcerts.txt"
    SSL_DEDUP_ONLY
EXPORTER END

OR

SSL_CONFIG "exportername"
    SSL_DEDUP
SSL_CONFIG END

By default, super_mediator will write both types of records (certificate and dedup) to the filename given to "PATH" in the EXPORTER block. However, if the CERT_FILE keyword is present in the SSL_CONFIG block, super_mediator will write "certificate" records to the filename given to CERT_FILE. This file will rotate and/or lock using the same configuration settings given in the EXPORTER block associated with the SSL_CONFIG block. The CERT_FILE keyword is ignored for all exporter types other than TEXT.

Example TEXT Exporter

The following is an example configuration file that enables SSL Certificate de-duplication and exports all characteristics of an SSL certificate to the rotating file prefix "/data/ssl/sslcerts".

EXPORTER TEXT "e1"
  PATH "/data/ssl/certs_dedup"
  SSL_DEDUP_ONLY
  ROTATE 300
  LOCK
EXPORTER END

SSL_CONFIG "e1"
  ISSUER [*]
  SUBJECT [*]
  OTHER [*]
  EXTENSIONS [*]
  CERT_FILE "/data/ssl/certs"
  MAX_HIT_COUNT 25000
  FLUSH_TIME 3600
SSL_CONFIG END

The following is an example of the data that the above configuration produces:

$ cat /data/ssl/certs_dedup.20150408192918.txt
2015-04-08 19:14:29.556|2015-04-08 19:28:57.914|0x008620ad42a17aea20|4|Go Daddy Secure Certificate Authority - G2
2015-04-08 19:29:14.389|2015-04-08 19:29:14.389|0x01fe4a238b2e7ce313c506df7fd7ca4e|4|DigiCert SHA2 Secure Server CA
2015-04-08 19:16:20.469|2015-04-08 19:29:14.389|0x01fda3eb6eca75c888438b724bcfbc91|38|DigiCert Global Root CA
2015-04-08 19:29:14.391|2015-04-08 19:29:14.391|0x040bd4f82588c5|4|Go Daddy Secure Certificate Authority - G2
2015-04-08 19:17:14.651|2015-04-08 19:29:14.404|0x5cc17e9b9b4933fe|10|Google Internet Authority G2


$ cat /data/ssl/certs.20150408191312.txt

0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|6|I|0|US
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|8|I|0|Arizona
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|7|I|0|Scottsdale
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|10|I|0|GoDaddy.com, Inc.
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|11|I|0|http://certs.godaddy.com/repository/
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|3|I|0|Go Daddy Secure Certificate Authority - G2
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|11|S|0|Domain Control Validated
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|3|S|0|load.exelator.com
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|15|E|0|03 02 05 a0
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|31|E|0|http://crl.godaddy.com/gdig2s1-87.crl
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|32|E|0|60 86 48 01 86 fd 6d 01 07 17 01
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|32|E|0|http://certificates.godaddy.com/repository/
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|17|E|0|load.exelator.com
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|17|E|0|www.load.exelator.com
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|17|E|0|meta.exelator.com
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|2015-04-08 19:14:29.556|17|E|0|loadm.exelator.com

As you can see from the above example data the Go Daddy certificate with serial number 0x008620ad42a17aea20 was seen four times within a 14 minute time period.

De-duplicating IPs and Certificates

Now that the SSL certificates have been collected and de-duplicated, it might be necessary to determine which IP address on the network received a particular certificate. The SSL certificate de-duplication feature can be combined with the DEDUP_CONFIG block to determine which IP used a particular certificate.

EXPORTER TEXT "ssl_ip_dedup"
    PATH "/data/ssl/"
    ROTATE 300
    LOCK
EXPORTER END

DEDUP_CONFIG "ssl_ip_dedup"
    PREFIX ssl_ip_dedup [244]
DEDUP_CONFIG END

Adding the above DEDUP_CONFIG block and EXPORTER block to the above configuration will configure super_mediator to de-duplicate unique IP address, certificate chain tuples. super_mediator will store in memory every unique serial number, issuer name tuple for a certificate. Furthermore, it will maintain information about the certificate chain an IP address receives in the TLS handshake. super_mediator will export the IP and first two certificate tuples when MAX_HIT_COUNT or FLUSH_TIMEOUT period has been met. The CSV format for these records is as follows:

first_seen | last_seen | IP | flowkeyhash | count | serial1 | issuer1 | serial2 | issuer2

Typically, the first certificate is an end-user certificate that cannot be trusted as it is not embedded in the web browser or operating system. The second certificate is the intermediate or root certificate that may be explicitly trusted if it is issued by a CA that is embedded in the web browser or OS.

The serial number, issuer name pair will let the analyst pivot between the "certificate records" and the "IP dedup" records to determine when an IP saw a particular certificate and the particular characteristics of that certificate.

The above additions to the configuration file will produce the following data:

$ cat /data/ssl/ssl_ip_dedup.20150408192918.txt
2015-04-08 19:14:29.556|2015-04-08 19:14:29.680|10.27.33.66|2154341740|2|\
0x008620ad42a17aea20|Go Daddy Secure Certificate Authority - G2|\
0x07|Go Daddy Root Certificate Authority - G2
2015-04-08 19:15:24.633|2015-04-08 19:17:14.722|10.27.33.66|3741584532|6|\
0x0754|GeoTrust SSL CA - G4|0x023a79|GeoTrust Global CA
2015-04-08 19:14:54.239|2015-04-08 19:17:14.724|10.27.33.66|3730640023|10|\
0x0765|GeoTrust SSL CA - G4|0x023a79|GeoTrust Global CA
2015-04-08 19:18:10.483|2015-04-08 19:19:04.602|10.27.33.66|395876596|6|\
0x516f2670a7991b70|Google Internet Authority G2|0x023a76|GeoTrust Global CA

Using MySQL

The data produced by super_mediator can easily be imported into a MySQL database. super_table_creator will create the appropriate tables for the data produced by the above configuration.

$ super_table_creator -n root -p password -d ssl_database --ssl-certs
certs table successfully created
certs_dedup table successfully created

$ super_table_creator -n root -p password -d ssl_database --ssl-dedup
Ignoring Warning: Database ssl_database 1007: Can't create database 'ssl_database'; database exists
ssl_ip_dedup table successfully created

The Warning produced by super_table_creator just means that the database already exists. super_table_creator tries to create the database every time it is run. If it already exists, this error is ignored.

Now the data can be easily imported using the mysqlimport tool:

$ mysqlimport -u root -p --fields-terminated-by="|" ssl_database /data/ssl/certs_dedup.20150408192918.txt
Enter password:
ssl_database.certs_dedup: Records: 440  Deleted: 0  Skipped: 0  Warnings: 0

$ mysqlimport -u root -p --fields-terminated-by="|" ssl_database /data/ssl/ssl_ip_dedup.20150408192918.txt
Enter password:
ssl_database.ssl_ip_dedup: Records: 338  Deleted: 0  Skipped: 0  Warnings: 0

$ mysqlimport -u root -p --fields-terminated-by="|" ssl_database /data/ssl/certs.20150408192355.txt
Enter password:
ssl_database.certs: Records: 1540  Deleted: 0  Skipped: 0  Warnings: 0