This prototype deployment uses a GetFile NiFi processor to bring in SiLK binary files from a local directory available to NiFi, but you could use any NiFi-supported means to ingest these files, such as via S3 (ListS3 + FetchS3Object).

Scheduled Lambda functions remove expired ASN info items, update and remove CSP info items from Elasticsearch.

Per protocol/dataset, baseline traffic in S3 is read by nightly baseline regeneration script (from dedicated per-protocol EC2 Auto Scaling instance), new baseline metrics are calculated and loaded to DynamoDB. Old protocol traffic from S3 is removed. This enables a rolling baseline (generally the last 90 days of protocol traffic).

Expired items in certain DynamoDB tables are automatically removed by utilizing the DynamoDB Time-To-Live (TTL) feature, which is a free AWS-managed background process which automatically scans and removes expired items based on a set TTL attribute. This enables simple updating of DynamoDB items without having to worry about the cost of tracking, scanning, and removing old items ourselves.