Skip to content

IPFIX flow exporter with DPDK support capable of bi-directional flows, per-packet-information statistics, and extensibility via processing plugins (e.g., for application layer parsers).

License

Notifications You must be signed in to change notification settings

CESNET/ipfixprobe

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation


The ipfixprobe is a high-performance, modular flow exporter that processes packets into bidirectional flows and exports them via a selected output plugin. It supports a wide range of application-layer protocol parsers, including TLS, QUIC, HTTP, DNS, and many others. These protocol-specific extensions can be enabled via process plugins in the configuration.

Need more details? Check out our documentation for a full list of supported protocols and usage examples.

Coverity Scan GitHub top language

✨ Key Features

  • Modular input–parser–output architecture
  • High-speed processing (DPDK, multi-threaded, NUMA-aware)
  • Built-in protocol parsers: TLS, QUIC, HTTP, DNS, …
  • Bidirectional flow (biflow) support
  • Real-time telemetry and statistics

πŸ“¦ Installation

If you are running a RHEL system or one of its derivatives (e.g. Oracle Linux, Rocky Linux, CentOS Stream), the easiest way to install ipfixprobe is from our copr repository.

$ dnf install dnf-plugins-core # Extra step necessary on some systems
$ dnf copr enable @CESNET/ipfixprobe
$ dnf install ipfixprobe

This installs the main ipfixprobe binary along with core functionality.

πŸ—ƒοΈ Available Packages

The Copr repository provides modular RPM packages, so you can install only what you need. The following packages are available and can be installed individually as needed:

Package Name Description
ipfixprobe Core binary with common process/output plugins.
ipfixprobe-msec Core binary with common process/output plugins. Uses millisecond timestamps (compatible with Flowmon collector)
ipfixprobe-input-pcap Input plugin for PCAP files and live capture
ipfixprobe-input-dpdk High-speed input plugin using DPDK
ipfixprobe-input-nfb Input plugin for CESNET NFB/NDP cards
ipfixprobe-process-experimental Extra (possibly unstable) process plugins

For other systems, follow the build instructions below.

πŸ› οΈ Build

You can build ipfixprobe from source using standard CMake. This lets you customize the build by enabling optional plugins and features as needed.

Note: Some plugins may require additional dependencies beyond the basic requirements.

RHEL/CentOS:

🧰 Requirements

$ dnf install epel-release git make cmake gcc-c++ rpm-build
$ dnf install libunwind-devel lz4-devel openssl-devel fuse3-devel

# for RHEL 8/9
$ dnf install gcc-toolset-14-libatomic-devel

# for RHEL 10+
$ dnf install libatomic

Debian/Ubuntu:

🧰 Requirements

$ apt install git make cmake g++ pkg-config rpm
$ apt install libunwind-dev liblz4-dev libssl-dev libfuse3-dev libatomic1

🧱 Build steps

git clone https://github.com/CESNET/ipfixprobe.git
cd ipfixprobe
mkdir build && cd build
cmake ..
make -j$(nproc)
# make install

βš™οΈ Optional build flags

You can enable or disable optional plugins and features via CMake flags:

Flag Default Description
-DENABLE_MILLISECONDS_TIMESTAMP=ON OFF Use millisecond precision timestamps (for Flowmon compatibility)
-DENABLE_INPUT_PCAP=ON OFF Enable PCAP input plugin (live & file) (requires libpcap)
-DENABLE_INPUT_DPDK=ON OFF Enable high-speed DPDK input plugin (requires dpdk-devel)
-DENABLE_INPUT_NFB=ON OFF Enable input plugin for CESNET NFB/NDP cards (requires netcope-common)
-DENABLE_PROCESS_EXPERIMENTAL=ON OFF Enable experimental process plugins
-DENABLE_NEMEA=ON OFF Enable support for NEMEA modules (requires nemea-framework-devel )

Run the command to view all available build options:

cmake -LAH

Example

To build with DPDK and PCAP input support, and install to /usr:

cmake .. \
  -DCMAKE_INSTALL_PREFIX=/usr \
  -DENABLE_INPUT_PCAP=ON \
  -DENABLE_INPUT_DPDK=ON

🧩 Available Plugins

Input Plugins

List of input plugins with estimated performance and configuration complexity.

Plugin Max Throughput Usage Complexity Description
pcap_live ~1 Gbps Easy captures packets from a live network interface
pcap_file ~1 Gbps Easy reads packets from an offline PCAP file
raw ~1 Gbps Easy captures packets using a raw socket
ndp 400 Gbps Medium uses CESNET NFB/NDP hardware for packet input
dpdk 400 Gbps Complex receives packets via high-performance DPDK
dpdk-ring 400 Gbps Complex receives packets from a shared DPDK memory ring

Process Plugins

These plugins extract protocol-specific or behavioral information from packets and enrich flow records with metadata.

Plugin Description
basic extracts basic L3/L4 flow fields (IPs, ports, protocol)
icmp extracts ICMP type/code and related metadata
http extracts HTTP methods, hosts, URIs, status codes
tls extracts TLS handshake info (SNI, version, JA3, etc.)
ovpn extracts metadata from OpenVPN tunnels
wg parses WireGuard handshake and endpoint metadata
quic parses QUIC protocol including SNI, versions, ALPN
basicplus adds common L3/L4 flow fields (e.g., ports, IPs, TCP flags)
bstats basic flow statistics (packet/byte counters, duration, ...)
dns extracts DNS queries, responses, and domains
dnssd parses DNS Service Discovery (mDNS) traffic
flowHash extracts a flow hash
idpContent parses IDP content in flows
mpls extracts MPLS labels and encapsulation metadata
mqtt parses MQTT protocol traffic (IoT messaging)
netbios extracts NetBIOS session and name service info
nettisa parses NETTISA related metadata (experimental)
ntp extracts NTP timestamps and server info
osquery parses osquery-generated data streams
passiveDns generates passive DNS entries from observed DNS traffic
phists parses phishing-related signatures (heuristic)
pstats advanced packet statistics (e.g., inter-packet gaps)
rtsp extracts RTSP stream metadata
sip parses SIP call setup, headers, and codecs
smtp extracts SMTP envelope data (from, to, subject, etc.)
ssaDetector performs simple anomaly detection based on traffic patterns
ssdp parses SSDP (UPnP discovery) protocol
vlan extracts VLAN IDs and QinQ encapsulation

Output Plugins

These plugins export flow records to various formats and external systems.

Plugin Description
ipfix exports flow records in IPFIX format to a remote collector (UDP/TCP)
text writes flow records in human-readable text to a file or stdout
unirec exports flow records using the UniRec format for NEMEA/TRAP ecosystem

πŸ“˜ Usage

The ipfixprobe processing pipeline can be launched directly or via the ipfixprobed wrapper β€” a lightweight runtime designed to simplify plugin configuration using YAML files. This method is preferred for production deployments due to its clarity and flexibility.

βœ… Recommended (YAML-based configuration)

/usr/bin/ipfixprobed <CONFIG_NAME> [LIBRARY_PATH]
  • <CONFIG_NAME> – Name of the YAML configuration file (without the .conf extension). The full path must be /etc/ipfixprobe/<CONFIG_NAME>.conf.

  • [LIBRARY_PATH] – (Optional) Path to the ipfixprobe plugin library directory. If not provided, the default /usr/lib64/ipfixprobe is used.

Example:

/usr/bin/ipfixprobed example

This will run ipfixprobe using the YAML configuration from /etc/ipfixprobe/example.conf.

πŸ‘‰ See the full YAML configuration reference here: https://github.com/CESNET/ipfixprobe/blob/master/init/link0.conf.example

⚠️ Legacy usage (CLI parameters)

You may also run the processing pipeline using the ipfixprobe binary directly, with CLI parameters. This method is not recommended for production use, as it lacks flexibility and clarity.

/usr/bin/ipfixprobe [OPTIONS]

Available options:

  • -i ARGS Activate input plugin (-h input for help)
  • -s ARGS Activate storage plugin (-h storage for help)
  • -o ARGS Activate output plugin (-h output for help)
  • -p ARGS Activate processing plugin (-h process for help)
  • -q SIZE Size of queue between input and storage plugins
  • -b SIZE Size of input queue packet block
  • -Q SIZE Size of queue between storage and output plugins
  • -B SIZE Size of packet buffer
  • -f NUM Export max flows per second
  • -c SIZE Quit after number of packets are processed on each interface
  • -P FILE Create a PID file
  • -t PATH Mount point of AppFs telemetry directory
  • -d Run as a standalone process
  • -h [PLUGIN] Print help text. Supported help for input, storage, output and process plugins
  • -V Show version and exit

πŸ“– Examples

Below are practical examples showcasing common plugin configurations using both CLI and YAML formats.

πŸ”Ή Basic Interface Capture

Capture network traffic from the wlp2s0 interface using a raw socket. Flow records are printed in plain-text format to the console.

πŸ§ͺ Command-line usage:

/usr/bin/ipfixprobe -i 'raw;ifc=wlp2s0' -o 'text'

πŸ“„ Equivalent YAML configuration:

input_plugin:
  raw:
    interface: 'wlp2s0'

output_plugin:
  text: {}

πŸ”Ή PCAP File β†’ IPFIX Collector

Process packets from a .pcap file and export flows via IPFIX over UDP

πŸ§ͺ Command-line usage:

/usr/bin/ipfixprobe -i 'pcap;file=/data/capture.pcap' -o 'ipfix;udp;host=collector.example.com;port=4739'

πŸ“„ Equivalent YAML configuration:

input_plugin:
  pcap_file:
    file: '/data/capture.pcap'
output_plugin:
  ipfix:
    collector:
      host: collector.example.com
      port: 4739
    protocol:
      udp: {}

πŸ”ΉLive Capture with Cache Configuration and Telemetry

Capture from eth0 interface using libpcap, split biflows into uniflows and use active/inactive timeouts, print flows to console. Telemetry data are exposed via the appFs library in /var/run/ipfixprobe directory.

πŸ§ͺ Command-line usage:

/usr/bin/ipfixprobe -i 'pcap;ifc=eth0' -s 'cache;split;active=300;inactive=60' -o 'text' -t /var/run/ipfixprobe

πŸ“„ Equivalent YAML configuration:

input_plugin:
  pcap_live:
    interface: 'eth0'

storage:
  cache: {}
  timeouts:
    active: 300
    inactive: 60
  split_biflow: true

output_plugin:
  text: {}

telemetry:
  appfs:
    enabled: true
    mount_point: /var/run/ipfixprobe

πŸ”Ή High-speed DPDK Capture with HTTP, TLS, and QUIC Processing

Capture packets using DPDK from port 0 with 2 queues bound to a specific PCI device (0000:17:00.0), enable HTTP, TLS, and QUIC process plugins, and export flows via IPFIX to a local collector at 127.0.0.1.

πŸ§ͺ Command-line usage:

/usr/bin/ipfixprobe -i "dpdk;p=0;q=2;e=-a 0000:17:00.0" -i dpdk -p http -p tls -p quic -o "ipfix;h=127.0.0.1"

πŸ“„ Equivalent YAML configuration:

input_plugin:
  dpdk:
    allowed_nics: "0000:17:00.0"
    rx_queues: 2

process_plugins:
  - http
  - tls
  - quic

output_plugin:
  ipfix:
    collector:
      host: '127.0.0.1'
      port: 4739
    protocol:
      udp: {}

πŸ“Š Telemetry

πŸ§ͺ Testing & Validation

🧰 FAQ

About

IPFIX flow exporter with DPDK support capable of bi-directional flows, per-packet-information statistics, and extensibility via processing plugins (e.g., for application layer parsers).

Topics

Resources

License

Stars

Watchers

Forks

Contributors 19