Sentinel is a packet analyzer and, eventually, a rule-based intrusion detection system, written in Python from scratch. the protocol parsers are hand-written with struct: no Scapy, no dpkt, no pyshark. it reads pcap capture files and prints one line per packet, tcpdump style. it has no runtime dependencies.
this post covers all seven stages. stage 1 is the pcap reader and writer, and parsers for Ethernet (with VLAN tags), ARP, IPv4, IPv6, TCP, UDP and ICMP. stage 2 is the application layer: DNS, plaintext HTTP and the TLS ClientHello (server name, versions, cipher list). stage 3 is flow tracking and TCP stream reassembly. stage 4 is a filter language. stage 5 is the intrusion detection part: four detectors (port scans, SYN floods, ARP spoofing, DNS tunneling), rules, and JSON alerts. stage 6 is live capture on Linux: the same pipeline, fed from a network interface. stage 7 measures it (packets per second), adds a fuzzer, and sets up CI. the parts that need other machines (Linux, GitHub) could only be tested there, and I say what that showed.
I worked the same way as with Impulse: one stage at a time, and the next one does not start until the tests pass and ruff and strict mypy are clean. most of this post is about what the parsers do with bad input, and how I tried to prove they are right.
##the rule I set first
a packet analyzer reads bytes that anyone on the network can send. so the first rule is that a parser never raises on bad input: not on a truncated packet, not on a header that lies about its own length, not on random bytes. every parser returns a small dataclass instead, and they all share the same three fields.
@dataclass(frozen=True, slots=True, kw_only=True)
class Layer:
"""Base of every parse result. Parsers never raise.
`error` set: the header could not be parsed,
all other fields hold defaults and mean nothing.
`anomalies`: the header parsed, but something is off
(bad checksum, inconsistent length...).
`payload`: the bytes after this header, trimmed to
the length the header declares.
"""
payload: bytes = b""
error: str | None = None
anomalies: tuple[str, ...] = ()two kinds of trouble are kept apart on purpose. an error means the header cannot be trusted at all: too short, the wrong IP version, a TCP data offset below 5. every other field is then a default and means nothing. an anomaly means the header parsed but something is off: a bad checksum, or a packet cut short by the capture's snap length. the packet is kept and the oddity is reported, because the IDS in stage 5 will want to see it, not lose it.
# parse_tcp(bytes(12))
Tcp(error='truncated tcp header: 12 of 20 bytes',
anomalies=(),
src_port=0, dst_port=0, seq=0, flags=0,
# ...every other field is a default)# parse_tcp(segment, src=a, dst=b), one bit flipped
Tcp(error=None,
anomalies=('bad tcp checksum',),
src_port=1, dst_port=2, seq=3, flags=2,
# ...every field parsed)checksums need care. TCP and UDP checksums include a pseudo-header with the IP addresses, so those parsers only verify when they are given the addresses, and the decoder only gives them when the whole segment was captured. without that rule, a packet cut short by the snap length would report a bad checksum that is not real.
def _decode_l4(
proto: int,
payload: bytes,
src: IPv4Address | IPv6Address,
dst: IPv4Address | IPv6Address,
complete: bool,
) -> Layer | None:
# Checksums are only checked when the whole L4 message was captured.
s, d = (src, dst) if complete else (None, None)
if proto == PROTO_TCP:
return parse_tcp(payload, src=s, dst=d)
if proto == PROTO_UDP:
return parse_udp(payload, src=s, dst=d)
if proto == PROTO_ICMP and isinstance(src, IPv4Address):
return parse_icmp(payload, verify_checksum=complete)
return Nonethere is one known false positive I have not solved. captures taken on the sending machine often show bad TCP checksums, because the network card fills them in later (checksum offload). those packets are reported as anomalies, not errors, so a detector can choose to ignore them.
##what it looks like
tools/gen_pcap.py builds a small pcap with the project's own writer, covering every protocol above. reading it back:
$ python -m sentinel read demo.pcap 2023-11-14 22:13:20.000000 ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 28 2023-11-14 22:13:20.001000 ARP, Reply 10.0.0.2 is-at 02:00:00:00:00:02, length 28 2023-11-14 22:13:20.002000 IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 1, seq 1, length 40 2023-11-14 22:13:20.003000 IP 10.0.0.2 > 10.0.0.1: ICMP echo reply, id 1, seq 1, length 40 2023-11-14 22:13:20.004000 IP 10.0.0.2 > 10.0.0.1: ICMP destination unreachable, code 3, length 36 2023-11-14 22:13:20.005000 IP 10.0.0.1.40000 > 10.0.0.2.80: Flags [S], seq 1000, win 64240, options [mss 1460,sackOK,TS val 1000 ecr 0,nop,wscale 7], length 0 2023-11-14 22:13:20.006000 IP 10.0.0.2.80 > 10.0.0.1.40000: Flags [S.], seq 5000, ack 1001, win 64240, options [mss 1460,sackOK,TS val 1000 ecr 0,nop,wscale 7], length 0 2023-11-14 22:13:20.007000 IP 10.0.0.1.40000 > 10.0.0.2.80: Flags [.], seq 1001, ack 5001, win 64240, length 0 2023-11-14 22:13:20.008000 IP 10.0.0.1.40000 > 10.0.0.2.80: Flags [P.], seq 1001, ack 5001, win 64240, length 38: HTTP: GET / HTTP/1.1, host example.test 2023-11-14 22:13:20.009000 IP 10.0.0.1.40000 > 10.0.0.2.80: Flags [F.], seq 1039, ack 5001, win 64240, length 0 2023-11-14 22:13:20.010000 IP 10.0.0.1.53000 > 10.0.0.2.53: UDP, length 29: DNS query 48879, A? example.com 2023-11-14 22:13:20.011000 vlan 100, IP 10.0.0.1.53001 > 10.0.0.2.53: UDP, length 29: DNS query 48879, A? example.com 2023-11-14 22:13:20.012000 IP6 2001:db8::1.41000 > 2001:db8::2.443: Flags [S], seq 1, win 65535, length 0 2023-11-14 22:13:20.013000 IP6 2001:db8::1.53002 > 2001:db8::2.53: UDP, length 29: DNS query 48879, A? example.com 2023-11-14 22:13:20.014000 IP 10.0.0.1.53003 > 10.0.0.2.53: UDP, length 33: DNS query 4660, A? www.example.com 2023-11-14 22:13:20.015000 IP 10.0.0.2.53 > 10.0.0.1.53003: UDP, length 63: DNS response 4660 NOERROR, A? www.example.com, answers [CNAME example.com, A 192.0.2.1] 2023-11-14 22:13:20.016000 IP 10.0.0.1.42000 > 10.0.0.2.53: Flags [P.], seq 1, ack 1, win 64240, length 31: DNS query 17185, AAAA? example.com 2023-11-14 22:13:20.017000 IP 10.0.0.2.80 > 10.0.0.1.40000: Flags [P.], seq 5001, ack 1039, win 64240, length 77: HTTP: HTTP/1.1 200 OK 2023-11-14 22:13:20.018000 IP 10.0.0.1.43000 > 10.0.0.2.443: Flags [P.], seq 1, ack 1, win 64240, length 161: TLS ClientHello, sni example.com, versions [TLS 1.3, TLS 1.2], ciphers (15) [TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, +12 more]
19 packets: an ARP request and reply, three ICMP messages, a TCP connection from SYN to FIN with an HTTP request in it, DNS over UDP (also inside a VLAN tag and over IPv6) and over TCP, a CNAME answer, an HTTP response and a TLS ClientHello. timestamps are UTC, so the output does not depend on the machine's time zone.
##anatomy of a packet
the SYN from that capture is 74 bytes: 14 of Ethernet, 20 of IPv4, 20 of fixed TCP header and 20 bytes of TCP options. each parser reads its own header, checks every length before it reads, and passes the remaining bytes on. decode() returns the layers as a tuple and stops at the first layer that has an error.
decode(frame) returns (Ethernet, IPv4, Tcp)
every byte is copied from the generated packet. the checksums read "valid" because the parsers verified them.
the options are the part to be careful with. each one is a kind, a length and data, and the length comes from the packet. a length below 2, or one that runs past the end of the options, ends the walk and is reported as an anomaly. it never raises.
##the capture file
a classic pcap file starts with a 24-byte global header, and every packet has a 16-byte record header in front of it. the first four bytes, the magic number, say two things at once: the byte order the file was written in, and whether timestamps are in microseconds or nanoseconds. so there are four variants, and the reader and the writer handle all of them.
| first 4 bytes on disk | byte order | timestamp unit |
|---|---|---|
d4 c3 b2 a1 | little-endian | microseconds |
a1 b2 c3 d4 | big-endian | microseconds |
4d 3c b2 a1 | little-endian | nanoseconds |
a1 b2 3c 4d | big-endian | nanoseconds |
inside the program a timestamp is an integer number of nanoseconds. floats would round and break exact round trips. a microsecond file drops the last three digits on write, and a test pins that down.
the reader streams from the file instead of loading it. the captured-length field in each record comes from the file, so it is capped at 262,144 bytes (libpcap's own limit) before anything is read, and a corrupt value cannot make it try to read gigabytes. a corrupt or cut-off record raises PcapError only after every earlier packet has been yielded, so the command prints the good packets, then the error, and exits with 1. this is the one place that raises, because a broken file is not a packet.
##stage 2: what the payload says
stage 1 stopped at the transport layer. stage 2 looks at what TCP and UDP carry. the new parsers follow the same rules as the old ones: never raise, put trouble in error or anomalies. decode() now returns up to four layers, for example (Ethernet, IPv4, Udp, Dns).
finding the protocol
| protocol | how it is found |
|---|---|
| DNS | port 53 on UDP, or on TCP when the whole length-prefixed message is in the segment |
| HTTP | the payload starts with a request method or HTTP/1. |
| TLS | the payload starts with 16 03 and has a ClientHello (01) as the handshake type |
DNS has no magic bytes, so it goes by port. HTTP and TLS are found by content, so a web server on 8080 or TLS on 8443 still works. the weak spot is that there is no stream reassembly yet. a segment from the middle of a stream that happens to start with GET would be read as HTTP. stage 3 fixes that by parsing streams instead of segments.
for the same reason every application parser works on one segment or datagram. a DNS-over-TCP message split across two segments gets no DNS layer. HTTP headers cut off by the segment are parsed as far as they go, with an anomaly. a ClientHello that continues in another record is parsed as far as it is present, with a truncated client hello anomaly. buffering across packets here would do the job of stage 3 twice.
DNS names and compression pointers
a DNS name is a list of length-prefixed labels. to save space, a name can end in a pointer (two bytes starting with binary 11) to an earlier name. that is where the danger is: a pointer that points at itself, or two that point at each other, loops forever. so a pointer has to point backward, into the message body and not the header, a name may follow at most 32 of them, and a name is at most 255 bytes.
the answer to www.example.com A from the generated capture, 63 bytes. both answers use pointers: c0 0c is offset 12, c0 10 is offset 16, where example.com starts.
if kind == 0xC0:
if pos + 1 >= len(data):
raise _Malformed("truncated compression pointer")
target = ((n & 0x3F) << 8) | data[pos + 1]
if end < 0:
end = pos + 2
jumps += 1
if jumps > _MAX_JUMPS:
raise _Malformed("too many compression pointers")
if not _HEADER_LEN <= target < pos:
raise _Malformed("compression pointer does not point backward into the message")
pos = target
continuethe header also carries the number of records in each section, up to 65,535, and it is just as untrusted. the loops stop at the first record that is missing, so a lying header costs nothing.
# a header that claims 65,535 records in each section, and nothing after it
parse_dns(header).anomalies == ('truncated dns message: question 1 of 65535 is missing',)only a header that cannot be read is an error. a broken or missing record ends parsing as an anomaly, and the records before it are kept. the record data is turned into text for A, AAAA, NS, CNAME, PTR, MX and TXT, and the raw bytes are always kept too, because the DNS tunneling detector in stage 5 will need the real bytes.
HTTP
the HTTP parser reads the start line, the headers and the body bytes that are in the segment. HTTP/1.x only: HTTP/2 needs TLS and is binary. it has limits (8 KB for the start line, 16 KB of headers, 100 headers) so a hostile packet cannot make it work hard. two anomalies are there because they are cheap and an IDS will want them: conflicting Content-Length values, and a non-numeric one. both are signs of request smuggling.
the TLS ClientHello
the ClientHello is the first thing a client sends, in the clear, and it says a lot: the server name it wants (SNI), the versions it offers and every cipher suite it supports. the parser reads those, plus the list of extension ids in order. real clients mix in GREASE values (RFC 8701), random placeholders that servers must ignore. the parser keeps them in the data and the printed line hides them.
the 161-byte ClientHello from the generated capture. long fields are shortened with an ellipsis. the command prints it as: TLS ClientHello, sni example.com, versions [TLS 1.3, TLS 1.2], ciphers (15) [TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, +12 more]
printing what the network sends you
names, header values and the server name come from the network, and the command prints them. a terminal escape sequence inside a hostname would run in the analyst's terminal. so the parsers escape everything before they hand it out: control characters become \xNN, and DNS labels use the escapes from RFC 4343. this happens at parse time, not print time, because stage 4 filters and stage 5 JSON alerts will use the same strings and should not have to remember.
def printable(data: bytes) -> str:
return "".join(chr(b) if 0x20 <= b < 0x7F else f"\\x{b:02x}" for b in data)# request line: GET /a<ESC>[2J HTTP/1.1
target = "/a\x1b[2J"
host = "\x07evil"the cost: a literal backslash-x written in the original data looks the same as an escape. for a display and detection string that is fine, and the raw bytes stay available where they matter (the DNS record data).
##stage 3: from packets to conversations
stages 1 and 2 look at one packet at a time. but a web request is not one packet, and the packets of a TCP connection do not always arrive in order, or once. stage 3 adds a flow table that groups packets into conversations, and a reassembler that puts each direction of a TCP connection back into one byte stream. the new command is python -m sentinel flows.
flows
a flow is a protocol plus two endpoints (an address and a port), in either direction. the client is whoever sent the SYN. a new flow starts when the same endpoints are used again after the old one ended, when a SYN arrives with a different initial sequence number, or after the flow sat idle for an hour (TCP) or two minutes (UDP). the state is worked out from what was seen: syn-sent, established, closing, closed, reset, or midstream when the capture missed the handshake. ICMP, ARP and IP fragments are not flows. they are only counted.
this stage has its own demo capture (32 packets, gen_pcap.py --streams): segments that arrive out of order and twice, messages split across segments, a lost segment, a reset, one UDP exchange and one ICMP echo. one line per flow:
$ python -m sentinel flows streams.pcap tcp 10.0.0.1:44000 > 10.0.0.2:8080: closed, pkts 8/3, bytes 129/40, 0.010000s [1 retransmitted client segment] [1 out-of-order client segment] | -> HTTP: POST /upload HTTP/1.1, host files.test | <- HTTP: HTTP/1.1 200 OK tcp 10.0.0.1:45000 > 10.0.0.2:8443: established, pkts 4/1, bytes 161/0, 0.004000s [1 out-of-order client segment] | -> TLS ClientHello, sni example.com, versions [TLS 1.3, TLS 1.2], ciphers (15) [TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, +12 more] tcp 10.0.0.1:46000 > 10.0.0.2:53: established, pkts 4/1, bytes 66/0, 0.004000s | -> DNS query 1, A? example.com | -> DNS query 2, AAAA? www.example.com tcp 10.0.0.1:47000 > 10.0.0.2:80: closing, pkts 5/1, bytes 160/0, 0.005000s [100 client bytes missing] tcp 10.0.0.1:48000 > 10.0.0.2:22: reset, pkts 1/1, bytes 0/0, 0.001000s udp 10.0.0.1:53004 > 10.0.0.2:53: pkts 1/1, bytes 33/63, 0.001000s | -> DNS query 4660, A? www.example.com | <- DNS response 4660 NOERROR, A? www.example.com, answers [CNAME example.com, A 192.0.2.1] # 6 flows, 31 packets in flows, 1 not in a flow
6 flows. the numbers after pkts and bytes are client/server. anything odd is a [note], and the messages found in the streams follow the bars: -> from the client, <- from the server.
putting segments back in order
every direction of a connection gets its own stream. segments are kept as sorted byte ranges and merged when they touch. the first flow above is a POST request cut into three segments. they arrive out of order, and one of them arrives twice. this is that flow, replayed through the real reassembler one packet at a time:
a few rules that could each have gone the other way:
when two copies of the same bytes overlap, the first copy that arrived wins. if the later copy has different bytes, that is counted as a conflict, because sending different bytes for the same position is exactly how an attacker confuses an IDS. a capture does not say which operating system receives the data, so picking a side would be a guess. reporting the conflict is not.
merged_start = min(start, self._starts[lo]) if lo < hi else start
merged_end = max(end, self._starts[hi - 1] + len(self._chunks[hi - 1])) if lo < hi else end
buf = bytearray(merged_end - merged_start)
buf[start - merged_start : end - merged_start] = data
for k in range(lo, hi): # bytes already stored win over the new copy
s, chunk = self._starts[k], self._chunks[k]
a, b = max(start, s), min(end, s + len(chunk))
if a < b and buf[a - merged_start : b - merged_start] != chunk[a - s : b - s]:
self.overlap_conflicts += 1
buf[s - merged_start : s - merged_start + len(chunk)] = chunk
self._starts[lo:hi] = [merged_start]
self._chunks[lo:hi] = [buf]only the bytes that are contiguous from the start of the stream are returned. after a gap, later bytes are kept for the statistics and for a late arrival, but never handed out, and a gap is never filled with zeros, because a parser could then read straight across a hole. positions are signed offsets from the first sequence number seen, so sequence numbers that wrap past 232 need no special case. a stream with no SYN starts at the lowest byte received, and the flow says so.
the counters follow how Wireshark counts: a segment is out of order when it has new bytes below the highest byte already seen, and retransmitted when it carries bytes already held.
what read gets wrong and flows gets right
stage 2 said it out loud: reading one segment at a time can be fooled. the demo capture has both cases. in the POST flow, the body contains text that looks like a request line, and it starts a segment of its own. in the TLS flow the ClientHello is split in two, and the second half arrives first.
$ python -m sentinel read streams.pcap # two of the packets 2023-11-14 22:13:20.005000 IP 10.0.0.1.44000 > 10.0.0.2.8080: Flags [P.], seq 7064, ack 9001, win 64240, length 33: HTTP: GET /inside-the-body HTTP/1.1 2023-11-14 22:13:20.015000 IP 10.0.0.1.45000 > 10.0.0.2.8443: Flags [P.], seq 12001, ack 13001, win 64240, length 90: TLS ClientHello, versions [TLS 1.2], ciphers (0) [] [truncated client hello: 81 of 152 handshake bytes] [client hello ends before its fields do]
$ python -m sentinel flows streams.pcap # the same two connections tcp 10.0.0.1:44000 > 10.0.0.2:8080: closed, pkts 8/3, bytes 129/40, 0.010000s [1 retransmitted client segment] [1 out-of-order client segment] | -> HTTP: POST /upload HTTP/1.1, host files.test | <- HTTP: HTTP/1.1 200 OK tcp 10.0.0.1:45000 > 10.0.0.2:8443: established, pkts 4/1, bytes 161/0, 0.004000s [1 out-of-order client segment] | -> TLS ClientHello, sni example.com, versions [TLS 1.3, TLS 1.2], ciphers (15) [TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, +12 more]
read is fooled twice: it prints a request nobody made, and a half ClientHello with no server name. flows parses the streams. it looks for an HTTP message only where one can start, uses Content-Length to jump over the body, and reads the ClientHello from the whole client stream, so it finds one POST and one complete hello with example.com.
I did not make read stateful. it prints as it reads, in constant memory, and a packet's line does not depend on the rest of the capture. making it stateful would fix these two cases and cost both of those. so read keeps its known weaknesses, and flows is the command to use when they matter.
limits, all counted
a stream is attacker-controlled input too. a sender can announce a sequence number four gigabytes ahead, or send a million one-byte segments with gaps between them. so everything that grows has a limit, and hitting a limit is counted and shown in the flow's notes, never silent:
| what | limit | when it is hit |
|---|---|---|
| bytes kept per stream | 1 MiB from its start | later bytes are counted as not buffered |
| separate byte ranges per stream | 4,096 | a segment that would start another range is dropped and counted |
| bytes buffered in all streams | 256 MiB | a segment that does not fit is dropped and counted |
| flows | 200,000 | packets that would start another flow are counted; existing flows keep working |
a stage 2 bug that stage 3 found
the HTTP parser reported malformed http header line when a segment ended exactly at a line break, because the empty last line was counted as a header. per-packet reading rarely ends there. reassembly tests cut messages at every position, and they hit it at once. it is fixed, with a test.
##stage 4: asking for the packets you want
after stage 3 the tools print everything they see. stage 4 adds a filter language, close to the one tcpdump uses, so you can ask for what you want: --filter "tcp and (port 80 or port 443) and not src host 10.0.0.1". it works on both commands. it is a small language with its own lexer, parser and evaluator, and no library.
from text to a tree
the lexer cuts the text into tokens and remembers where each one started. the parser builds a tree from them, and it follows the usual precedence: not binds tighter than and, and and binds tighter than or. parentheses group. a protocol followed by a qualifier is an implicit and, so tcp port 80 means tcp and port 80, as in tcpdump.
$ tree of the filter
and
├─ tcp
├─ or
│ ├─ port 80
│ └─ port 443
└─ not
└─ src host 10.0.0.1the tree prints back as tcp and (port 80 or port 443) and not src host 10.0.0.1, which parses to an equal tree. a test does that round trip for 1,500 random trees. and and or keep all their operands in one flat node, so a chain of a thousand ands is one node, not a thousand deep. that keeps both the parser and the evaluator from recursing on a long filter.
asking the evaluator
the evaluator answers one question about one decoded packet: does it match? here is what 11 filters pick out of the 19 packets of the demo capture. each column is a packet, in the order of the read output above, and orange means it matches.
the number on the right is how many of the 19 packets match. the last row is the filter from the top of this section.
def matches(expr: Expr, layers: Sequence[Layer]) -> bool:
"""Does the packet, as decoded by `decode()`, match the filter?"""
match expr:
case Proto(name):
return any(isinstance(layer, _PROTO_LAYERS[name]) for layer in layers)
case Vlan(vid):
eth = layers[0] if layers else None
if not isinstance(eth, Ethernet) or eth.error:
return False
return bool(eth.vlans) if vid is None else vid in eth.vlans
case Host(direction, addr):
return _side(_addresses(layers), direction, lambda a: a == addr)
case Net(direction, net):
return _side(_addresses(layers), direction, lambda a: a in net)
case Port(direction, lo, hi):
return _side(_ports(layers), direction, lambda p: lo <= p <= hi)
case Not(operand):
return not matches(operand, layers)
case And(operands):
return all(matches(o, layers) for o in operands)
case Or(operands):
return any(matches(o, layers) for o in operands)a few meanings could each have gone the other way. a protocol word is true when the layer is there, even if the layer has an error: a broken TCP header is still a TCP packet. but host, net and port need an intact header. a layer with an error leaves zeros behind, and those zeros must not match port 0 or host 0.0.0.0. host and net also look at the addresses in ARP packets, and an IPv4 address never matches an IPv6 network.
IP fragments are not decoded past the IP header, so they match ip and host but not tcp or port. and http matches the packets where a segment starts an HTTP message, not every packet of an HTTP connection. that is the per-packet view from stage 2. the whole-connection view is what flows is for.
a typo is not a traceback
a filter comes from a person, so a mistake in it has to be an ordinary answer. like the packet parsers, parse_filter never raises. it returns a result with an error and the position in the text, and the command shows a caret under it and exits with code 2, before it opens the file.
$ python -m sentinel read demo.pcap -f "tcp and port http" # exit code 2
sentinel: invalid filter: expected a port number (0-65535), got 'http'
tcp and port http
^| filter | what it says | at |
|---|---|---|
tcp and port http | expected a port number (0-65535), got 'http' | 13 |
host example.com | expected an IP address, got 'example.com' | 5 |
(tcp or udp | expected ')' | 11 |
tcp udp | unexpected 'udp', expected 'and', 'or' or end of filter | 4 |
portrange 90-80 | invalid port range '90-80' | 10 |
port 70000 | expected a port number (0-65535), got '70000' | 5 |
src tcp | expected host, net, port or portrange, got 'tcp' | 4 |
tcp & udp | unexpected '&', did you mean '&&'? | 4 |
not | unexpected end of filter, expected a filter primitive or '(' | 3 |
two things I found by trying to break my own parser. Python's int() raises on a very long string of digits, so a filter with a five-thousand-digit port could have been a traceback after all. numbers are now limited to ten digits, and nesting to 100 levels and the filter to 1,000 tokens. and my first error messages quoted the whole offending token, so that same port filled the terminal. messages now cut a token at 40 characters.
filters and flows
the filter picks packets before anything else happens, so flows builds its flows from the matching packets only. a filter on the endpoints keeps whole conversations:
$ python -m sentinel flows streams.pcap -f "port 53" tcp 10.0.0.1:46000 > 10.0.0.2:53: established, pkts 4/1, bytes 66/0, 0.004000s | -> DNS query 1, A? example.com | -> DNS query 2, AAAA? www.example.com udp 10.0.0.1:53004 > 10.0.0.2:53: pkts 1/1, bytes 33/63, 0.001000s | -> DNS query 4660, A? www.example.com | <- DNS response 4660 NOERROR, A? www.example.com, answers [CNAME example.com, A 192.0.2.1] # 2 flows, 7 packets in flows, 0 not in a flow
a filter that drops some packets of a connection does not. http matches the packets that start an HTTP message, so the rest of the connection disappears from the flow, and reassembly says so:
$ python -m sentinel flows streams.pcap -f http tcp 10.0.0.1:44000 > 10.0.0.2:8080: midstream, pkts 2/1, bytes 63/40, 0.003000s [client stream start not captured] [33 client bytes missing] [server stream start not captured] | -> HTTP: POST /upload HTTP/1.1, host f [http headers incomplete: no blank line in this segment] | <- HTTP: HTTP/1.1 200 OK # 1 flows, 3 packets in flows, 0 not in a flow
I did not add flow-level filters. src and dst describe one packet, while the two ends of a flow have no direction, so a flow filter needs its own meaning. that can come later.
how it is tested
the evaluator is compared with predicates I wrote by hand, straight from the meaning of each word and without using the filter code: 34 primitives over a corpus of 80 packets (both demo captures, fragments, cut headers, stacked VLAN tags, garbage). then 400 random combinations of and, or and not are compared with Python's own logic over the same predicates. the parser is tested with exact trees, 31 error messages with positions, the print-and-parse round trip, and 25,000 random texts that must never raise.
##stage 5: raising the alarm
until now the tools describe traffic. stage 5 judges it. the ids command runs four detectors over a capture and prints an alert for each thing it finds, as one JSON object per line. the detectors and their thresholds are read from rule files. there is no library and no machine learning: each detector counts something simple over a sliding window, and says what it counted.
the four detectors
| detector | what it looks for | default | a false positive to expect |
|---|---|---|---|
port_scan | one source probing many ports of one host, or one port of many hosts. SYN scans and the stealth scans (no flags, FIN only, FIN+PSH+URG) count. | 15 ports or 30 hosts in 10 s | vulnerability scanners you run yourself |
syn_flood | many SYNs to one address and port, and few of them completed by the handshake's ACK. | 100 SYNs in 1 s, at most 30% completed | a burst of connections to a server that then fails to answer |
arp_spoof | an IP address that moves to another MAC address, or an ARP sender that is not the Ethernet source. | any change | a replaced network card, a failover pair |
dns_tunnel | very long names or labels, random-looking subdomains, many different subdomains of one domain, bursts of "no such name". | 100 / 50 characters, 4.2 bits, 50 subdomains, 20 answers | long generated names of some CDNs and security products |
each detector is fed one decoded packet at a time, in capture order, and sees only intact layers. a packet with a broken header cannot fool it with the zeros the parser leaves behind. detectors work on packets, not on flows, because the attacks they look for mostly produce one-packet flows, and a flow table would only spend memory on every spoofed SYN.
two captures: one harmless, one not
every detector needs a test in both directions: the attack must raise the alert, and ordinary traffic must not. the generator writes both. the harmless one (647 packets) is kept deliberately close to every threshold: a web server that gets 150 SYNs in one second and completes every handshake, a client that uses 14 ports of one host, another that visits 25 servers on port 443, 40 different subdomains looked up in a few seconds, long but readable hostnames, ten "no such name" answers, and repeated ARP announcements. the attack capture (1044 packets) has one of each attack from separate sources.
$ python -m sentinel ids attacks.pcap --format text 2023-11-14T22:13:20.140000Z [medium] port-scan: 10.9.9.1 probed 15 ports on 10.0.0.30 in 0.1s 2023-11-14T22:13:23.700000Z [medium] port-scan: 10.9.9.2 probed 15 ports on 10.0.0.30 in 0.7s 2023-11-14T22:13:24.700000Z [medium] port-scan: 10.9.9.3 probed 15 ports on 10.0.0.30 in 0.7s 2023-11-14T22:13:25.700000Z [medium] port-scan: 10.9.9.4 probed 15 ports on 10.0.0.30 in 0.7s 2023-11-14T22:13:29.160000Z [medium] port-scan: 10.9.9.5 probed port 22 on 30 hosts in 1.2s 2023-11-14T22:13:32.099000Z [high] syn-flood: 100 SYNs to 10.0.0.2:80 in 0.1s from 100 sources, 20 completed 2023-11-14T22:13:41.000000Z [high] arp-spoof: 10.0.0.1 moved from 02:aa:00:00:00:01 to 02:ee:00:00:06:66 2023-11-14T22:13:42.000000Z [high] arp-spoof: ARP says 10.0.0.1 is at 02:aa:00:00:00:01, but the frame came from 02:ee:00:00:06:66 2023-11-14T22:13:42.000000Z [high] arp-spoof: 10.0.0.1 moved from 02:ee:00:00:06:66 to 02:aa:00:00:00:01 2023-11-14T22:13:50.000000Z [medium] dns-tunnel: 10.0.5.5 asked for a random-looking name under evil-cdn.test 2023-11-14T22:13:52.450000Z [medium] dns-tunnel: 10.0.5.5 asked for 50 different subdomains of evil-cdn.test in 2.5s 2023-11-14T22:13:55.000000Z [medium] dns-tunnel: 10.0.5.6 asked for a very long name under example.org (124 characters) 2023-11-14T22:13:56.951000Z [medium] dns-tunnel: 10.0.6.6 received 20 'no such name' answers in 0.9s $ python -m sentinel ids benign.pcap # 647 packets, no output
13 alerts for the attacks, and nothing at all for the harmless traffic. the alerts are in time order. every JSON alert also carries the evidence behind it (the ports, the counts, the MAC addresses).
$ python -m sentinel ids attacks.pcap # the first alert
{"detector": "port_scan", "dst": "10.0.0.30", "evidence": {"ports": 15, "sample": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], "seconds": 0.14}, "message": "10.9.9.1 probed 15 ports on 10.0.0.30 in 0.1s", "rule": "port-scan", "severity": "medium", "src": "10.9.9.1", "ts": "2023-11-14T22:13:20.140000Z", "ts_ns": 1700000000140000000}counting in a window
a port scan is not "many ports", it is many ports quickly. the detector keeps, for each source and target, the probes of the last ten seconds, and counts the different ports in them. below, a scanner and a browser, with the time counted from each one's first probe. the browser opens 14 ports of one host, every connection completes, and it stops one short of the threshold. the scanner crosses it after 0.14 seconds and goes on to 200.
a probe is a TCP SYN, or one of the stealth scans: no flags at all, FIN only, or FIN+PSH+URG. an ACK, a SYN-ACK or a PUSH belongs to a real connection and never counts. UDP does not count either, because DNS and mDNS would make it noisy. an alert fires once per source and target and window, so one scan is one alert, not two hundred.
the window itself is a small class: a queue of timestamped events with a size limit and a count per key. it drops events older than the window when a new one arrives, and the oldest event when it is full, so a sender cannot make it grow. every detector keeps its state in these, and the whole state of a detector is capped at 100,000 keys. when a limit forces a packet to be ignored, the engine says so in an alert of its own instead of staying quiet.
a busy server is not a flood
the first idea for a SYN flood detector is a rate: too many SYNs per second. it is wrong, because a busy server has exactly that rate. what differs is what happens next. a real client answers the server's SYN-ACK with an ACK, and a spoofed source never does. so the detector matches each ACK to its SYN by the 4-tuple, and alerts when there are 100 SYNs to one service within a second and at most 30% of them were completed.
the busy server gets 150 SYNs in a second and every handshake completes, so the two grey lines lie on top of each other. the flood sends 400 SYNs in 0.4 seconds and 20 of them complete. the alert fires at the hundredth SYN.
measuring before choosing a threshold
the DNS detector has an entropy test: names whose subdomain looks random, as data encoded into a name would. I guessed a threshold first, then measured, and the guess was wrong. this is the entropy of 40-character subdomains, in bits per character, for data encoded in three ways and for readable hostnames that I made up to look like real ones:
base32 and base64 land mostly above 4.2 (290 of 300 base32 samples, 300 of 300 base64). hex tops out at 4, so it is never caught (0 of 300). and the readable hostnames reach 4.10, close enough to the line that a lower threshold would flag them. so the default is 4.2, only names of 40 characters or more are judged, and the entropy test is one of four signals, not the only one. it misses hex-encoded tunnels. the long-name, many-subdomains and "no such name" signals are there for what entropy cannot see.
rules
the thresholds come from rule files in TOML, which the standard library reads. --rules takes a file or a folder of them, and without it the built-in defaults are used. a test keeps the built-in defaults equal to rules/default.toml. this is the first rule in it:
[[rule]]
id = "port-scan"
detector = "port_scan"
severity = "medium"
distinct_ports = 15 # ports of one host within the window (0 turns this check off)
distinct_hosts = 30 # hosts on one port within the window (0 turns this check off)
window_seconds = 10.0
a rule is a detector, its parameters, a severity and, if wanted, a filter written in the language from stage 4. the filter decides which packets the rule sees. that is how one rule can watch only SSH, more closely than the rest, while the default rule stays as it is:
# my-rules.toml: watch SSH more closely than everything else
[[rule]]
id = "ssh-scan"
detector = "port_scan"
severity = "high"
filter = "dst port 22" # this rule only sees packets to port 22
distinct_hosts = 5 # five different SSH servers is already a sweep
distinct_ports = 0 # port counting is not interesting for one port
window_seconds = 30loading a rule file never raises. every problem comes back as a message with the file and the rule number, all of them at once, and the command exits with 2. parameters are typed and range-checked. an unknown key is an error, so a typo in a parameter name cannot silently keep the default. two rules of the same detector are independent, and a rule can be switched off with enabled = false.
the exit code is 0 whenever the capture was read, with or without alerts. it is 1 when the capture is unreadable, after printing the alerts raised before the corrupt record, and 2 for bad rules. I did not make it non-zero when there are alerts, because a script could not tell a finding from a failure.
##stage 6: listening to a real network
until now every command read a file. stage 6 reads a network interface instead. live receives the Ethernet frames of one interface, on Linux, and sends them through the steps that already exist: decode, filter, then print them, run the detectors, or write them to a pcap file. what is new is small. most of this stage is about what a file never shows.
I have to say how this was tested. I write this on Windows, with no WSL and no Docker, and the packet socket (AF_PACKET) does not exist here. so the code that talks to the real socket could not be tested here. everything around it was tested against a stand-in, and the real socket was tested later, in CI (stage 7).
the same pipeline
a capture gives Packet values, the same as the pcap reader: a time in nanoseconds, the length on the wire and the bytes. nothing after that knows where a packet came from. the socket is opened raw, for all protocols, and bound to one interface, so what arrives is every frame the interface sends or receives, header included, which is where the parsers start. it only listens: the object the loop talks to has no way to send. it is not put in promiscuous mode, because that changes the state of the interface and would outlive the program.
the time on a packet is the system clock when the program receives it, not the kernel's own timestamp. that is simpler, and a few tens of microseconds late when the machine is busy. a pcap file written by --write keeps microseconds, like every capture this project writes.
a socket I can't open here
so the socket had to be the thinnest part. the capture class takes anything that has recvfrom, settimeout, getsockopt and close. the clock is an argument, and so is the folder that stands for /sys/class/net. only open_capture, a few lines, touches the real socket module, and that is tested against a stand-in module.
the stand-in hands out the packets of the generated captures, and the clock replays their timestamps. that allows the strongest test I could think of: live on the attack capture must print exactly what ids prints for the file, the same 13 alerts. and each alert must come out when it is raised, not at the end. the test asserts that the first one is printed before the 28th packet arrives. the two blocks below are the real command on the stand-in:
$ sudo python -m sentinel live eth0 --filter "tcp and port 80" --count 4 2023-11-14 22:13:20.005000 IP 10.0.0.1.40000 > 10.0.0.2.80: Flags [S], seq 1000, win 64240, options [mss 1460,sackOK,TS val 1000 ecr 0,nop,wscale 7], length 0 2023-11-14 22:13:20.006000 IP 10.0.0.2.80 > 10.0.0.1.40000: Flags [S.], seq 5000, ack 1001, win 64240, options [mss 1460,sackOK,TS val 1000 ecr 0,nop,wscale 7], length 0 2023-11-14 22:13:20.007000 IP 10.0.0.1.40000 > 10.0.0.2.80: Flags [.], seq 1001, ack 5001, win 64240, length 0 2023-11-14 22:13:20.008000 IP 10.0.0.1.40000 > 10.0.0.2.80: Flags [P.], seq 1001, ack 5001, win 64240, length 38: HTTP: GET / HTTP/1.1, host example.test # 4 packets # on stderr
$ sudo python -m sentinel live eth0 --ids --format text 2023-11-14T22:13:20.140000Z [medium] port-scan: 10.9.9.1 probed 15 ports on 10.0.0.30 in 0.1s 2023-11-14T22:13:23.700000Z [medium] port-scan: 10.9.9.2 probed 15 ports on 10.0.0.30 in 0.7s 2023-11-14T22:13:24.700000Z [medium] port-scan: 10.9.9.3 probed 15 ports on 10.0.0.30 in 0.7s ... 10 more alerts, the same as ids prints for the file # 1044 packets # on stderr
every printed line is flushed. without that, a pipe would show nothing for minutes, because a program that writes to a pipe buffers its output. --write flushes after every packet too, so a capture that is killed still leaves a file that can be read. a test checks that the first packets are on disk when the next one is received.
what can go wrong when it starts
the filter, the rules and the options are checked before the interface is opened, so a typo never needs root to be noticed. then come the ways to fail that only a real system has. each has a message that says what to do, and exit code 1. these are the real messages (the last two come from the stand-in, because I cannot cause them here):
| what goes wrong | what it prints |
|---|---|
| the system is not Linux | sentinel: live capture needs Linux (AF_PACKET), and this system has no such socket |
| a name that could point outside /sys/class/net | sentinel: '../eth0' is not a valid interface name |
| no interface of that name | sentinel: no such interface: eth9 |
| Wi-Fi in monitor mode (not Ethernet frames) | sentinel: wlan0mon has link type 803; only Ethernet is supported |
| not root | sentinel: live capture needs root or the CAP_NET_RAW capability |
| the interface disappears before the bind | sentinel: cannot capture on eth0: [Errno 19] No such device |
the interface name is checked with the kernel's own rules before it is used to build a path, so a name like ../eth0 cannot read a file outside the folder. and only Ethernet and loopback are accepted, because the parsers only know Ethernet.
the loopback copy
one thing I know only from how libpcap works: on the loopback interface the kernel gives a packet socket every packet twice, once leaving and once arriving. libpcap drops the leaving copy on loopback, and so does live, but only when the hardware type is loopback. on a real interface the leaving packets are wanted. if I am wrong about the kernel, nothing breaks: the arriving copy is always kept, so each packet is seen once either way. a test for it (a datagram must be captured exactly once, and twice with the filter off) runs on a Linux runner in CI, and stage 7 says what happened.
what a file never shows
a capture file ends. a live capture does not, so I asked what changes when the input never stops, and found a bug in stage 5. each detector kept one window per source, target or client, and never removed one. the tables had a limit of 100,000 keys, and past it the detector stopped following new ones. on a network that sees 100,000 different sources in a week, that means being blind from the second day. the chart replays it: one new source every two milliseconds for 150,000 sources (300 seconds), then a real port scan at 310 seconds.
the old table is full after 200 seconds, then ignores 50,000 packets, and the scan at 310 seconds is not seen. the new one stays near 5,000 sources, which is 10 seconds of them, and finds the scan. the fix keeps each table in order of last use, and when a new key arrives it forgets the keys at the front that have been idle for longer than the window. that is safe because such a window would be empty anyway. it costs nothing on average, since each key is removed once, and it uses an ordered dictionary: removing from the front of a plain one and asking for the first key again scans the deleted slots, which I measured as 40 times slower at 100,000 keys. the boundary is tested to the microsecond: a source is kept for exactly one window and forgotten one microsecond later.
two smaller things turned up on the way. the engine kept every alert until the end of the run, which never comes in a live capture, so it now has pop_alerts(): give me what was raised so far, and forget it. and the count in the "packets were not tracked" alert counted a port scan packet twice when the tables were full, because two tables refused it. it is counted once now.
##stage 7: measuring it
the last stage adds no feature. it answers three questions about what exists: how fast is it, what does it do with input nobody planned for, and does it still work on a machine that is not mine. the answers are a benchmark tool, a fuzzer, and a CI workflow. the first two ran here. the third could only be tested by pushing it to GitHub, and that took four tries.
how fast is it
python -m tools.bench runs every stage over the same traffic and reports packets per second. the traffic is the four generated captures one after the other, repeated with the time moved forward, 100,000 packets in all, about 65 bytes each on average. the workload is the same on every machine. only the timings change. each stage runs three times and the fastest run counts, because the fastest run is the one the rest of the system disturbed least. and every stage must handle every packet, or the tool stops with an error instead of printing a number for a stage that quietly did less work.
this is one core of an AMD64 Family 25 Model 33 Stepping 2, AuthenticAMD with 16 logical CPUs, Windows-11-10.0.26200-SP0, Python 3.13.5. each dot is one stage, on a log scale, because the slowest and the fastest differ by a factor of about 93:
reading a pcap file takes 1,564,676 packets per second and writing one 2,567,763, which is close to nothing next to decoding, at 98,710. everything the commands do comes on top of decoding: printing a line costs 13 microseconds more, the four detectors 14 more, and the flow table 7 more.
I expected one slow function. I profiled decoding, 30,000 packets, to find it:
there is none. the biggest share is 13% and the rest is spread over the parsers, the checksums and the ipaddress objects the parsers build. a profiler makes small functions look bigger than they are, so read it as a rough picture. it says that no single fix would make decoding faster, and I did not change anything to improve a number. this stage measures.
the number that matters most is the last three dots. the live loop costs 27 to 36 microseconds per packet, which is 27,635 to 37,716 packets per second. at 65 bytes a packet that is only about 2 MB/s. a busy network sends more, and then the kernel drops the packets the program is too slow for. the summary line at the end says how many. that is a real limit of a pure Python program on one core, and I would rather print it than hide it.
the loop also answers the stage 6 question about flushing. writing after every packet costs 2.4 microseconds more than not flushing (2.8 against 0.4). that is about a tenth of the loop, and it buys a capture file that can be read even when the program is killed. I keep it.
a fuzzer
stages 1 to 6 each fuzzed their own part. python -m tools.fuzz fuzzes all of it at once: it damages packets from the generated captures, one to three damages each (a flipped bit, a length field set to an extreme, a cut, a piece deleted or repeated, the end of another packet spliced on) and sends them through the decoder, the summary line, eight filters, one flow table and one detector engine. the flow table and the engine live for the whole run, so state builds up the way it does on a long capture. every fifth input is a small capture file, damaged the same way, that the pcap reader has to read.
a failure is an exception (the pcap reader may only raise PcapError), a summary line that is not one line of printable ASCII, or an alert that is not one line of JSON. the same seed makes the same inputs. a failing packet is printed as hex that --replay runs on its own. the tool does about ten thousand inputs a second.
I ran 3,000,000 inputs (seed 7: 2,400,000 packets and 600,000 capture files). it found nothing. I do not take that as proof, for two reasons. the damages are random, so an input that needs three specific bytes to line up will not turn up. and the checks are only the ones above: the fuzzer cannot tell a wrong parse from a right one.
so I measured what it is worth. I took the 148 deliberate breaks of stages 1 to 6 from the sabotage run, applied each one to the code, and ran the fuzzer alone on it (30,000 inputs, the same seed). the tests catch every one of them. the fuzzer alone caught 2:
2 of 148 is 1.4%, which is very little. what it catches are the breaks that make something raise or print a broken line: "let control characters through" put a control character in a line it prints, and "counted any ACK as a completed handshake" made the detector raise a KeyError. what it cannot catch is a wrong answer that is well formed: "skipped the header checksum" or "let an option run past the end" leave every output looking fine. that is the job of the tests with expected values, and this is why the project has both. one more break, "stopped reading tcp port 80 as tcp and port 80", made the fuzzer's own list of filters stop parsing, so it crashed before its first input. I do not count that as a catch.
CI: four pushes
I developed this project with Python 3.13 on Windows, but it should also work on Python 3.12 and above. so I wrote a workflow, .github/workflows/ci.yml (58 lines), that runs everything on Python 3.12 and 3.13, on Linux and on Windows: ruff, the format check, strict mypy, the tests, 200,000 fuzz inputs, and a small benchmark. the fuzz seed is the run number, so a failure can be repeated with that number. the benchmark is for information only: no threshold, because a speed test that fails on a slow runner teaches people to ignore red.
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
python: ["3.12", "3.13"]
...
- name: Capture on the loopback interface (needs root)
shell: bash
run: |
sudo "$(which python)" -m pytest tests/test_live_linux.py -v -rs | tee live.txt
# The tests that need root must have run, not been skipped.
! grep -q "needs Linux and root" live.txtthe second job is the reason it matters to me. it runs tests/test_live_linux.py as root on a Linux runner, on the loopback interface, and fails if those tests were skipped. that is the only place the real packet socket can run.
the first two runs failed without starting a single job. GitHub had refused the file. one step was named "Benchmark (for information: the numbers depend on the runner)", and YAML reads the colon and space inside an unquoted name as the start of another key. I found out by asking GitHub's API about the run: no jobs meant an invalid file. my tests had read the workflow as text and did not see it, so I added a check for that mistake, and confirmed that it fails on the old line.
the third run started all five jobs, and one of them failed: Python 3.12 on Windows, one test. it checked that every wait of the capture loop was at most the time limit, 0.05 seconds. the loop computes the time left as a deadline minus the clock, and when the clock reading is large a float loses a few billionths: (123456.789012 + 0.05) - 123456.789012 is 0.0500000000029, not 0.05. the code was right and the test was too strict, so the test got a tolerance of a millionth. it only showed on one of the four combinations, which is the reason to run more than one.
the fourth run was green, all five jobs. that includes the one I cared about. on a real Ubuntu runner, as root, the capture opened a real packet socket on the loopback interface, saw a UDP datagram that the test sent, decoded it, and the command line saved it to a file. the log of that job says it plainly: three passed, one skipped, and the skipped one is the test for running without root, which skips itself because the job is root. it is the first time that code ran, and it worked. the Linux jobs without root also checked the real error messages: "no such interface" for a name that does not exist, and the "needs root or the CAP_NET_RAW capability" message from the kernel's refusal. the next push added one more test to that job: a datagram sent to loopback must be captured exactly once. it passed too. but it does not prove that the kernel gives two copies, which is what the capture drops: a datagram is seen once either way. so I added one more test, that counts the copies with the filter switched off and expects two. it passed as well: two copies with the filter off, one with it on. the thing I knew only from how libpcap works is true on a real kernel, and the filter is needed.
the fuzzer ran 200,000 inputs on each of the four combinations (the seed is the run number, 3 and 4): no failures. the small benchmark ran too, once on each, on runners with 4 logical CPUs. it is 20,000 packets and one run, so it is noisier than the table above:
| packets per second | my machine | Ubuntu, 3.12 | Ubuntu, 3.13 | Windows, 3.12 | Windows, 3.13 |
|---|---|---|---|---|---|
| decode | 98,710 | 59,831 | 56,736 | 115,977 | 61,334 |
| read: decode and print a line | 44,135 | 33,416 | 26,335 | 67,282 | 28,174 |
| ids: decode and run 4 detectors | 42,033 | 28,788 | 25,198 | 57,074 | 26,715 |
| live loop, printing lines | 34,551 | 27,389 | 21,929 | 55,896 | 22,174 |
| pcap write, flushed after every packet | 357,768 | 369,740 | 406,820 | 483,074 | 118,459 |
the runners are not one machine. decoding runs from 56,736 to 115,977 packets per second across them, against 98,710 on mine, and the two Windows columns differ by almost a factor of two: the Windows 3.12 log reports a newer AMD processor (family 26, where the other Windows log says 25). so compare within a column, and across columns only roughly. it also means I cannot say from these numbers whether Python 3.12 or 3.13 is faster. writing with a flush after every packet adds 1.6 to 7.7 microseconds per packet across the four runs, against 2.4 on my machine, so the flush cost I measured earlier is a property of the machine as much as of the program.
##seven stages
| stage | what it adds | state |
|---|---|---|
| 1 | pcap reader and writer, Ethernet, ARP, IPv4, IPv6, TCP, UDP, ICMP parsers, command line summary | done |
| 2 | DNS, plaintext HTTP, TLS ClientHello (server name, versions, cipher list) | done |
| 3 | flow tracking, TCP stream reassembly (out of order, retransmitted), per-flow statistics | done |
| 4 | a filter language like tcp and (port 80 or port 443) and not src host 10.0.0.1 | done |
| 5 | IDS engine: port scan, SYN flood, ARP spoofing, DNS tunneling, rule config, JSON alerts | done |
| 6 | live capture on Linux, feeding the same pipeline | done |
| 7 | benchmarks, a fuzzer, CI | done, this post |
##the numbers
tests per file. 626 in total, 6 of them run only on Linux as root.
each parser is tested with a valid packet, a truncated one, a malformed header (where the format has one) and random bytes. on top of that, seven kinds of fuzzing run against the whole decoder, the flow table, the filter, the detectors and the live loop, with fixed random seeds so a failure can be reproduced exactly:
| fuzz | what it does | inputs |
|---|---|---|
| truncation | every prefix of each of the 19 generated packets, through the decoder and the summary printer | 1,639 |
| corruption | every byte of each generated packet set to 0, 255 and two random values, one byte at a time | 6,480 |
| random bytes | 500 seeded random strings of 0 to 200 bytes plus 9 all-zero or all-255 ones, into every parser, and into the decoder behind four ethertypes | 509 per parser |
| capture files | every prefix of a 191-byte valid pcap, and every byte of it replaced once. the reader may only ever raise PcapError | 192 + 191 |
| flow table | every prefix of every packet of both demo captures (51 packets), three random single-byte corruptions of each, and the 509 random strings, all fed to one flow table, then every flow printed | 4,817 |
| filter | 25,000 random texts (symbols, Unicode, and a soup of filter words) through the lexer and parser, 1,500 random trees printed and parsed back, and 300 random frames tried against every primitive | 26,800 |
| detectors | 509 random frames, every third prefix and three random corruptions of every packet of the demo captures and of every 25th attack packet, and the attack capture with scrambled timestamps, all through the engine with the default rules | no exceptions |
| whole pipeline | the stage 7 fuzzer: 3,000,000 damaged packets and capture files, through decoder, summary, eight filters, flow table and detectors that keep their state, checking the output is well formed | 0 failures |
| live frames | an empty frame, one zero byte, 14 bytes of 255, a random 64 KiB frame, 300 random short ones, and a cut-off and a corrupted copy of every 20th attack packet, handed out by the stand-in socket to live in both modes (print and detectors): one line per packet, no exception | 410 x 2 |
speed is measured in stage 7, above. the fuzzer of stage 7 is separate from these: it runs longer and through everything at once.
how to read these numbers: all traffic is synthetic, made by my own generator. that is a rule of the project, no real captures from other networks. so the fuzzing shows the parsers do not raise on bad input, not that they agree with every odd thing a real network does. one machine, Windows 11, Python 3.13. I developed the project with 3.13. CI also ran the tests on 3.12, on Linux and Windows, and they passed, and you can try 3.12 yourself too.
##how I checked it's right
the test packets come from builders in the generator, and the generator uses the same checksum function as the parsers. a bug in that function would cancel itself out. so wherever I could, the tests compare against something written separately:
| claim | what it is checked against |
|---|---|
| IPv4 checksum | a textbook header with a known checksum (0xb861), independent of my builders |
| TCP checksum | a plain sum written separately in the test, over a pseudo-header laid out by hand |
| field extraction | one hand-written hex packet per protocol, not made by my builders |
| pcap reader | a big-endian file built by hand with struct, without my writer |
| round trip | all four variants, write then read, packets identical |
| DNS names | hand-built messages for every malformed pointer and label, with the exact anomaly text asserted, not just that something was reported |
| ClientHello | a hand-built minimal hello and one cut at every byte, plus a browser-shaped one with GREASE |
| safe printing | escape sequences in a request line, a host header and a server name; the printed line must contain only printable characters |
| reassembly | the bytes that were sent: 300 random payloads cut into overlapping, duplicated pieces, shuffled, with the SYN anywhere (even last) and initial numbers near the 32-bit wrap. the stream must equal the payload. the same check runs through the flow table with real packets |
| filter evaluator | 34 predicates written by hand from the meaning of each word, over 80 packets, and 400 random and/or/not combinations compared with Python's own logic |
| filter parser | exact trees for the documented cases, 31 error messages with exact positions, and random trees that must print and parse back to themselves |
| detectors | a harmless capture built to sit just below every threshold (647 packets, no alerts) and an attack capture with exactly the 13 alerts listed one by one, with their messages |
| thresholds | for each detector, a test at one below the threshold and at the threshold, at the edge of the window, for the once-per-window rule, and for switching the check off |
| rule files | 24 malformed rules, each with its exact message, and a test that rules/default.toml equals the built-in defaults |
| live output | the packets of the generated captures, served by a stand-in socket with their own timestamps: live must print what read prints and the 13 alerts ids prints, each one when its packet arrives. a pcap written by --write must read back equal |
| starting up | interface names and link types against a folder that stands for sysfs, including a name that points outside it. every error message and every exit code. bad options, bad rules and a bad filter must fail before the interface is opened (the stand-in raises if it is) |
| forgetting sources | ten sources 30 s apart never fill a table of three. a source is kept for exactly one window and forgotten a microsecond after. a source that keeps sending is not the one forgotten. the same for SYN flood targets, DNS clients and their subdomains |
| benchmark tool | the corpus (order, spacing, exact length), the arithmetic with a fake clock, the fastest run kept, a stage that does less than all the work is an error, and the flushed writer checked against the file on disk |
| fuzzer | every damage against its definition, with brute force over every slice, the same seed giving the same run, and every kind of failure detected by swapping in a part that misbehaves |
| CI workflow | a test reads it and checks it runs the commands of the README, in order, on the claimed Python versions, and has no unquoted value with a colon and space (the mistake that made GitHub refuse it). then GitHub ran it: five jobs, green on the fourth push |
| real socket | tests/test_live_linux.py, as root on a Linux runner: a UDP datagram sent to loopback is captured by a real packet socket, exactly once (twice with the copy filter off), decoded, and saved by the command line. the errors for a missing interface and for a missing permission are real too |
| flow demo | the 6 lines above, read through by eye once and kept as a golden test |
| command output | the 19 lines above, read through by eye once and kept as a golden test; running it twice gives the same text |
that already paid off once. I worked out the ICMP checksum of an 8-byte echo request header by hand and got 0xf7ff. the test failed, and the parser was right: 0x0800 + 0x0001 + 0x0001 = 0x0802, so the checksum is 0xf7fd. my arithmetic was the bug.
##breaking it on purpose
a test that has never failed hasn't proven much. so I broke the code on purpose, one line at a time, and counted how many tests noticed. stage 7 has 48 breaks, this time in the tools and the workflow (a slowest run kept instead of the fastest, a benchmark that divides the wrong way round, a fuzzer check removed, Python 3.12 dropped from the CI matrix), grouped by the part they hit:
| part | breaks | tests that failed, fewest to most |
|---|---|---|
| benchmark corpus | 5 | 1 to 12 |
| benchmark timing | 5 | 1 to 4 |
| benchmark table | 1 | 1 to 1 |
| benchmark stages | 6 | 1 to 3 |
| benchmark command | 1 | 2 to 2 |
| fuzz damages | 10 | 1 to 2 |
| fuzz checks | 7 | 1 to 3 |
| fuzz run | 2 | 1 to 3 |
| fuzz report | 1 | 1 to 1 |
| fuzz command | 4 | 1 to 2 |
| ci workflow | 6 | 1 to 1 |
all 48 breaks are caught, and the 148 from stages 1 to 6 still are, so 196 of 196. none survived the first run this time. I do not read that as luck: I wrote these tests by asking how each tool could be wrong (a writer that checks the size of the file on disk before every packet, brute force over every possible slice), and the run confirms them. but the run also found two things. two breaks made the test suite hang instead of fail: a live loop that asks for one packet too many waits for ever on a socket that only times out, and a corpus of zero packets looped for ever. a hang is a failure, but a slow and unfriendly one, so the stand-in socket now gives up after a thousand idle receives and the corpus loop stops by comparison. and one test had no time limit of its own and hung under a new break, so I gave it one. the fuzzer is measured separately, above.
110 of the 196 breaks are caught by a single test, so the coverage is not deep. a mutation run like this only proves that the tests can fail. it says nothing about the cases nobody thought of, and nothing about the real socket or the real CI, which none of these breaks touch.
##what it still can't do
IPv6 is the fixed 40-byte header only. extension headers are not walked and ICMPv6 is not parsed, so those packets print as ip-proto-N.
IP fragments are not reassembled. TCP streams are, but only offline (read the result when the capture is done, not as it arrives), only the first megabyte of each, and with first-copy-wins for overlaps. another operating system may resolve an overlap the other way, and I cannot tell from a capture.
flows are TCP and UDP only. VLAN tags are not part of a flow's identity, and a connection that sits idle longer than the timeout is split in two. the read command still works one segment at a time, with the weaknesses shown above.
filters have no host names (a lookup would make the output depend on the network), no ether host, no len, no byte offsets like tcp[13] and no TCP flag words. dns, http and tls match per packet, and flows --filter filters packets, not whole flows.
the detectors were tuned on traffic I generated, and nothing else. the thresholds are guesses that hold on my two captures and mean nothing about a real network. a real network has scanners you run yourself, monitoring that looks like a flood, and failover that looks like ARP spoofing. the entropy test cannot see hex-encoded tunnels. and there are only four detectors: no signatures, no payload inspection, no reputation of any kind.
detectors work on packets, so a scan that is slower than the window (one probe every two seconds, say) is not seen, and neither is one spread over many sources.
live capture is Linux only, and needs root. and the real socket was tested once, in CI, on the loopback interface of a Linux runner: a datagram was captured, decoded and saved. everything else about it was tested with a stand-in. one thing is still unchecked: what the frames of a real network card look like. (on loopback a datagram comes out once, and twice with the copy filter switched off, so the kernel does give two copies.)
what the frames look like matters. the interface is not put in promiscuous mode, so it sees only what it would receive anyway (or what a mirror port sends it). packets that the machine itself sends often have checksums that are not filled in yet, because the network card does it, so they can show up as [bad ... checksum]. a VLAN tag may already be removed. flows does not work on a live capture, and a detector's state-limit alert is printed when the run ends, not while it runs.
it is slow, in the way a pure Python program on one core is slow: about 98,710 packets per second to decode and about 34,551 through the live loop. a busy network makes the kernel drop packets, and I did not try to fix that. the numbers are from one machine.
the fuzzer only checks that nothing raises and that the output is well formed. random damage will not find what needs specific bytes in specific places, and a smarter fuzzer that watches which code it reaches would find more.
ARP is Ethernet and IPv4 only. only the Ethernet link type is read; any other type is refused with a message. VLAN tags keep the VLAN id and drop the priority bits.
HTTP is HTTP/1.x only. TLS is the ClientHello only: no ALPN, no fingerprints, and the cipher and version name tables cover the common ones (the rest print as hex).
the printed timestamps have microsecond precision, so a nanosecond file loses its last three digits on screen (not in memory).
and it has only ever seen traffic I generated. a real capture may show me things the tests do not cover.
that is all seven stages. what is left is what only other machines can tell me: the first CI run, the real socket tests on Linux, and a capture that I did not make myself.