part 2: taking sentinel to the next level

the first post ended with seven finished stages and a list called "what it still can't do". Sentinel worked, but it had only ever seen traffic I generated myself, and it could not open the files people actually have. so I am taking it to the next level: this post is the sequel, and each stage in it goes after one item of that list, the one that matters most first.

the rules are the same as before. one stage at a time, tests and ruff and strict mypy clean before the next one, a sabotage run, and a plain note about what I could not check. this post grows a section per stage. it has five so far.

##stage 8: the file people actually have

the first thing anyone does with a packet analyzer is open a capture from Wireshark. Wireshark has saved pcapng by default for years, and the reader I wrote in stage 1 knew only classic pcap. this is what it said to a pcapng file:

$ python -m sentinel read demo.pcapng
sentinel: demo.pcapng: not a pcap file: bad magic 0a0d0d0a

that message is true and useless. after this stage the same command reads it, and prints exactly what it prints for the same packets in a classic file:

$ python -m sentinel read demo.pcapng
2023-11-14 22:13:20.000000 ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 28
2023-11-14 22:13:20.001000 ARP, Reply 10.0.0.2 is-at 02:00:00:00:00:02, length 28
2023-11-14 22:13:20.002000 IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 1, seq 1, length 40
2023-11-14 22:13:20.003000 IP 10.0.0.2 > 10.0.0.1: ICMP echo reply, id 1, seq 1, length 40
2023-11-14 22:13:20.004000 IP 10.0.0.2 > 10.0.0.1: ICMP destination unreachable, code 3, length 36
... 14 more lines, the same as for demo.pcap

the rest of this section is what that took. the whole reader is 183 lines in sentinel/pcap/pcapng.py, and one more small function, open_reader, chooses between the two readers. read, flows and ids use it, so all three take either format.

what is inside a pcapng file

a classic pcap file is a 24-byte header and then one record per packet. a pcapng file is a list of blocks. every block starts with a type and its total length, and ends with its total length again, and its length is a multiple of four, so a reader can skip a block it does not understand without knowing anything about it. that is the whole design, and it is the reason the format could grow. these are the first three blocks of the demo capture, and the bytes are read from the real file:

the section header says what follows and how to read the numbers. the interface description says what the packets came from (a link type, and here one option that tells the resolution of the timestamps). every packet then costs one enhanced packet block: seven fixed fields, the frame padded to a multiple of four bytes, and the length repeated. here is where the 92 bytes of the first one go:

type and length 8 · interface and time 12 · captured and original length 8 · the frame 60 · padding 0 · length again 4

so a packet costs 33.3 bytes of overhead on average here, against 16 in a classic file. the same 19 packets are 1,948 bytes as pcap and 2,312 bytes as pcapng.

which format, and which byte order

the first four bytes of a pcapng file are 0a 0d 0d 0a. that is the block type of the section header, and it reads the same forwards and backwards, so it is the same in a little-endian and a big-endian file. it also cannot be the start of a classic file, whose magic numbers are d4 c3 b2 a1 and its three relatives. so the choice is four bytes:

the byte order comes next, from the byte-order magic in the section header. it is the number 0x1A2B3C4D written in the file's own order: 4d 3c 2b 1a on disk means little endian, 1a 2b 3c 4d big endian. a file can hold several sections, each with its own header, so the order can change in the middle of a file (a capture that two machines appended to each other). the reader keeps the byte order per section and forgets the interface list at every new section, because a packet's interface number only means something inside its own section.

timestamps

classic pcap has two variants, microseconds and nanoseconds, told apart by the magic number. pcapng lets each interface say it, with an option, if_tsresol, one byte. if the top bit is 0 the timestamp counts units of 10^-n seconds, and if it is 1, units of 2^-n seconds. if the option is missing the units are microseconds. a second option, if_tsoffset, is a number of seconds to add. the timestamp is a 64-bit count split in two 32-bit halves. the same moment, three seconds, is stored like this, and read back as the same 3,000,000,000 nanoseconds each time:

the reader turns every timestamp into integer nanoseconds, like the classic reader does, and the arithmetic is integers only:

there is a reason for no floats. a timestamp from the demo capture is 1,700,000,000,000,000,000 nanoseconds, a number of 61 bits, and a float keeps 53. a float that big can only step by 256 nanoseconds, so it would round every one of them, and two packets 100 nanoseconds apart could get the same time. integers keep them exact.

the last two lines are there because of a mistake I almost made. with a resolution of whole seconds, a 64-bit count can name a moment 585 billion years from now. the summary line turns a timestamp into a datetime, and a datetime stops at the year 9999. so a corrupt or hostile timestamp would make read raise an OverflowError on the printing, which the first rule of the project forbids. so the reader refuses a timestamp before 1970 or after 31 December 9999 with a PcapError, and one test checks that the last second it accepts can be printed and the next one cannot.

what it reads, what it skips, what it refuses

and it refuses, with a PcapError that says what is wrong: a block length that is under 12 (28 for a section header), not a multiple of four, or over 16 MiB; a length that is not repeated at the end of the block; a block that is cut short; a version other than 1; a packet for an interface the section never described; a captured length over 262,144 or past the end of its block; a known option of the wrong size, or one that runs past its block; a timestamp out of range. as in the classic reader, the packets before the damage are still yielded first, so read prints them and then reports the error.

a pcapng file can describe several interfaces, say an Ethernet card and a Wi-Fi card, with different link types. the rest of Sentinel decodes Ethernet only, so the file's link type is the first interface's, and a packet that arrives on an interface with a different link type stops the read with an error naming both. it does not decode those bytes as Ethernet by mistake, and it does not drop them without saying so.

checking it against something I did not write

my tests build their files by hand, block by block, from the description of the format, and not with the generator that I also wrote. that keeps a misreading of mine from cancelling out. but it is still me reading my own reading. so I also checked it against a package that another person wrote, python-pcapng, which I installed in a scratch folder for this one check. it is not a dependency of Sentinel and it is not part of the test suite.

it agreed on every packet. the attempts that failed while I wrote the check failed on how to call their writer (it refuses an empty packet, for one), not on the reader. what this does not show, at this stage: I had no file written by Wireshark itself, so a habit that only Wireshark has was not covered. stage 9 below closes that.

the fuzzer learned a second format

the fuzzer of stage 7 sends a damaged small capture file to the reader every fifth input. those files are now half classic and half pcapng, and some of them are one of each spliced together, so a header from one format sits in front of the blocks of the other. I ran 3,000,000 inputs (seed 8: 2,400,000 packets and 600,000 capture files): 0 failures. that is the same modest claim as in the first post: the reader only ever raises PcapError and everything after it stays well formed, and random damage will not find what needs three specific bytes to line up.

the tests do the systematic part: every prefix of a valid pcapng file, in both byte orders, is either read as a shorter capture or refused with a PcapError, and never gives back a packet that was not in the file. every byte of the file replaced in turn by three different values does the same. and random strings of 0 to 200 bytes behind a valid section header, behind a valid section and interface header, and on their own.

breaking it on purpose

the sabotage run has 57 new breaks for this stage, one line each: the byte order read from the wrong place, a block length that is not a multiple of four let through, base 2 read as base 10, an offset subtracted instead of added, an option that is not padded, an off-by-one on the interface number, and so on. all 57 are caught, and the 196 from stages 1 to 7 still were, so 253 of 253. 25 of the new breaks are caught by a single test.

none survived. but the first run hung on three of them, and that taught me something about a test, not about the code. the three breaks changed how the high half of a timestamp is read, so every packet got a wrong time and the test that compares the pcap and pcapng output of the 647-line benign capture failed. and when an assert a == b fails on two long texts, pytest tries to explain it by building a diff of them, which took minutes. mutate.py stops a run after 300 seconds and counts that as one failure, so the results said "1 failed of 0 run" and hid it. the code was fine and the test was the problem. the test now compares first and reports the number of the first line that differs. the three breaks fail in about six seconds (20 to 30 tests each), and I re-ran those three by hand to replace the timeouts.

what this stage does not do

a simple packet block has no timestamp, so it borrows the previous packet's. a file of only simple packets has every time at 0, and its flow durations are all zero. Wireshark cannot do better, since the time is not in the file, but it is worth knowing.

name resolution blocks (the addresses Wireshark saved next to their names) and interface statistics blocks (which hold the kernel's drop count) are skipped, not shown. comments, hashes and the packet flags are skipped too. a compressed file (.pcapng.gz) is not opened.

only Ethernet is decoded. a Linux capture from the any pseudo-interface (link type 113) is refused up front, with its link type in the message.

there is no pcapng writer in the package, only the generator in tools/gen_pcap.py (one section, one interface, enhanced packets). live --write still saves classic pcap. reading was the missing half, and I did not add a feature nobody asked for.

and, at the end of this stage, I had not read a file that Wireshark wrote. the reader agreed with the format description, with my hand-built files and with another library. that is a lot, and it is not the same thing. stage 9 fixes it.

##stage 9: comparing with tshark

everything so far was tested against traffic that I generated and against my own reading of the protocols, so a misreading of mine could sit in both. tshark, the command line of Wireshark, was written by other people and has read far more traffic than my generator will ever make. so this stage puts the two side by side, and fixes what differs. the rule from the plan was: fix every difference. what that came to is below.

a tool that compares two decoders

tools/compare_tshark.py is a development tool. Sentinel does not need it and does not depend on tshark. it runs tshark on a capture with reassembly, sequence analysis and defragmentation switched off, because read looks at one packet at a time, and with checksums verified. then it asks for about 75 fields per packet: the frame length, Ethernet and VLAN, ARP, IPv4, IPv6, TCP with its options, UDP, ICMP, DNS, HTTP and the TLS ClientHello. it turns each one into the form Sentinel has and compares. a field that only one of them reports is listed apart, and so is a difference in the packet count.

most of the work was in what "equal" means. tshark prints a 12-bit TCP flags field where Sentinel keeps nine bits, it counts the IPv4 fragment offset in units of eight bytes, it prints the window after scaling unless asked for the raw value, and the ethertype of a VLAN frame is in another field. none of that is a difference in the packets, so the tool translates it, and a test checks every translation.

the tool also compares conversations: the TCP and UDP streams tshark numbers, against the flows Sentinel finds (protocol, both ends and packet count, with the idle timeouts of Sentinel switched off since tshark has none).

ten captures

four are the captures my generator makes. six are public: small files from the Wireshark sample captures page, 1,245 packets and about 110 KB in all, that other people recorded on their own networks. I asked before downloading them, and they are not in the repository, which holds only traffic that I generated. one of them, 200722_win_scale_examples_anon.pcapng, is a pcapng that Wireshark itself wrote. the tool ran with Wireshark 4.6.8.

10 captures, 2,987 packets, 68,756 values compared, and no difference left. that sentence hides three fixes and 744 explained differences, and the order matters: the first run on the public captures had differences that were real.

three bugs that other people's traffic found

a checksum that the network card had not finished. the win_scale capture looks like one recorded on the sending machine. there the network card computes the TCP checksum after the capture point, so the field holds the sum of the pseudo-header alone, not complemented, and the card adds the rest. Sentinel reported 14 of the 26 TCP packets as bad. tshark calls them correct, and says why: "matches partial checksum, likely caused by TCP checksum offload". this was the known false positive that the first post listed as unsolved.

$ python -m sentinel read win_scale.pcapng   # before
20:22:25.360596 IP 192.168.200.135.6711 > 192.168.200.21.2000: Flags [S] [bad tcp checksum]
20:22:25.363991 IP 192.168.200.21.2000 > 192.168.200.135.6711: Flags [S.]
20:22:25.364075 IP 192.168.200.135.6711 > 192.168.200.21.2000: Flags [.] [bad tcp checksum]

the fix is small once the cause is known. for the first packet of that capture the pseudo-header is these six 16-bit words, and their folded sum is 0x1215, which is exactly what the packet holds in its checksum field:

$ python -m sentinel read win_scale.pcapng   # after
20:22:25.360596 IP 192.168.200.135.6711 > 192.168.200.21.2000: Flags [S]
20:22:25.363991 IP 192.168.200.21.2000 > 192.168.200.135.6711: Flags [S.]
20:22:25.364075 IP 192.168.200.135.6711 > 192.168.200.21.2000: Flags [.]

now a checksum that is exactly the partial sum is not reported as bad, for TCP and UDP, over IPv4 and IPv6. any other wrong value still is: the tests try the partial sum plus one, minus one, with the top bit flipped, complemented, for another length, and, for TCP, for other addresses. what is still not handled is an IPv4 header checksum that the card fills in.

the first fragment. decode stopped after the IP header of every fragment, on the argument that the transport header may be in another packet. but the first fragment holds it. ipv4frags.pcap is one ICMP echo cut in fragments:

$ python -m sentinel read ipv4frags.pcap   # before
2017-10-02 12:03:32.535132 IP 2.1.1.2 > 2.1.1.1: ip-proto-1, length 976
2017-10-02 12:03:32.535197 IP 2.1.1.2 > 2.1.1.1: ip-proto-1, length 432
2017-10-02 12:03:32.535641 IP 2.1.1.1 > 2.1.1.2: ICMP echo reply, id 5058, seq 1, length 1408

$ python -m sentinel read ipv4frags.pcap   # after
2017-10-02 12:03:32.535132 IP 2.1.1.2 > 2.1.1.1: ICMP echo request, id 5058, seq 1, length 976
2017-10-02 12:03:32.535197 IP 2.1.1.2 > 2.1.1.1: ip-proto-1, length 432
2017-10-02 12:03:32.535641 IP 2.1.1.1 > 2.1.1.2: ICMP echo reply, id 5058, seq 1, length 1408

now the first fragment is decoded, without verifying its checksum (the message is not whole), and the later ones still stop at the IP header, since they have no such header. the filters and the flows follow: tcp and port match a first fragment, and a first UDP fragment is a flow. this also matches what tshark shows.

a length that looked like an ethertype. a frame whose type field is below 0x0600 has a length there and an LLC header behind it: spanning tree, NetBIOS. vlan.cap, recorded in 1999, has a lot of them, and read printed the length as if it were a type:

$ python -m sentinel read vlan.cap   # frame 44
before: vlan 5, 00:20:18:62:73:a1 > 03:00:00:00:00:01, ethertype 0x00a6, length 184
after:  vlan 5, 00:20:18:62:73:a1 > 03:00:00:00:00:01, 802.3, length 184

that one is a wrong statement, not a missing feature, so it is fixed: it prints 802.3. the frame is still not decoded above Ethernet. ARP inside LLC and SNAP, which the same capture has, is not seen.

differences that are not disagreements

744 field differences are left, and none is a disagreement about a packet. each has a rule with a reason, and the report counts them by reason instead of dropping them, so a new kind of difference cannot hide among them. a rule covers one kind of difference and one group of fields: the tests check that a rule for "only Sentinel" does not explain "only tshark", and that a field which is different, not missing, is not explained.

Wireshark's own pcapng

stage 8 could only check the pcapng reader against files that I and a library wrote. this stage had Wireshark to write them. editcap -F pcapng converted 9 of the captures, and the reader gave back the same 2,961 packets as the classic files, timestamps included. and win_scale, a pcapng that Wireshark wrote, is in the table above: it compares clean.

does the tool notice anything?

a comparison that always says "equal" would look the same. so there is a test that runs the real tshark and changes one of Sentinel's fields before comparing (every TCP window becomes 1). the tool must report exactly that field, and once for each of the nine TCP segments of the demo capture. and the sabotage run has 46 new breaks for this stage, in the three fixes and in the tool itself (a flags mask that keeps too many bits, the fragment offset counted in units of 4, a rule that explains too much).

all 46 are caught, and so are the 253 before them: 299 of 299. but not at the first try. seven of the 46 survived the first run, and every one was in the tests of the tool: a rule that explained a difference of any kind and not just the kind it was written for, a count of compared values that took the fields of either side, a packet count that was reported only when tshark had fewer, a flow that only tshark has that was counted once however many there were, a line of tshark output with too many columns that was taken for a row, and two more like them. I tightened the tests and ran those seven again. 27 of the 46 are caught by a single test.

the fuzzer ran again, with the changed decoder: 3,000,000 inputs (seed 9), 0 failures. the tests of stage 9: 69 new, 796 in all.

what this stage does not show

it compares header fields and the number of packets in a conversation. it does not compare the bytes of a reassembled stream, or the content of an HTTP body. it ran against one version of tshark, and it is skipped where tshark is not installed, so CI does not run it. six public captures of about 110 KB are a small sample of what a network sends, and two of them are from 1999. and the two programs can be wrong in the same way, although they were written by different people, from the same documents.

one more thing it does not show is a capture of my own machine. everything public in it was recorded by someone else on some other network, which is why they are the right test for the parser and the wrong thing to keep in the repository.

##stage 10: two more detectors

the first four detectors see scans, floods, spoofed ARP and odd DNS. two things they cannot see: someone guessing SSH passwords, and data hidden in pings. this stage adds a detector for each, built the way the first four were: typed parameters with ranges, a rule in rules/default.toml, bounded state, one alert per attack and window, and the numbers behind the alert in the evidence.

$ python -m sentinel ids attacks.pcap --format text   # the last three of 16
2023-11-14T22:14:00.902000Z [medium] ssh-brute-force: 10.9.9.6 made 10 connections to the SSH port of 10.0.0.31 in 0.9s
2023-11-14T22:14:05.850000Z [medium] icmp-tunnel: 10.0.8.8 got 5 echo replies from 10.0.0.40 that do not repeat the data it sent, in 0.8s
2023-11-14T22:14:06.800000Z [medium] icmp-tunnel: 10.0.8.8 sent 10 echo requests of 512 bytes or more to 10.0.0.40 in 1.8s

what can be seen of a password guesser

SSH is encrypted, so a failed login cannot be seen. what can be seen is how often a client connects. a tool that opens a new connection for every few guesses shows that rate, and a person who logs in shows much less of it. so the detector counts connections per pair of client and server, and alerts at ten within a minute.

only connections that complete the handshake count, and that choice is the one I thought about most. a SYN with no answer is a scan, and a refused one is a scan too. those already have detectors, and a scan of port 22 should not also be reported as a guesser. so the detector remembers each SYN to the port, and counts a connection when it sees the ACK that ends its handshake. a retransmitted SYN is still one connection, and the ACKs that follow on the same connection do not count again, or every packet of an ordinary login would count as one more connection.

what can be seen of a tunnel in pings

a real ping is answered with exactly the data it sent, and it sends a few dozen bytes: 56 on Linux, 32 on Windows. a tunnel that hides data in echo messages sends a lot more, and what comes back is not what went out. so there are two signals: requests of 512 bytes or more (ten in a minute), and replies whose data differs from their request's (five in a minute).

the second one needs memory: to compare a reply with its request, the detector must keep what the request carried. it keeps an 8-byte digest of the data, not the data, because a hundred thousand requests of 64 kilobytes would be six gigabytes. it matches a reply to a request by the two addresses, the identifier and the sequence number, and a reply that matches nothing is ignored. a message that was cut short by a small snap length is ignored too: a cut request would look small and a cut reply would look changed, and both would be my mistake, not the network's. ICMPv6 is not decoded here, so a tunnel over IPv6 is not seen.

the thresholds are guesses

like those of stage 5, and I want to say so plainly: I had no capture of real SSH guessing or of a tunnel. what I can say is where my two generated captures sit. the harmless one has an admin who opens 9 SSH connections in 8 seconds, 9 pings of 1400 bytes (an MTU test) that are answered with the same data, and 4 replies that a device rewrote. the attack one has a guesser who makes 30 connections in 3 seconds, and a tunnel of 20 pings of 800 random bytes answered with 800 other random bytes. measured by asking the detectors with a higher and higher threshold, this is the most that one window holds:

the harmless capture sits one below the threshold in all three, on purpose, and the tests lower each threshold by one and check that the harmless capture then does raise exactly one alert, so it really is at the edge and not somewhere comfortable. the attack capture is far above. neither of them says where a real network sits.

real traffic, and one thing it showed

I ran the default rules over the six public captures of stage 9 (1,245 packets). neither new detector raised anything, which is a small test: none of them has SSH or large pings in it. but the run showed a false positive of an older detector. dns.cap raised one dns-tunnel alert, a lookup that looks like this:

that is an Active Directory client looking for a domain controller, and the label in the middle is a GUID. the entropy test reads a GUID as random, which it is, and calls it a tunnel. stage 5 tuned that threshold on names I generated, and this is the first real DNS traffic it met. I did not fix it in this stage, which is about two new detectors, and it is in the limits of the README. the fix is not a threshold: GUIDs have a recognisable shape, and the detector should know it.

a limit I had to pin in a test

every window in these detectors holds at most the threshold plus one event, so that a flood cannot use unbounded memory. the effect I had not thought about: when an attack goes on, the second alert, after the alert window has passed, says 11 connections and not 20, because the window never held more than 11. the count in an alert is the count the window could keep. I left it as it is, since the alert does what it should, and wrote the number down in a test so the next change does not move it by accident.

breaking it on purpose

the sabotage run has 50 new breaks for this stage: in the SSH detector (the source port watched instead of the destination port, a reset counted as the end of a handshake, the half-open table left to grow), in the ICMP detector (a size one byte short counted as large, a reply matched with its request by identifier only), in the packet view, in the rules file (each threshold moved by one) and in the generators (the harmless capture made to cross a threshold). they were grouped by the part they hit:

all 50 are caught, and so are the 299 before them: 349 of 349. one survived the first run: a break that made the SSH message say the threshold instead of the count. it lived because the first alert of a burst always has exactly as many connections as the threshold, so the message read the same either way. the test for a later alert checked the count in the evidence and the time in the message, but not the count in the message. now it does. 16 of the 50 are caught by a single test.

two sabotage entries of stage 5 broke in this stage, and not because of a bug: they matched two places in detectors.py instead of one, since I had copied the half-open bookkeeping of the SYN flood detector into the SSH one. each got the line that only the old detector has next to it. that is what copying costs. a shared helper would be cleaner, and I did not write it here.

the fuzzer ran again with the new detectors: 3,000,000 inputs (seed 10), 0 failures. the tests of this stage: 55 new, 851 in all.

what this stage does not show

it does not show that the thresholds are right for any network I do not have. backups and configuration tools open many SSH connections, and a run of large pings for an MTU test looks like the first signal, so both detectors will alert on things that are fine, and a rule can raise a threshold or add a filter. and it does not show that a real tunnel is caught: ptunnel and similar tools were not run, only my own idea of what they send.

##stage 11: telling clients apart by how they say hello

the ClientHello of stage 2 already gives the name a client asks for. that says who it talks to. it does not say what is talking, and that is a different question: the TLS library inside a program decides which cipher suites it offers, which extensions it sends and in which order, so one program says hello the same way whatever server it calls. a fingerprint of that is called JA3, and it is an MD5 of a few numbers from the hello. this stage adds it to read and flows, and a filter to search for it.

$ python -m sentinel read sample.pcap --filter "ja3 61279becc80ab0e3aca57f5913c3e1a0"
2023-11-14 22:13:20.018000 IP 10.0.0.1.43000 > 10.0.0.2.443: Flags [P.], seq 1, ack 1, win 64240, length 161: TLS ClientHello, sni example.com, versions [TLS 1.3, TLS 1.2], ciphers (15) [TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, +12 more], ja3 61279becc80ab0e3aca57f5913c3e1a0

what goes into it

five fields, written in decimal, each list in the order the client sent it and joined with dashes, the five joined with commas. for the hello above:

so the string is 771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-10-43,29-23-24, and its MD5 is 61279becc80ab0e3aca57f5913c3e1a0. the server name is not in it, and neither are the random bytes or the session id, and that is the point: the same program reaching two names has one fingerprint. I checked that with two hellos that differ in those three things, and checked that the same ciphers in another order give a different one (3c4d2ba2… against aac0f218…).

one detail decides whether the fingerprint is usable at all: GREASE. clients add made-up values (0x0a0a, 0x1a1a, up to 0xfafa) to their lists at random, to keep servers from breaking on values they do not know. they change on every connection, so a fingerprint that kept them would change with them. JA3 leaves them out of every list. the hello above has one in the ciphers and one in the extensions (the 6682 in its list is 0x1a1a), and a test draws GREASE twice and gets the same hash. values that only look like it, such as 0x0a1a, are real and stay.

a fingerprint of half a hello is worse than none

the parser of stage 2 was built to keep what it could read from a damaged hello: a hello cut off in the middle still gave its server name and the ciphers before the cut. that is right for a summary line and wrong for a hash, because the hash of a partial list is a valid-looking hash that belongs to no client. so TlsClientHello now says whether it is complete: nothing was cut, the extensions block is as long as it says, no extension is cut off, and both lists that JA3 needs were well formed. ja3 is empty otherwise. a test cuts a good hello at every possible position and checks that every proper prefix has no fingerprint.

this has a visible effect. the demo capture has a hello that arrives in two pieces on the wire. read sees one piece at a time, so its line ends with [truncated client hello: 81 of 152 handshake bytes] and has no ja3. flows puts the stream together first, and prints it:

$ python -m sentinel flows streams.pcap   # the same hello, in two pieces on the wire
tcp 10.0.0.1:45000 > 10.0.0.2:8443: established, pkts 4/1, bytes 161/0, 0.004000s [1 out-of-order client segment] | -> TLS ClientHello, sni example.com, versions [TLS 1.3, TLS 1.2], ciphers (15) [TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, +12 more], ja3 61279becc80ab0e3aca57f5913c3e1a0

finding one

--filter "ja3 HASH" works in read, flows, live and in the filter of a rule of ids. the hash is 32 hex digits in any case, and it prints back in lower case. a hash of another length, a word that is not hex, or no hash at all is an error that points at the place, like every other error of the filter language. a cut hello, another protocol or a packet with no hello does not match, so not ja3 X matches all of them.

checked against two things

the JA3 specification gives one example: a hello whose string is 769,47-53-5-10-49161-49162-49171-49172-50-56-19-4,0-10-11,23-24-25,0 must give ada70206e40642a3e4461f35503241d5, and it does. and tshark computes JA3 as well, so the comparison tool of stage 9 now compares it for every hello it sees, with the two lists. the demo capture and 6 hellos that a test builds byte by byte (GREASE in every list, no extensions, an empty extensions block, repeated values, values that only look like GREASE) agree: 203 field values, no difference. I had no capture of a real browser saying hello, so every hello here is one of mine and none is Firefox or Chrome. what a fingerprint of a real browser looks like, I have not seen.

breaking it on purpose

the sabotage run has 32 new breaks for this stage: GREASE kept in the lists, the record version used in place of the hello version, a space after each comma, the ciphers sorted, the last two fields swapped, SHA-1 in place of MD5, the hash written in capitals, each way a hello could be taken for whole when it is not, each malformed list let through, and in the filter each length and hex rule. grouped by the part they hit:

all 32 are caught, and so are the 349 before them: 381 of 381. but this run had survivors too: four of 383. two of them were not tests I was missing but code I did not need. the two breaks read a groups or a point formats extension that is too short to hold its own length as an empty list, and they lived because the line after it (2 + n > len(body)) rejects that case anyway, so the two branches were the same. I deleted the dead branch and the two breaks with it. the other two were real: a hello whose extensions block claims more bytes than it holds, and one whose last extension does, were taken for whole when every outer length was right. cutting a hello only gives those two when the lengths are made to lie, and my cut-at-every-position test cannot make them. two tests were added, and I ran the whole set again, 381 of 381. 9 of the 32 are caught by a single test.

the fuzzer ran again, with the new fields: 3,000,000 inputs (seed 11), 0 failures. the tests of this stage: 44 new, 895 in all. the cost is small: reading a hello takes 8.27 microseconds, computing its fingerprint 4.56, and it is only computed when something asks for it.

what this stage does not show

only the client side. JA3S, the same idea for the server hello, is not done, and neither is JA4, the newer form. JA3 depends on the order of the extensions, and some clients now shuffle it on every connection, so their JA3 changes each time. and a fingerprint tells software apart, it does not name a program or a person: two different tools that use the same library share one.

##stage 12: capture on Windows, if it can be trusted

the plan for this stage said "if it turns out to be reliable", so the first job was to find out. I develop on Windows, but the live capture of stage 6 needs Linux's AF_PACKET, and I only ever saw it work in CI. Windows has no such socket. the choices are a driver (Npcap, the one Wireshark installs), which is a dependency in a project that has none, or a raw socket switched to receive everything (SIO_RCVALL), which needs an Administrator prompt and nothing installed but sees less. I tried the raw socket first, and before writing any code I opened one on this machine and sent a few things through it.

asking the socket what it sees

the packets are whole IP packets, with no Ethernet header, and the length in the header was the length received for all 66 packets of the first try. the TCP checksums of the packets this machine sent were the unfinished sum that the network card completes (stage 9 already recognises those), and of the ones that came in, right. IPv6 gave nothing in that one test, so it is IPv4 only. Windows also reports no dropped packets, so a run that is too slow loses some and says nothing.

the ACK before the SYN-ACK

one thing was wrong, and I only saw it because I recorded the direction of each packet. in every TCP connection I looked at, seven of them, the ACK that completes the handshake was delivered before the SYN-ACK it answers. this is the start of a real run, three lines of 87:

$ python -m sentinel live 192.168.1.136 --write out.pcap --duration 5
2026-09-20 02:55:55.363998 IP 192.168.1.136.51293 > 192.168.1.1.80: Flags [S], seq 944683354, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
2026-09-20 02:55:55.364504 IP 192.168.1.136.51293 > 192.168.1.1.80: Flags [.], seq 944683355, ack 2354516038, win 255, length 0
2026-09-20 02:55:55.364588 IP 192.168.1.1.80 > 192.168.1.136.51293: Flags [S.], seq 2354516037, ack 944683355, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 2], length 0
...

the ACK, second line, acknowledges a sequence number that the third line has not shown yet. inside one direction the order is right. between the two directions, a packet that comes in is handed to the socket after the machine has answered it. I measured this on seven connections (one in the first probe, four in the third, two in a real run of Sentinel), and I did not find out why. the times are the moment of reading, so they show the same order and cannot be used to sort it out (the ACK is 84 microseconds ahead here). there is nothing honest to repair it with, so it stays, and it is in the README.

what it does to the rest: flows on the saved capture still says the connection is closed, 4 packets one way and 3 the other, and the detectors that follow a handshake read the ACK of the client, which comes after the client's own SYN either way. a test feeds 150 handshakes in this order to the default rules and gets no SYN-flood alert, and the same 150 SYNs without their ACK do raise one. but anyone who reads the lines will see it.

what changed in the program

very little. Capture only ever needed recvfrom, settimeout, getsockopt and close, so Windows is one more socket. IpSocket wraps it and puts a 14-byte Ethernet header in front of each IP packet: both addresses zero, because an invented MAC address would look like data, and the ethertype taken from the version nibble. anything that is not IPv4 or IPv6 goes on to the IPv4 parser, which refuses it with a message, so nothing is dropped and nothing raises. the wrapper costs 0.274 microseconds a packet. a file saved with --write has those zero addresses in it.

an interface is named by its IPv4 address, live 192.168.1.136, because bind needs one and Windows has no /sys/class/net; 127.0.0.1 is the loopback. open_capture picks the Linux socket if the system has AF_PACKET, the Windows one if it has SIO_RCVALL, and says so if it has neither. a name that is not an address, and a missing Administrator token, each have a message that says what to do.

I ran it for real once: live 192.168.1.136 --write out.pcap --duration 5 from an elevated prompt while the machine opened two connections to the router and sent a datagram and a ping. it captured 87 packets (the rest was the machine's other traffic), read on the file printed the same 87 lines, none of them has a bad checksum, flows counted the two connections as closed and ids said nothing.

the tests

45 tests run everywhere, with a stand-in for the socket: the header and the ethertype for each version nibble, an empty read, 2,000 random reads that must decode without raising, every IP packet of the four generated captures coming back the same above Ethernet and printing the same read line, the opening of the socket with each error and its message (and the socket closed after it), the names that are accepted and refused, and the handshakes above. 5 more run only on Windows as Administrator, on the real socket, on the loopback address: a datagram captured, decoded and seen exactly once, a TCP conversation with its payload once in each direction, and the command line saving a datagram. that file passed eight times in a row here. CI has a sixth job that runs it as Administrator and fails if the tests were skipped, and its first run on a GitHub runner passed four of the five Administrator tests. the fifth failed, and it was the machine and not the code: on the runner the IPv4 header checksum of a loopback packet is 0, left to the network card, so Sentinel calls it bad, as it does for any wrong checksum. on my machine the checksum is filled in, so my probes never showed it. I left the decoder alone and made the test accept that one anomaly, only when the checksum is exactly 0. the same test failed in the two Windows jobs of the main matrix too (Python 3.12 and 3.13), since the runner is an Administrator there as well. with that fix all six jobs passed: on the Windows runners the real socket captured a datagram, a TCP conversation and a command line, all five Administrator tests, in the Administrator job and in the two matrix jobs, and the Linux ones stayed green. this is what the job is for: one machine had told me everything was fine.

breaking it on purpose

24 new breaks for this stage: the ethertype always IPv4, the wrong nibble read as the version, a header one byte short, a drop count asked of Windows, a socket never switched to receive everything, every string accepted as an address, and so on. all 24 are caught on the first run. but one of the old ones was not: a stage 7 break that takes Windows out of the CI matrix survived. its test only looked for the word windows-latest in the workflow, and the new job of this stage has that word, so the test could not tell a matrix without Windows from one with it. now it checks the matrix line itself, and with that all 405 of 405 are caught, the 381 from the stages before and the 24 new. three of the old ones from stage 6 had to be rewritten because the code they broke had moved. 10 of the 24 new ones are caught by a single test.

the fuzzer ran again: 3,000,000 inputs (seed 12), 0 failures. the tests of this stage: 47 new, 942 in all.

what this stage does not show

IPv6, Ethernet addresses, a drop count, promiscuous mode, Wi-Fi monitor mode, and an interface named by anything but its address. that needs a driver, and it would be a different project. one machine, one Windows version and one network were tried, and the run was five seconds on a quiet link, so this says the raw socket is good enough to use, not that it is good.

##where this goes

the list was my plan, in the order I thought it should go. it is finished.

there was no stage 13 planned when I wrote this. what was left was the "limits" list of the README, and the honest top of it was that almost everything here was tested on traffic I made or on a handful of public files, and hardly at all on a busy real network. that became part 3.