part 3: sentinel meets real traffic

the last post ended with the honest top of the limits list: almost everything in Sentinel had been tested on traffic I made myself or on a handful of small public files, and hardly at all on a busy real network. this post is that test. I took real captures, some of them attacks made by other people and some of them ordinary traffic that nobody has labelled, and ran the detectors over them. I did not change a line of the program. the rules are the ones it ships with.

it is a different kind of stage from the others, because there is nothing to build and the interesting part is what I am allowed to conclude. so most of this post is about that.

##finding real traffic

this took longer than the measuring. what I wanted was labelled data, where someone says which packets are the attack, and the well-known datasets have it. they are also big. CIC-IDS2017 is about 50 GB and asks for a registration form. Kitsune has labels for every packet, including ARP spoofing and SYN floods, and its page lists files of 2 GB, but those are feature tables and not captures; the captures are 147 to 334 MB. the only way to get them is one zip of 19 GB, and the server answers a request for a part of a file with the whole file, so I could not take six files out of it. CTU-13 has captures that hold only the botnet's own traffic. on a slow connection, that was three ways of not getting it.

what I used instead is small, and I wrote each file down with its source before I fetched it:

7 files, 2,716,636 packets. two of them needed a trick. the backbone trace of MAWI is 3.9 GB compressed for 15 minutes, but that server does answer a request for a part of a file, and a gzip cut in the middle still opens up to the cut, so I took the first 60 MiB and got 2,163,268 packets, the last one cut in half. the file of the web server is a zip with a password, and the password is on the site's about page, as a picture.

one file I did not trust. the two cs3 captures come from a course repository that does not say where they come from, so before using them I looked inside with tshark: in the ARP file one MAC address (7c:d1:c3:94:9e:b8) answers for both the gateway and another host, which is what ARP spoofing looks like, and in the other 192.168.0.100 sends SYNs to 1,000 different ports of 192.168.0.101, which is what nmap's default scan looks like. they are what their names say. where they come from is still not known, and this post says so again at the end.

##do the detectors fire on real attacks?

yes, all five, and each raised only the alert it should. this is what the port scan and the ARP spoofing look like from the command line, the real output:

$ python -m sentinel ids --format text portscan.pcap
2014-02-25T05:11:20.956948Z [medium] port-scan: 192.168.0.100 probed 15 ports on 192.168.0.101 in 0.1s
2014-02-25T05:11:30.974438Z [medium] port-scan: 192.168.0.100 probed 46 ports on 192.168.0.101 in 1.4s
2014-02-25T05:11:41.015451Z [medium] port-scan: 192.168.0.100 probed 64 ports on 192.168.0.101 in 2.7s

$ python -m sentinel ids --format text arpspoofing.pcap
2014-02-25T06:11:11.070534Z [high] arp-spoof: 192.168.0.1 moved from f8:1a:67:cd:57:6e to 7c:d1:c3:94:9e:b8
2014-02-25T06:11:15.399497Z [high] arp-spoof: 192.168.0.1 moved from 7c:d1:c3:94:9e:b8 to f8:1a:67:cd:57:6e
2014-02-25T06:14:03.123374Z [high] arp-spoof: ARP says 192.168.0.103 is at d8:96:95:01:a5:c9, but the frame came from 7c:d1:c3:94:9e:b8

in the ARP file the alerts include 7c:d1:c3:94:9e:b8, the address that tshark showed claiming two hosts. the labels of these files are one per file, "this is a SYN flood", so the honest reading is small: on one real example of each attack the detector for it fires. the two StopDDoS files have nothing but attack traffic in them, so they cannot show a false alarm, and one file of each kind says nothing about how many attacks would be missed.

##traffic that nobody labelled

the other two captures are the ones I was afraid of: a week of what reaches a web server on the internet, and 14.6 seconds of a link that carries a gigabit. nobody can say which of their packets are attacks, so I cannot score the alerts as right or wrong. I asked tshark two things that I can check without trusting Sentinel. is what an alert says true: a port-scan alert says a source probed 30 hosts on one port within 10 seconds, so I count the SYNs again from tshark's own output. and does an independent recount of the same rule find scanners that Sentinel did not report.

a week at a web server

27 port-scan alerts and 1 DNS alert in 500,923 packets. tshark counts the same number of ports for all 27 of the alerts, and the recount finds 16 scanning sources, the same 16 that Sentinel reported. none missed, none extra. the one DNS alert is a random-looking name under icann.org, which I did not look into.

a gigabit for fourteen seconds

2,163,268 packets and 795 alerts: 627 port scans, 165 DNS tunnels and 3 SYN floods.

the port scans. 624 of them are one source probing one port on 30 or more hosts. they come from 376 sources, and the recount finds the same 376. all 626 alerts that can be counted again reach the threshold in tshark's count, 551 of them with exactly the number in the alert. in the others the recount is higher, and that fits something I could see in the code: a window holds at most four times the threshold in events, so the number in an alert is between 30 and 120, and 443 say exactly 30, whatever the size of the sweep. the alert is true, and it understates a fast sweep. the ports that were probed most were 4117, 23, 22, 80, 443, 34567, 8080, 2222: mostly telnet, ssh and web ports, and a few odd ones such as 4117 and 34567.

the state limit. the last port-scan alert is not a scan:

every detector keeps bounded state, and reports it when it has to forget something. one table of the port-scan detector, the sources it is following, reached its 100,000 entries. that took fourteen seconds on this link. Sentinel said so, as it was written to in stage 5, and the recount found no scanning source that it had failed to report, so the limit did not hide a scanner from this slice. on a longer one it could.

the DNS alerts. all 165 come from one check: twenty "no such name" answers to one client within a minute. tshark confirms all 165, and shows 31,006 such answers to 1,815 clients in those 14.6 seconds, one client with 757. none of the other signals, a long name, a random-looking name or many different subdomains, fired once. so these are not tunnels, I think: on a backbone a client is often a resolver or a gateway that asks for many names for many people, and a threshold made for one client fires at once. I cannot prove it, because nobody labelled it.

the SYN floods. three alerts, all for one address and one port, 131.112.115.241:993 (IMAPS): 100 SYNs from 16, 18 and 22 addresses within a second, and tshark counts the same. over the whole slice 844 SYNs from 110 addresses went to that port, and no SYN-ACK from it was captured. that could be an attack, or clients trying again and again against a mail server that is down. the capture cannot tell me which. the link does not help: only 5.3% of the SYNs on it have a SYN-ACK in the capture (57.5% for port 443), because a backbone carries only some of the return paths.

a mistake of mine

the first recount told me Sentinel had missed a scanner. 202.121.34.125 looked like a source of SYNs to 30 hosts within 10 seconds, and Sentinel had not reported it. I looked at its packets: they were ip:icmp:ip:tcp. it was a router. what it sent were ICMP errors that quote the packet they refuse, and my tshark filter counted the SYN inside the quote as if the router had sent it. Sentinel ignores those, and it was right. I fixed the filter, and with it, and IPv6 counted, the two lists of sources are the same. it would have been easy to write "one scanner missed" and move on, and the number would have looked like a finding.

speed

50,394 packets a second through the detectors on the backbone slice (35,073 on the week at the web server, 30,618 on the SYN flood), on the same machine as the benchmark of stage 7. the link carried about 148,000 packets a second, so Sentinel ran at a third of real time. pure Python on one core. for a gigabit link that is a limit, and the README says so.

##what this does not show

how many packets of an attack Sentinel got right, and a false alarm rate. the labels are per file, and the two big captures have none, so what I checked is that what the alerts say is true, and not that each one is an attack. it is one slice of one link and one week at one server. the backbone is not symmetric, so "unanswered" cannot be read as "a scan", and a detector that waits for a handshake sees half of it. there is one ARP spoofing file, one iodine capture, and no SYN flood with ordinary traffic in it. where the cs3 files come from is not known.

and I changed nothing. what I found is three limits: a DNS check whose threshold suits one client and not a backbone, a state limit that a backbone reaches in fourteen seconds, and a count in a port-scan alert that stops at four times the threshold. I could turn a number down until this slice looked clean, but that would only tune Sentinel to fourteen seconds of one link. they are in the limits list of the README, where the next person will find them.

##where this goes

there is no stage 14 planned. what would make this stronger is data with a label for every packet, so that "fires" can become "gets right, this many times". Kitsune has it, and it is 19 GB behind a server that will not hand over a part. if I find a mirror that does, that is the next measurement.