axa 3.0.1
Farsight Security Advanced Exchange Access (AXA)
|
This is the Advanced Exchange Access (AXA) toolkit. It contains tools and a C library to bring DomainTools' real-time data and services directly from the Farsight Security Information Exchange (SIE) to the subscriber's network edge.
AXA-based solutions are often preferable over procuring a direct, physical connection to the SIE, usually via a co-located blade.
In order to use the tools and C library to access DomainTools's data, you must have a previously provisioned account. If are you not yet a customer, but are interested, please reach out to DomainTools Technical Support for more information.
AXA enables subscribers to connect to Farsight's subscription-based SRA (SIE Remote Access) and RAD (Real-time Anomaly Detector) servers. These servers provide access to data and services built from Farsight's SIE. SRA streams real-time SIE data while RAD streams real-time anomaly detection data.
The axa-tools distribution contains the following:
sratool
: A test/debug/instructional command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data.radtool
: A test/debug/instructional command-line tool used to connect to a RAD server, set watches, enable anomaly detection modules, and stream data.sratunnel
: A production-quality command-line tool that streams SIE data to the local network.radtunnel
: A production-quality command-line tool that streams anomaly data to the local network.libaxa
: A C library providing an API for the AXA protocol including:For usage details on sratool
, radtool
, sratunnel
, and radtunnel
, please see their respective man pages (included in the distribution).
Note that "AXA" is a bit of an overloaded term. Depending on the context, it can refer to actual AXA wire protocol, the C API, the suite of tools presented here, or even a clustered set of SRA and RAD servers. In this document, when apropos, context is provided in order to disambiguate these situations.
Some of the channels offered by the SIE network burst to an extremely high bitrate (some over 500Mbps). AXA has two ways to deal with such network-hungry situations: optional filtering and loss-tolerance built into the protocol.
Filtering can take one of the following forms:
Finally, AXA is a deliberately lossy protocol. If a subscriber requests more data than the network can carry, data overruns will occur. When this happens, loss markers are transmitted reliably within the AXA stream to inform the subscriber via the AXA accounting subsystem. At this point, the subscriber's possible mitigation strategies include:
AXA can built manually from source or, on Debian systems, installed by using pre-built packages.
AXA should compile on most modern Unix-like operating systems with little or no modiciation. The following (very complete) guide will walk you through getting it built.
This guide assumes the following support programs are installed (most of the which can be installed via your operating system's package management system). If you need help with any of the following, please reach out to DomainTools.
The AXA suite has the following external library dependencies that must be installed in the order they appear. Please note that, while in most cases the latest version of a depedency is preferred, sometimes an older version is required.
strlcpy()
(should already be installed on BSDish systems)Optional dependencies:
After satisfying the above, build, axa:
If you installed libcheck you can run the unit tests by the following command:
Please report any errors to DomainTools.
If you installed Doxygen you can build the API documentation (including an HTMLized version of this document):
The HTML documentation will be in the html
directory and can be rendered in any modern browser.
Congratulations, you're ready to stream data! You can proceed to the "AXA Transport Layer" section below.
The binary packages of AXA and its dependencies are available from a Debian package repository maintained by DomainTools. These packages should be used instead of building from source on Debian-based systems.
To install the AXA Tools sratool
, radtool
, sratunnel
, radtunnel
:
To install AXA development files (if you wish to use the libaxa C API):
Once axa-tools is installed, you'll need to work with your DomainTools account manager to obtain your access credentials. In most cases, this will be an apikey.
AXA offers an encrypted transport for setting up sessions and tunneling data:
ECDHE-RSA-AES256-GCM-SHA384
suite offering "currently infeasible to break" encryption and perfect forward secrecy.AXA compresses all nmsgs before transmission across the network using nmsg's built-in compression capability. IP packets are not compressed.
Your DomainTools account manager will provide you with an alphanumeric apikey string. This apikey is used to both identify you and authenticate your session to the AXA servers.
The AXA apikey transport listens at the following URIs:
apikey:<your_apikey_here>@axa-sie.domaintools.com,49500
apikey:<your_apikey_here>@axa.sie-remote.net,1012
SRA listens on TCP/49500
and RAD listens on TCP/1012
and both transit standard TLS data.
You can connect as per the following:
Connecting via sratool
Connecting via radtool
Connecting via sratunnel
Connecting via radtunnel
AXA supports a subscriber-side configuration file used as a convenience to specify session defaults. By default AXA will look for it ~/.axa/config
. Currently it is used to store "connection aliases" that provide a facility to create shortcut mnemonics to specify the AXA server connection string. This is especially useful for long connection strings associated with the apikey transport. As it can contain sensitive information, if it exists, the file must be readable/writable only by "owner" or AXA-based tools will refuse to load.
For example:
After creating the above aliases, you can replace the server connection URI with your shortcut name as per the following:
The AXA config file is shared for sratool
, radtool
, sratunnel
, radtunnel
, and is available via the API. As such, all of the tools (including the ones you may build) have access to these alias shortcuts.
The following are a few examples of how to use sratool
, sratunnel
and radtool
.
Here's a simple example using sratool
to stream five nmsgs seen on the Newly Observed Hostnames (NOH) channel (channel 213):
sra> connect sra-apikey
: we connected to an SRA server using the apikey transport. Under the hood, sratool
consulted the AXA config file to search for the "sra-apikey" alias and extract the server connection URI. The HELLO
response from the remote end provides its version number and the protocol level.sra> count 5
: we asked our sratool
client to stop after emitting five messages.sra> channel 213 on
: we then asked the remote end to listen to SIE channel 213.sra> 1 watch ch=213
: we then asked to watch all content on channel 213 (with no rate limiting or filtering), which is a common choice for 213 since its volume is low (~200kbps).The following example shows how to use sratunnel
to stream nmsgs from NOH to a file and then read the resultant nmsg file with nmsgtool display a single nmsg record as new line delimited JSON using the jq program.
sratunnel -s sra-apikey -c 213 -w ch=213 -o nmsg:file:213.nmsg
: we invoked sratunnel and connected to SRA using the apikey transport. Channel 213 is enabled, and a 213 "all watch" is set. Finally, nmsgs are written to a file. The program runs for some time then we kill it via ctrl-c.$ nmsgtool -c 1 -r 213.nmsg -J - -- | jq .
: The nmsgtool
program is run to read a single nmsg from the output file and pipeline to the jq program to pretty print it.Next, radtool
is used to load the Brand anomaly module to watch for suspected brand infringement in the Internationalized Domain Names (IDN) namespace for four well-known brands.
rad> connect rad-apikey
: We connected to RAD over the apikey transport using the "rad-apikey" alias. The HELLO
response from the remote end tells us its version number and the protocol level.rad> verbose on
: We turn on verbose mode to get more information about each hit.rad> 1 watch dns=*.
: We set a DNS wildcard "all-watch". This will match all dns hostnames which is what we want for the Brand module – we want to look at the entire DNS namespace (except for four level domains, as per below).rad> 1 anomaly brand brand=facebook,apple,netflix,google matcher=idn_homoglyph whitelist=*.facebook.com,*.apple.com,*.netflix.com,*.google.com
: We switched on the anomaly detector. This command enables the brand anomaly module looking for Internationalized Domain Names (IDNs) "suspiciously close" to "facebook, google, apple", or "netflix". Hostnames in the *.facebook.com
, *.apple.com
, *.netflix.com
, and *.google.com
are considered safe and are ignored.$ idn -u xn--fcebook-s3a.com.
: We use the GNU libidn idn command line tool to convert the punycode encoded domain into its Unicode representation.With sratunnel
and nmsgtool
, you can create your own local "SIE node". This can be useful if you want to cobble together your own local SIE cloud with the channels you're subscribed to. First, invoke sratunnel
as per the following:
Before continuing, we confirm traffic flowing by checking with tcpdump
(on OS X this looks like):
Finally, we use nmsgtool
to receive and decapsulate the data. Here we invoke it to capture and display three payloads:
sratunnel -s sra-apikey -c 204 -w ch=204 -o nmsg:127.0.0.1,8000 &
: invoke sratunnel
and stream all of channel 204 (Passive DNS traffic that has been de-duplicated, filtered, and verified). The output is set to be nmsgs to the loopback interface on port UDP/8000 (UDP is the default nmsg transport protocol).tcpdump -n -c 3 -i lo udp port 8000
: tcpdump is used to verify packets are being received.nmsgtool -l 127.0.0.1/8000 -c 3
: nmsgtool
is invoked to listen on the loopback interface on UDP/8000
and three nmsgs are emitted to stdout.Both sratunnel
and radtunnel
support a "kicker" feature where it will continuously rotate the output file and optionally run a command on the rotated file. In this mode output file names are suffixed with a timestamp and {rad,sra}tunnel
runs continuously, rotating output files as payload counts or time intervals expire. Optionally, a shell command may be specified to run against files after rotation (if the command is set to an empty string '', then no command is executed and only file rotation is performed).
sratunnel -s sra-apikey -c ch213 -w ch=213 -o nmsg:file_json:213.jsonl -C 100 -k gzip -dd
: We invoked sratunnel
against channel 213 and asked for the kickfile option to run the gzip
command on each output file after every 100 watch hits.The AXA protocol consists of a pair of streams of messages between a subscriber's client (such as sratool
) and an AXA server (such as SRA), one stream in each direction, currently over a single TCP connection.
AXA must sit on top of a reliable stream protocol such as TCP and so has no provisions to detect or recover from duplicate, out-of-order, lost, or partially lost data. Note that SIE data can be lost before encapsulation into AXA protocol messages due to issues such as network congestion, CPU overload, etc.
AXA relies on the TLS and TCP protocols to provide authentication, confidentiality, and integrity as shown below.
The authoritative definition of the AXA protocol is contained in the axa/protocol.h file.
Values that originate in SRA or RAD servers such as message lengths use little endian byte order in the AXA protocol. Other values such as IP addresses and port numbers are big endian for consistency with their sources such as host tables. SRA and RAD data such as nmsg messages and IP packets have their original byte orders.
The stream protocols below the AXA protocol are responsible for authentication and authorization. An AXA client and server pair on a computer can use unadorned TCP through the loop-back interface or use a UNIX domain socket. The AXA protocol assumes this is safe.
The AXA client starts by waiting for an AXA_P_OP_HELLO
message from the server. Over a local stream, the client then sends an AXA_P_OP_USER
message to tell the server which parameters to use.
Every AXA message starts with the following 8 octet header:
len
: A 32-bit value covering the entire length of the AXA message including the header. Many AXA messages are variable length.tag
: A 16-bit identifier used to uniquely "tag" specific events during the lifetime of an AXA session. To refer to these events, the client or server will use this tag. Some AXA messages do not require tags, in that case the tag field should be AXA_TAG_NONE
. Required tags must be unique during the lifetime of the corresponding client request. Some client requests such as a "watch" can last indefinitely and will elicit many server responses all with the same tag.pvers
: An 8-bit protocol version number that allows AXA clients and servers of different ages to find a mutually compatible version of the AXA protocol.op
: The 8-bit op (opcode) specifies an operation requested by the client, a response from the server, or data from the server. The universe of opcodes is discussed below.For a detailed discussions of the AXA protocol message types, see the doxygen generated page for protocol.h
.
The following is an AXA protocol quick reference chart intended for application developers building libaxa programs.
AXA_TAG_NONE
, as described above.OPCODE | VAL | SENT BY | TAG | DESCRIPTION |
---|---|---|---|---|
AXA_P_OP_NOP | 0 | CLIENT / SERVER | NO | carries no data, is intended only to ensure that the TCP connection is still up |
AXA_P_OP_HELLO | 1 | SERVER | NO | helps the client choose a compatible AXA protocol version |
AXA_P_OP_OK | 2 | SERVER | YES | indicates the success of the preceding client request with the same tag |
AXA_P_OP_ERROR | 3 | SERVER | YES | indicates the failure of a preceding client request with the same tag |
AXA_P_OP_MISSED | 4 | SERVER | NO | carries details about data or packet loss due to rate limiting or network congestion |
AXA_P_OP_WHIT | 5 | SERVER (SRA) | YES | reports a "watch hit" or packet or nmsg message that matched an SRA watch with the same tag |
AXA_P_OP_WLIST | 6 | SERVER (SRA) | YES | reports a current watch in response to AXA_P_OP_WGET from the client referenced by tag |
AXA_P_OP_AHIT | 7 | SERVER (RAD) | YES | reports an "anomaly hit" or packet or nmsg message detected by a set of anomaly detector |
AXA_P_OP_ALIST | 8 | SERVER (RAD) | YES | reports a current anomaly detector in response to AXA_P_OP_AGET |
AXA_P_OP_CLIST | 9 | SERVER (SRA) | NO | reports the on/off state and specification of an SRA channel |
AXA_P_OP_USER | 129 | CLIENT | NO | indicates the AXA protocol is used over a local stream and rejected otherwise |
AXA_P_OP_JOIN | 130 | CLIENT | NO | used to bundle TCP connections |
AXA_P_OP_PAUSE | 131 | CLIENT | NO | ask the server to temporarily stop sending packets or nmsg messages |
AXA_P_OP_GO | 132 | CLIENT | NO | ask the server to resume sending packets or nmsg messages |
AXA_P_OP_WATCH | 133 | CLIENT | NO | specify interesting packets or nmsg messages |
AXA_P_OP_WGET | 134 | SERVER (SRA) | – | requests one (with specified tag) or all (tag 0) current watches in AXA_P_OP_WLIST messages |
AXA_P_OP_ANOM | 135 | SERVER (RAD) | YES | specify an anomaly detector |
AXA_P_OP_AGET | 136 | SERVER (RAD) | – | requests one or all current anomaly detectors |
AXA_P_OP_STOP | 137 | CLIENT | NO | ask the server to delete the watch or anomaly detector by tag |
AXA_P_OP_ALL_STOP | 138 | CLIENT | NO | ask the server to delete all watches or anomaly detectors |
AXA_P_OP_CHANNEL | 139 | CLIENT (SRA) | NO | tell the SRA server to enable or disable one channel or all channels |
AXA_P_OP_CGET | 140 | CLIENT (SRA) | NO | get the specifications and states of all channels |
AXA_P_OP_OPT | 141 | CLIENT / SERVER | NO | set various options (rate limiting) report rate limit, how much has been used |
AXA_P_OP_ACCT | 142 | CLIENT / SERVER | NO | request accounting information |
AXA_P_OP_RADU | 143 | SERVER (RAD) | NO | request RAD Unit balance |
See the JSON schema describing the JSON output format.
For a detailed walkthrough of a "Hello World", AXA-style, please see the following AXA API Reference:
The online AXA User Guide and AXA Technical Overview are at: