In this article we take a look at the avax-python network_listener.py utility and how to use it to snoop on Avalanche AVAX network traffic.
tl;dr;
If you’re in a hurry, here’s how to quickly start capturing Avalanche AVAX network traffic.
git clone https://github.com/cryptobi/avax-python.git
cd avax-python
. setenv.sh
python3 bin/network_listener.pyavax-python requires the following libraries. If you get a missing library error message, then you need to install these first:
sudo pip3 install numpy
sudo pip3 install pyopenssl
sudo pip3 install cryptography
sudo pip3 install asn1crypto
sudo pip3 install plyvelIf everything goes as planned, you should now see Avalanche network traffic messages scroll down your window, like so:
Message Types
Avalanche message types may be roughly categorized into two groups: network-only messages and consensus messages.
Network-only messages deal with protocol events such as getting and sending peer lists, pinging other peers and so on.
Consensus messages, on the other hand, require more processing. This is where all the Avalanche magic happens.
Once a consensus message is received, the network layer peels out the network wrapper and processes the core, which usually comes in a ContainerBytes field. Therefore, consensus messages are binary messages within network messages.
How does avax-python handle these? Using custom handlers.
Handlers
While migrating Golang code to Python, we found that maybe, since the Py version isn’t meant for production (yet?), we thought we could add a layer between the network events and the actions they trigger. So, instead of calling Ping, Pong, Get, Put and so on directly, we implement a Handler interface which then receives these messages from the network layer.
The Golang implementation is meant to be the reference code for all of Avalanche. So they have all the protocol rules hard-coded in Go. Since avax-python lives on the hobbyist/experimental side of things, we decided that the protocol should be plugged in. For this, we use handlers that respond to each message as it is received.
Our first experiment was to scrape Avalanche network peers using this technique. We simply implemented a custom network handler which ignores all messages except peers notifications. Note that network handlers should not ignore handshake messages, as peers will likely terminate the connection unless the handshake is performed. So, to e accurate, this handler ignores all messages except for handshake and peers.
Next, we have the consensus message handlers.
As mentioned earlier, these take a bit more work to implement, since they must first parse the network messages, then parse the consensus payload within them. In a different article, we presented the Avalanche message pipeline in avax-python.
Avalanche isn’t a single blockchain.
It’s a platform where multiple blockchains and virtual machines can be implemented. You could have Bitcoin and Ethereum both running inside Avalanche. In fact, Ethereum already is implemented inside Avalanche! So, while Bitcoin developers may easily process every message received assuming it’s meant for Bitcoin itself, Avalanche developers must route messages to the correct consensus engine and virtual machine.
Basically, the network message payload contains information about which chain it’s is meant for. A chain is linked to a consensus engine and virtual machine. Once the message reaches the correct engine and virtual machine, it gets parsed into its components (vertices, blocks, transactions, inputs, outputs, NFT’s and so on).
Network Handler
For this article, we want to intercept vertices, blocks transactions, inputs, outputs and print them out to the console.
First, we implement the network handler, which is pretty simple. Just handle each network message by name and print out its components. You can see an example here.
Consensus Handler
The consensus handler itself is pretty simple. All the message routing complexity happens underneath it, before messages reach the handler as per the message pipeline.
Once messages reach the consensus handler, you can do whatever you wish. The Golang implementation, of course, decides whether messages are valid, check their inputs, outputs, resolve all dependencies and then decide whether they accept the transactions or not.
avax-python does not perform validation (yet). We simply listen to messages and provide an interface to handle them.
So, for this article, we implemented a trivial JSON printer handler which simply prints out blocks, vertices, inputs, outputs and so on. Here a block was just received from the network:
The above CommonDecisionBlock arrived inside a Put network message. The network handler noticed it was a consensus message and passed the message up to the correct chain handler via a chain router. The chain handler then passed the payload to a consensus engine (Snowman).
Snowman then parsed the block and ended up with a list of binary payloads representing transactions. Here, once more, a binary payload is contained within an outer layer. Just like network messages peeled out the network part to reach the consensus binary payload, here we strip out the block headers to reach the binary transactions, which is where all the goodies are contained.
Snowman then requested the ChainVM virtual machine parse the transactions contained inside the Block. The results were plugged back into the Block structure which was then serialized by our custom JSONPrinter consensus handler. As implied by the name, a JSONPrinter is a very trivial consensus handler which simply prints out messages as received from the network.
Custom Handler Ideas
You could do all sorts of interesting things using custom consensus handlers.
For instance, you could use avax-python to simply read messages off the network and push them into Kafka, using a cluster of avax-python Docker images.
Using Kafka and delegating the consensus work to other nodes, you could have a fully distributed Avalanche implementation instead of discrete avalanchego nodes. The networking and consensus components could be implemented as microservices. This could come in handy in the future when Avalanche processes thousands of transactions per second. For current payloads, a single monolithic process is just fine.
The future is bright for Avalanche! Keep hacking!
What Next?
Next up, we need to implement the Ethereum side of things, so we can parse NFT’s and ERC20 messages from the C-Chain.
In the meantime, implementing P-Chain block and X-Chain vertex explorers can be easily implemented by simply reading the JSON output from the network listener!
Links
Custom network handler example – scrape network peers