Quantitative Trading

A Different Way of Looking at Returns

Sun, 03 Aug 2025 00:00:00 +0000

It would be nice if it were possible to trade a moving average cross. The problem with this is always that the data lags. It’s not possible to trade the current value of a moving average since it requires trading prices in the past. The advantage to doing so is that, due to the smoothing, forecasts are less noisy.

The good thing is than many moving averages are finite impulse response (FIR) filters, meaning they are the dot product of prices and weights. But can we compare the current price to a moving average? Well, yes, so long as it is an average in the future.

Implementation

The problem with log returns

\[r_{t} = ln\frac{price_{t}}{price_{t-1}}\]

is that they are very noisy. But let’s say we want to buy a position today and then sell it bit by bit over the next 7 days. What would that look like?

Using a filter like this, we can see the effective returns of such a strategy.

[1.000, -0.143, -0.143, -0.143, -0.143, -0.143, -0.143, -0.143]

This vector represents the amount of trading we will do at each time step. The position over time is given by the cumulative sum.

[1.000, 0.857, 0.714, 0.571, 0.429, 0.286, 0.143, 0.000]

You can see the position holdings are similar to buying on day one and then selling a little bit over the next days. You can be creative with this and say also buy over X, hold for Y days, and then sell over Z days. The world is your oyster.

Code

import ccxt
import pandas as pd
import numpy as np
from datetime import datetime
from plotly.subplots import make_subplots
from numpy.typing import NDArray

def make_mask(entry_len: int, wait_time: int, exit_len: int) -> NDArray[np.float64]:
    entry_ma_weights = [-1] * entry_len
    exit_ema_weights = [+1] * exit_len
    
    mask = np.array(list(np.divide(entry_ma_weights, np.abs(np.sum(entry_ma_weights))))
        + ([0] * wait_time) + 
                    list(np.divide(exit_ema_weights, np.abs(np.sum(exit_ema_weights)))))
    return mask

def rolling_dot(series: NDArray[np.float64], mask: NDArray[np.float64]) -> NDArray[np.float64]:
    n = len(mask)
    if n > len(series):
        raise ValueError("mask longer than series")

    # Construct sliding windows
    strided = np.lib.stride_tricks.sliding_window_view(series, n)
    # Each row is one window, so we do row-wise dot
    res = strided @ mask

    output = np.empty(len(series))
    output[-(len(mask)-1):] = np.nan
    output[:len(res)] = res
    return output

# Download historical daily data
exchange = ccxt.binance()
symbol = 'BTC/USDT'
timeframe = '1d'

# Fetch daily OHLCV (timestamp, open, high, low, close, volume)
ohlcv = exchange.fetch_ohlcv(symbol, timeframe=timeframe, limit=500)

# Convert to DataFrame
df = pd.DataFrame(ohlcv, columns=['timestamp', 'open', 'high', 'low', 'close', 'volume'])
df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
df.set_index('timestamp', inplace=True)

# compute returns
mask = make_mask(1, 0, 7)
df["convolved"] = rolling_dot(df["close"], mask) 

subtitles = [f"{symbol} Prices"]
nrows = len(subtitles)
specs = [[{"secondary_y": True}]] * nrows
fig = make_subplots(rows=nrows, cols=1, specs=specs, shared_xaxes=True, subplot_titles=subtitles)

# Add close price trace (primary y-axis)
fig.add_scatter(
    x=df.index, y=df['close'],
    name='Close Price',
    row=1,col=1,
    line=dict(color='blue')
)

# Add convolved signal (secondary y-axis)
fig.add_scatter(
    x=df.index, y=df['convolved'],
    name='Returns',
    line=dict(color='red', dash='dash'),
    row=1,col=1,
    secondary_y=True
)

fig.show()

Strategy Returns

This is the resulting plot from the code above.

Variations on a Theme

Since this is easy to apply with any FIR-style filter, the position doesn’t need to be closed using an SMA. It is also easy to use triangle filters, an EMA, or whatever gives the strongest ability to forecast forward returns. The main advantage of this technique is that the noise of returns is reduced by using a smoothed exit. This means that the exit price is less dependent on chance, and it should be easier to find profitable opportunities.

It should also be noted that the autocorrelation of the returns should be avoided, as the returns themselves are not really known until much later. Forecasting should be done with signals independent of the return series.

Forecasting

The ultimate aim of this is to be able to predict returns using a set of features. This might be momentum, price extension, OI, funding, you name it. The hope is that using these returns as a dependent variable should make it much easier for machine learning to find something stable out of sample.

Trading

If you are holding positions for N periods, but forecasting every period, then how do you practically do that? Let’s say you have a filter that works over an 8-day period. Each period, you are rolling the position vector along one and then adding the position fixed vector. Then just check the current position and the first target position value, and trade the difference.

position = np.array([1.000, 0.857, 0.714, 0.571, 0.429, 0.286, 0.143, 0.000])

# Create a vector of length 14 to accumulate contributions
agg = np.zeros(14)

# Apply the position vector starting on days 0 to 6
for i in range(7):
    agg[i:i+8] += position

# Display the result
np.set_printoptions(precision=3, suppress=True)
print(agg)

Final Note

This was just a quick article about something I thought others might find helpful in case they were struggling with predicting returns. There is an old adage that if you get a good entry, the exit should be easy. This is a variation on that theme. As always, I appreciate the feedback, so please let me know if you found this useful.

Dealing with Data

Tue, 15 Jul 2025 00:00:00 +0000

It is always interesting to get feedback from people about my blog articles. One that a few people have talked to me about is this article on message rates and system throughput. It covers the importance of throughput for market making and how it relates to system latency. If you have a very performant system then likely you will not have issues, however that is only one possible solution.

The aim of this article is to elaborate on the topic of throughput, how to make a system more robust, how to avoid bottlenecks. It is important for specific types of trading which deal with vast quantities of data. Even if you have a fast system there are still design decisions that you might want to make to reduce the possibility of bottlenecks.

System Architecture

One important note about event driven HFT architectures is that they rely on a simple principle based on pipelines. Each system component has its own unique tasks, be it message parsing, book building, alpha generation, order management etc. Each one of these tasks is normally bound to a cpu/core and the communication done via some form of lockfree IPC.


flowchart LR
    A[WebSocket<br><sub>Core 0</sub>]
    B[Message Parsing<br><sub>Core 0</sub>]
    C[Alpha Generation<br><sub>Core 1</sub>]
    D[Order Management<br><sub>Core 2</sub>]
    E[OMS<br><sub>Core 3</sub>]

    A -->|IPC| B
    B -->|IPC| C
    C -->|IPC| D
    D -->|IPC| E

This is a simple diagram of a trading system architecture. The graph is directional and each component can be cpu bound to make the whole thing more cache efficient. It is also normal that each step in the pipeline is more complex than the previous one. This point will become more important later when we talk about decoupling. It’s important that the order management part is single threaded since this means there is less likely to be race conditions between orders placement and risk calculations. The last thing you want is to send taker orders when there are already fills that have not been processed.

Reducing bottlenecks

Reduce complexity

I know a lot of people who use python for HFT (beatz I’m looking at you). This is fine so long as you know what you are doing and make sure the average processing time is appropriate for the data ingestion rate. The simplest solution to make the amount of data manageable is to change the subscriptions. If you are trading low liquidity alt-coins, there is likely no need for a highly optimised trading system. (note. some smaller cap coins have insane volumes when they are pumping) There are aggregate trade feeds and gated order book feeds that are much less demanding to process. Since order books and trades are the majority of data, if the goal is to reduce the influx of data this is the best place to start.

Not all data needs to be process equally

Market data generally has two main types, deltas and snapshots. What does this mean and why does this matter? If you are listening to an L3 order book feed every update is a change to the order book (insert, cancel etc). These are deltas (changes) and to have correct picture of the order book, all the deltas need to be processed. This is the same for trades since each trade is its own unique event. So for this reason, it is hard to aggregate or skip some of these updates. To deal with delta style feeds, generally the code will need to be fast.

On the other hand, snapshot style feeds like L1 (top of book) or L2 feed (price snapshot, order book deltas) can be conflated. If the message queue has multiple L1 updates, only the latest needs to be processed. In the case for an L2 update, they can be merged and updates to the same side and price can be conflated. This means when things are busy the work load can be reduced by skipping or conflating messages.

Decouple the producer and consumer

What this means in practical terms is that not every message from the producer needs to lead to a single call to the consumer. This is especially true if the consumer takes a lot more time to process a message than the producer. Messages can either be skipped or they can be conflated to reduce the amount of work downstream. This in practical terms often means reading from the message queue into some form of data structure that can be used to find redundant messages or to aggregate say trades at the same price. Once the message queue is empty, or too many messages have been processed, then these cleaned messages can be processed.

The net effect of this is that there should be fewer calls downstream than raw messages from the producer. This should lead to a lower latency tail if there are bursts of messages. If there is a bottleneck it should also allow the system to recover more quickly.

Split fast path and slow path

Some systems might have a quant model that is quite complex and expensive to compute (say 100 mics to 1ms+). This can be a problem if it needs to be fed raw data as trying to process every tick will lead to processing stale market data.

The solution is to split the processing paths. There is nothing stopping processing from being done on multiple CPU cores. The producer, consumer pattern is very flexible as it is essentially just a directional graph (be careful of cycles).

flowchart LR
    A[WebSocket<br><sub>Core 0</sub>]
    B[Message Parsing<br><sub>Core 0</sub>]
    C[Fast Alpha Gen<br><sub>Core 1</sub>]
    F[Slow Model <br><sub>Core 2</sub>]

    D[Order Mgmt<br><sub>Core 3</sub>]
    E[OMS<br><sub>Core 4</sub>]

    A -->|IPC| B
    B -->|IPC| C
    B -->|IPC| F
    C -->|IPC| D
    D -->|IPC| E
    F -->|IPC| D

The event driven architecture is quite neat due to the message passing and decoupling. With the slow model, since we know it will need to just take the latest snapshots of books, trades etc it can use a very efficient IPC structure called a SeqLock for communication. Simply put the slow model can use a loop and process data as quickly as it can and simply publish the latest predictions/forecasts when they are ready. The order management can then, consume these forecasts as if there were any other update feed.

Conclusion

I hope this article shows that trading systems can be designed in different ways and that it is ok to not have microsecond tick to trade. The most important thing is to make sure the data being processed is not stale and bottlenecks are kept to a minimum. Even if you have a very performant system it is still good to be prepared for the worst possible events. Hopefully some techniques about here help and lead to you being better able to price and trade consistently irrespective of market conditions.

Book Reviews and Reading List

Tue, 03 Sep 2024 00:00:00 +0000

I have wanted to write a reading list for a while. I’ve been apprehensive because I didn’t want to include too much and wanted also to explain why the books were in the list. For example, if you trade crypto there is not much point reading the Hull interest rate models book. This list likely will be a work in progress so keep that in mind. As I write this I realise I have a problem with how many books I have.

These are not referral links, they are just Amazon links. Feel free to source the books from wherever you like.

General

This book has a lot of interesting information. It really deserves a book review of its own. I personally like the topics on fractional differencing, volume bar sampling, and hierarchical risk parity (HRP). My only critisim of the books is some of the more useful things cannot be used directly. Volume imbalance bars, for instance, seem to degenerate in the book’s implementation. This is easy to correct and a solution can be found here.

Advances in Financial Machine Learning

Kaufman’s book is more of an encyclopedia of techniques. Its useful as a reference book.

Trading Systems and Methods

Ernie Chan’s books also come well recommended.

Portfolio management

This book is a must-have for anyone who wants to trade multi-asset portfolio strategies. It covers a massive range of topics around portfolio risk and portfolio construction.

Quantitative Portfolio Management: The Art and Science of Statistical Arbitrage

Gappy’s book is an excellent guide to factor investing and also factor models for portfolio management.

Advanced Portfolio Management: A Quant’s Guide for Fundamental

Market Making / Trade Cost Analysis

This space is filled with books about Avellaneda & Stoikov models and stochastic control theory. A lot of it I don’t even really understand and I had to learn stochastic calculus at university. In my opinion, there are no good books on market making that I know. The closest one is about trade impact modelling and trade cost analysis. Market making is very similar to these ideas but in essence, you have a forecast model and you trade when cost < forecasts. So being efficient and being able to measure the costs is important.

Algorithmic Trading: A Practitioner’s Guide

Strategy Optimisation

This book is great. The issue with A/B testing is that you want to switch over to the best model as quickly as possible. The author is ex-getco and its very well written.

Experimentation for Engineers: From A/B testing to Bayesian optimization

Timothy Master’s books are a good read for those wanting to build quantitative strategies. He does a lot of data mining, so the books address common issues that come from this. They include topics like, probability of false discovery, combinatorial symmetric cross validation (CSCV), optimisation objectives and much more.

Statistical Arbitrage

This book has some interesting chapters about mid-correction and dealing with jumps.

Statistical Arbitrage: Algorithmic Trading Insights and Techniques

This is not a book but an article. It’s about how to build a risk neutral long short portfolio strategy. It is very simple and generic and can be used either with rich/cheap (mean reversion) or momentum alphas.

The Modern Spirit of Statistical Arbitrage

Volume Price Analysis

For discretionary trading Volume profiles are useful for understanding where to place stop losses and take profits. What happened last time the market was at a price often gives insight into what will happen next time the same price is reached. Volume profiles tell you where a disproportionate amount of trading happened in the past and also if those orders are still there.

A Complete Guide To Volume Price Analysis

These books are mostly printed ebooks. They are cheap but are an introduction to how to read volume profiles for scalping.

Macro / Cycle Analysis

This is a book about credit and the effects of credit cycles on asset prices. It has some interesting points about not trying to time the market, but trying to position relative to the correct point the credit cycle.

Mastering The Market Cycle

Risk Management

This was a course book from my master’s degree. It’s an excellent foundational book on portfolio risk modelling.

Value at Risk: The New Benchmark for Managing Financial Risk, 3rd Edition

Regime Change

This isn’t really that great a book. It is more of a paper that was converted into a book. The ideas around direction change and having a time-invariant way to classify price changes are interesting. The ultimate conclusion of the book is that their method doesn’t outperform other strategies. The general idea is however useful if adapted.

Detecting Regime Change in Computational Finance

Books About Traders

The New Market Wizards: Conversations with America’s Top Traders

This is a book about the life of someone who was a pit trader in the 80s. He was very successful and is just a very scrappy and likeable guy. There are a few interesting anecdotes to adapting to the world as it more from floor trading to screen trading.

Pit Bull: Lessons from Wall Street’s Champion Day Trader

I have not read this but own it as it was a gift when I started at a hedge fund. It is mainly on here as people say you should read it. And they are correct, I really should get around to reading it.

Reminiscences of a Stock Operator

Peter L. Bernstein

Peter is an economic historian and is one of the few people who could write a book about a history of probability theory that is actually interesting to read.

Against the gods, is about how probability theory was used initially to play games and then adapted for financial modeling and trading.

Against the Gods: The remarkable story of Risk

Power of gold is a story about the evolution of money and how gold came to be used as money.

The Power of Gold: The History of an Obsession

Tail Risk Management

These two books are interesting but are likely not useful for 99% of people. The idea summarised is that options can be used to hedge downside risk. Options are also expensive due to theta and also in a normal market, they have delta, which you may not want as it interferes with the core strategy. So by combining OTM puts and ATM calls, you can be long gamma and flat theta and delta. So you basically get free insurance. The reason I don’t think this is for most people is the risk of an options book rarely stays the same as exactly what you wanted so it needs to be actively managed. If you don’t know what you’re doing, you might add more risk than the portfolio you’re meant to be hedging.

Trading Platform and HPC

This book is not the best. That said, there are not many end-to-end books on the topic of HFT trading platforms. If you are new to trading platforms, this is one of the few overviews I have seen.

Developing High-Frequency Trading Systems: Learn how to implement high-frequency trading

This book is mainly about optimisation and instrumentation. What I liked about it most is that it says, “Don’t assume measure”. This is super important with building trading platforms as measuring end-to-end latency is not trivial. How do you know if it’s fast if you can’t measure it?

The Art of Writing Efficient Programs: An advanced programmer’s guide to efficient hardware utilization and compiler optimizations using C++ examples

General Programming

This book is a collection of Joel Spolsky’s blog articles from around 2000. It’s old but they are easy to read and some of the ideas like “eat your own dog food” or “cardboard Chewbacca” are timeless.

Joel on Software: And on Diverse and Occasionally Related Matters That Will Prove of Interest to Software Developers, Designers, and Managers

Refactoring

This book by Kent Beck covers a topic I often struggle with. When working on code you often want to clean up parts. This covers how to do that and how to organise PRs so that it is not a nightmare for other people to code review.

Tidy First

This book is about refactoring and although I don’t agree with the core premise which is that functions should only be 5 lines. It is one of the few books that shows how to step by step re-organise code, extract sections and clean it up.

Five Lines of Code: How and When to Refactor

The two hardest problems in programming are overflow errors, cache invalidation and naming things. This book is a small succinct discussion of the latter topic.

Naming Things: The Hardest Problem in Software Engineering

Rust

This is only a small book and covers some of the weird ways that rust works that you might not expect. It is good for helping you to understand some of the lesser-covered topics.

Rust Brain Teasers: Exercise Your Mind

This is super specific to writing interprocess communication libraries. In general, synchronisation is something to be avoided in HPC since it is super slow. It is important to deeply know about how atomics work since they are a lightweight way to achieve synchronisation if needed. Seq locks use atomics are are often used in trading platforms for L1 order book feeds.

Rust Atomics and Locks: Low-Level Concurrency in Practice

This is a rust reference book. So it is not really meant to be read cover to cover.

The Rust Programming Language

Permaculture

This section is mainly for me, but if you’re interested, these are the best books i’ve read.

This was a text book for my PDC and is a great book

Practical Permaculture: for Home Landscapes, Your Community, and the Whole Earth

The Seymour book is known as the bible of self-sufficiency. It is an old book but full of useful information.

The New Complete Book of Self-Sufficiency

Masanobu Fukuoka’s book about farming in harmony with nature rather than fighting it.

The One-Straw Revolution: An Introduction to Natural Farming

This is a short book that covers everything about the ecosystems of healthy soil.

A Soil Owner’s Manual

Sepp Holzer is really well known in the permaculture world. The book is about restoring the ecosystem of a large area of Spain. His techniques are relatively simple and are mainly about helping the land re-invigorate itself.

Desert or Paradise: Restoring Endangered Landscapes Using Water Management, Including Lake and Pond Construction

A story about Gabe Brown’s farm and how they used regenerative agriculture to save their farm. Regenerative agriculture is about mixing livestock and arable agriculture and working them together. The net result is healthy soil, healthy plants, healthy animals and higher yield for lower costs.

Dirt to Soil: One Family s Journey into Regenerative Agriculture

This is a collection of low-tech solutions to reduce your carbon footprint without reducing quality of life.

Building a Better World in Your Backyard

Worms are really important to healthy soil, they airate, integrate carbon into the soil and improve drainage. The best set up I have seen was in Spain and used upcycled bath tubs filled with worms. When it rained, the water was collected via the drain hole for compost tea, then every 6 weeks half the volume could be added to the garden.

Worm Farming 101: A Practical Guide to the Essentials of Vermiculture & Vermicomposting

If there is anything that you feel is missing from the list then please let me know and ill try to read it and add it.

Even Faster Logging in Rust!

Mon, 12 Aug 2024 00:00:00 +0000

I was re-reading some older posts, and I realised I owed some readers a follow up. Hopefully I will be forgiven that this took 2 years. This post will be short and the core of the ideas are a follow up to the original article here.

The key takeaways from the original article are:

IO and Logging should not be done on the strategy hot path.
Formatting is expensive, and it is better to log variables and to format them on the logging core.

This article introduces improvements to the original code.

Pushing a closure onto the ring buffer means a Box is created on the heap.
The format string needs to be pushed onto the ring buffer each time which can be larger than the data itself.
Performance can be better optimised when an object’s size is known at compile time.

The solution to the above issues is pretty simple. The RawFunc is replaced with a Logging Enum and a set of Logging Messages that define sets of data to be logged. Then these enums can be pushed on to the logging queue as if they were any other data. This avoids the need for both the Box and the formatting string. Using the rust type system, it is then easy to associate the data logged with any formatting process and logging string and make sure it is logged correctly.

#[derive(Debug)]
struct QuoteLogData {
    instrument_id: u32,
    bid: f64,
    ask: f64,
}

impl QuoteLogData {
    fn new(instrument_id: u32, bid: f64, ask: f64) -> Self {
        QuoteLogData { instrument_id, bid, ask }
    }
}

impl Display for QuoteLogData {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        write!(f, "Quote: instrument_id: {} bid: {} ask: {}", self.instrument_id, self.bid, self.ask)
    }
}

#[derive(Debug)]
struct AlphaLogData {
    instrument_id: u32,
    ema_fast: f64,
    ema_slow: f64,
    vola: f64,
}

impl AlphaLogData {
    fn new(instrument_id: u32, ema_fast: f64, ema_slow: f64, vola: f64) -> Self {
        AlphaLogData { instrument_id, ema_fast, ema_slow , vola}
    }
}

impl Display for AlphaLogData {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        write!(f, "Alphas: instrument_id: {} ema_fast: {} ema_slow: {} vola: {}",
                   self.instrument_id, self.ema_fast, self.ema_slow, self.vola)
    }
}

#[derive(Debug)]
enum LogMessage {
    Quote(QuoteLogData),
    Alphas(AlphaLogData)
}

impl Display for LogMessage {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        match self{
            LogMessage::Quote(msg) => { write!(f, "{}", msg) }
            LogMessage::Alphas(msg) => { write!(f, "{}", msg) }
        }
    }
}

The nice thing about using logging enums is the formatting and format string is defined alongside the data. Over time as the log messages evolve, it is trivial to change the formatting as well and keep it up to date.

Setting up the sender and receiver is as simple as creating a lockfree queue to receive the LogMessage enums and print them out or send them to trace.

use lockfree::channel::spsc::{create, Sender, Receiver};

let (mut tx, mut rx) = create::<LogMessage>();

// set up thread to drain logging queue
let guard = thread::spawn(move || {
    let core_ids = core_affinity::get_core_ids().unwrap();
    core_affinity::set_for_current(*core_ids.last().unwrap());

    loop {
        if let Ok(msg) = rx.recv() {
            println ! ("{}", msg);
        }
    }
});

// send a log messages
tx.send(LogMessage::Quote(QuoteLogData::new(2008, 6000.25, 60000.26))).expect("Error sending log message");
tx.send(LogMessage::Alphas(AlphaLogData::new(2008, 10.0, 20.0, 1.0))).expect("Error sending log message");

The new enum based code is also significantly more performant compared to the old code. These are the criterion timings from a very simple benchmark. Please note, the benchmark only measures the time taken to put a log message onto the respective queue.

raw_func_logging        time:   [172.23 ns 173.09 ns 173.97 ns]
enum_logging            time:   [127.78 ns 128.37 ns 128.96 ns]

Hopefully it is clear why this new code is an improvement over the old version. I still like the idea that rust closures are easy to send over a channel. In the case of logging, it is just more efficient to send simple data objects and use a type system to define the formatting to be applied later.

Hidden Dangers of Writing an OMS

Sat, 22 Jun 2024 00:00:00 +0000

Writing an OMS for an HFT platform is a really difficult task that is often taken for granted. It is made more difficult in crypto because the exchange infrastructure is unreliable. It is not uncommon to simply be told “go away and come back later” when trying to call api functions. There is also often a hidden danger when writing an OMS which I have not seen discussed. This article is about what can be the most dangerous part of a high-frequency trading platform.

Code

And here it is…

if abs(oms.position(instrument_id) + order_qty) < POSTION_LIMIT and not oms.has_live_orders() {
    oms.place_order(instrument_id, order_price, order_qty);
}

The code is deliberately simplified to highlight the issue. In a real trading system, you would be able to place multiple orders for multiple instruments. The code checks risk limits and places an order if there is not an order in the market already. So if it is so simple, what’s the issue?

What would happen if an order was placed and filled, but the order update arrives before the position update?

What would happen if the position feed subscription failed and the position no longer updates?

This leads me to the next important point. Risk is the most important metric in any trading system. You might think it is pnl, but risk impacts the rate of change of the pnl and that is more important. Not knowing your risk is an easy way to blow up.

What Would This Look Like In Real Life

A much more complicated real life example can be found in the Bitmex Sample Market Maker. The code generates a ladder of orders and posts them to the exchange. It has some simple logic to cancel/place orders if the required orders do not match those in the market. It has one section in market_maker.py where it builds the required orders if the risk has not been exceeded. Later it checks if the order exists in the market and if not it posts it.

Order creation

for i in reversed(range(1, settings.ORDER_PAIRS + 1)):
    if not self.long_position_limit_exceeded():
        buy_orders.append(self.prepare_order(-i))
    if not self.short_position_limit_exceeded():
        sell_orders.append(self.prepare_order(i))

Order placement

for order in existing_orders:
    try:
        if order['side'] == 'Buy':
            desired_order = buy_orders[buys_matched]
            buys_matched += 1
        else:
            desired_order = sell_orders[sells_matched]
            sells_matched += 1

        # Found an existing order. Do we need to amend it?
        if desired_order['orderQty'] != order['leavesQty'] or (
                # If price has changed, and the change is more than our RELIST_INTERVAL, amend.
                desired_order['price'] != order['price'] and
                abs((desired_order['price'] / order['price']) - 1) > settings.RELIST_INTERVAL):
            to_amend.append({'orderID': order['orderID'], 'orderQty': order['cumQty'] + desired_order['orderQty'],
                             'price': desired_order['price'], 'side': order['side']})
    except IndexError:
        # Will throw if there isn't a desired order to match. In that case, cancel it.
        to_cancel.append(order)

I would like to say, this is not just a dig at this code. It is a nice solution to managing an order ladder, and I used it back in the day. I emailed Bitmex about it and the CTO (who was the original creator) got back to me. They were super helpful and in general, I have a soft spot for the old days of Bitmex.

The market maker would, often have more risk than it should due to it thinking there is no order in the market just after one just got filled. It would then place another leading to it also getting immediately filled violating risk limits.

The Impact

For market makers, this is more of an inconvenience risk might be exceeded by one clip or one fill might get a bad price. If something like this happens to a taker strategy sending IOC orders it can be catastrophic. Just ending up with a lot of risk is not even the worst possible outcome. If the strategy buys too much due to a bug like this and then sells its position to clean up the risk, it can get in a loop. This is known as flip-flopping, and the limit to how much this can lose is only bound by the account size.

It is always good to add sanity-checking logic to an OMS to specifically check for order spamming and flip-flopping to prevent IOC strategies from going postal.

Solution

Unfortunately, there is not a simple good solution to this problem. The issue is that you are dealing with knowing the state in an atomic way without slowing things down. It is difficult to know both the outstanding risk and positions in an atomic fashion. A simple solution is to flag the position as “dirty” and to wait after there is a fill for the position update. Needing to wait is not an acceptable solution to an HFT problem. One key observation is that positions are the summation of fills. If you keep a running sum of all fills, you will know your position. This can in practice be a little tricky since due to networking issues order updates can get missed. So a better solution is to allow a certain discrepancy between the running sum of fills since the last position update. This in essence will allow for a certain number of fills but will block more orders from being sent if there are a lot of fills without a position update. On exchanges which have proper sequence numbers, this is simple as you can work out if a fill is included or not in the most recent position update. If there are no sequence numbers then more care has to be taken. Given the position and recent fills, it is possible to process the fill list until the last position update is found. This will let you know if all positions have been included. Then the process can just carry on. This allows the position to have a maximum deviation between updates. If things are working normally there will be zero blocking. If there is an issue however then the system will wait until the risk can be confirmed.

It seems simple to query the order state or query the positions via API to complement the websocket feed. This is useful but is often used as a last resort as the exchange makes the cost of using it quite expensive.

Final Note

The issue above, if not treated with the correct care, can cost the whole account. Maybe it is an obvious problem. I don’t think it is as I often use it as an interview question if anyone says they have worked in or around an OMS. To be forewarned is to be forearmed, so at least now you know about the devil that may live under your bed.

Message Arrival Rates and Latency

Mon, 13 May 2024 00:00:00 +0000

There is a common debate when people are discussing code optimisation that relates to how fast code needs to be. A recent Twitter post about parsing binance BBA messages stated processing times of around 200ns. This is, in my admission, is very fast.
To put it into perspective, Serde is a common rust deserialization library and is incredibly easy to use. It is however, a lot slower than the optimised alternative taking around 2-3 micro seconds to do the same work. But who cares if your code takes 2 micro seconds longer to process a message? Is this going to make a huge difference to your pnl give that network jitter can be measured in milliseconds? The answer is yes, and maybe not for the reason you think.

Tick to Trade

Tick to trade (t2t) is measured as the time taken to receive a message and to generate some form of exchange api call (place/cancel/amend). This processing time can be measured either in software or at the network interface card.
Most trading systems are event/message engines and at their core, the strategy is single threaded. If you are running a latency sensitive strategy, having a low tick to trade means an order get to the exchange sooner and has a higher chance of being filled.

For strategies like market making it is still really important to be fast, just for different reasons. The simplest reason is that if there is a signal that requires cancelling orders, we would like to do this as quickly as possible.

The main issue is related to the statistical distribution of message arrival rates. Financial market data are known to be hetero-skedastic. That is just a fancy way to say markets occasionally go bananas. When market making you might be monitoring many different markets and instruments at the same time. It would be much simpler if data arrived at regular intervals and everyone was given enough time to make decisions. This is however not the world we live in.

The image above is from a paper about queue behaviour and packet delays. It is the closest thing I could find that describes the problem. The time between messages is not uniform and it tends to follow an exponential distribution. The resulting effect is, when it rains, it pours and you will get a lot of messages all at once.

Where 2t2 latency becomes a problem is due to the limits of throughput. It is possible to get 100s or 1000s of messages in a very short space of time. Maybe (and this is likely) the message that triggers a trade is the last one in the queue. So if your per message processing time is 100 mics, during a burst of 1000 messages, you will take 100 milliseconds before you send a trade. So you think you have a t2t of 100 mics, but in actuality, the trade was delayed much longer than you would expect.

Analysis

I was interested to see the effect of this on real world crypto data. In reality, for a production system, the actual t2t for orders should be monitored and measured. This analysis is indicative of how the delay distribution changes as a function of message processing time. It also gives an idea of what kind of processing times you should be aiming for.

The best method to measure the impact of delays is to tag market data with an id. Exchange api calls (place/cancel/amend) can then be tagged with the same id of the market data that triggered the api call. If there are delays due to the queue being saturated, it is possible to detect it. This can be a little tricky to do as it requires tagging messages on the network interface card (NIC). If you are able to do this, it is worth it as it is the only way to truly know if the websocket is not processed quickly enough.

The data is from 12 liquid swap and spot crypto pairs on 4 exchanges. The data was collected on May 1st 2024 and the message count was 175,001,142. If all these messages were uniformly spaced in a day, there will be approximately 494.3 microseconds between each message. This is not a lot of time, but it should be more than enough.

Looking at the actual message rates, the data is much more clustered.

largest number of messages per micro second: 4
largest number of messages per milli second: 656
largest number of messages per second: 38,046

There are many times we receive over 650 messages in a single millisecond. If the system took 100 mics to process a single message, it would only be able to process 10 of them in this time. This likely, will very quickly become a problem and cause the queue to get backed up.

If you assume a fixed t2t of x nano seconds how much delay is there between getting the message and processing it?

now = 0
delays = []
for i in timestamps:
    if i > now:
        now = i
    delays.append(now - i)
    now += offset

I was actually quite surprised by the results. In the case where each message takes 100 micros each to process, 65% of messages will have some delay.

t2t	Num Delayed	Ratio Delayed
1 micro	1,016,411	0.58%
5 micros	18,935,745	10.82%
10 micros	62,505,279	35.72%
100 micros	114,178,670	65.24%

As a rule of thumb, it is better to have an internal latency below 10 micro seconds. I was surprised that even 10 micro second processing times mean that 35% of messages experience some delay. The distribution of delays can be seen in the table below.

t2t	90%	95%	99%	99.9%	99.99%
1 micro	0.0 mics	0.0 mics	0.0 mics	0.82 mics	0.99 mics
5 micros	0.34 mics	3.82 mics	34.81 mics	319.34 mics	1.217 ms
10 micros	35.69 mics	75.52 mics	234.32 mics	1.28 ms	11.11 ms
100 micros	10.52 ms	32.84 ms	4.14 s	26.38 s	27.22 s

I also did an analysis assuming a 2t2 of 1 milli-second. The delays were so long I think it is pointless to report. Languages like python are thus not really viable solution for market making a large number of liquid assets purely due to data volumes.

Problem => Solution

So if this is a problem, what are the potential fixes?

Be faster
Triage and conflate messages. This is only really possible with order book data using seq locks
Subscribe to less data
Parallelize parts of the pipeline that are pre strategy/risk engine

The main thing is to be aware of the potential problems and be able to measure its impact on your system and specific use case.
I just hope this lays to rest the discussion that being fast is not important unless you’re running a latency arbitrage strategy. Data tends to come in huge bursts, and the last thing you want is to be stuck working through a backlog when you should be sending orders.

Possible Analysis Issues

I had limited time to do the analysis, I can see some potential weaknesses if you want to be pedantic. I do not think the points below change the over all conclusion.

Trading systems are distributed systems to the processing is done over multiple cores.
The timestamps of the collected data is based on the systems ability to read from the ws and timestamp it. This takes about 350 ns so it is possible that the recorded data contains its own delays all be it small.
Paths through a system’s logic are varied. Each path will have its own t2t. Assuming a single constant value is simplistic.
I only looked at a single day of data.

Conclusion

Crypto websocket feeds produce a lot of data.
That data comes in very concentrated bursts.
You probably want to aim to be able to read and process from a websocket in < 5 microseconds to avoid delays > 1 ms at the 99.99 percentile.
Reading from websocket, parsing, timestamping and placing on a channel is possible in sub 1 micro in rust.

The next time someone asks, “why bother saving 1 micro second?” You can point them to this article. I know I will be…

Rust for the Rustless

Fri, 26 Apr 2024 00:00:00 +0000

It has been a while. I hope you have not missed me! It seems that Rust is gaining steam and more people are starting to use it for building trading platforms. I have been advising people in the most tactless way possible that their life would be better if they embrace the crustation. So here we are. I have about two years of Rust experience under my belt, Although time is not the only measure of ability. There are some very “Rusty” ways to do things that are good to know about when you first start. As I have become more experience with Rust there are some common problems that I have found myself refactoring a lot. This article is a list of some of the main things I wish I knew when I started and also some helpful resources to help you on your Rust journey.

Avoiding Unwraps Everywhere

Rust makes everything explicit. It also often makes you think

what if this doesnt work?
Are you sure the key is in the map?

Many functions in Rust return Options or Results. You cannot use them directly, so to get at the value in them, you can use unwrap(). At first it is temping to unwrap everything.

let lookup: HashMap<u32, &str> = HashMap::from([(1, "Binance"), (2, "Okex")])
let name = lookup.get(&1).unwrap();

The issue is if the Option is None, or Result is Err, the program panics. Using unwrap is often tempting as one of the correct ways to deal with Options is to use the match statement to check the type. Using match is quite verbose, and often you only want to deal with the happy case.

let lookup: HashMap<u32, &str> = HashMap::from([(1, "Binance"), (2, "Okex")])
match lookup.get(&1){
    Some(name) => { println!("{}", name); }
    None => { /* error handling */ }
}

So it is temping to just use unwrap everywhere to avoid the boilerplate, but eventually you will get really unstable code due to the code panicking caused by this.

The solution to this is using If let

let lookup: HashMap<u32, &str> = HashMap::from([(1, "Binance"), (2, "Okex")])
if let Some(name) lookup.get(& 3) {
    println ! ("{}", name);
} else {
    eprintln ! ("What are you doing?");
}

or it is possible to give defaults to unwrap so that it is able to deal with the cases where it would otherwise return None

let lookup: HashMap<u32, &str> = HashMap::from([(1, "Binance"), (2, "Okex")])
if name = lookup.get(&1).unwrap_or("NA");
println!("{}", name);

Pattern Matching

Where this gets quite neat is dealing with situations where you have many optional variables. So in the example

let A_opt = Some(4);
let B_opt = Some("cat");
let C_opt = Some(3.14);

println!("a: {} b: {} c: {}", A_opt.unwrap(), B_opt.unwrap(), C_opt.unwrap());

Using multiple unwraps is just bad.

You could have an early exit, or a series of early exit checks.

if A_opt.is_none() || B_opt.is_none() || C_opt.is_none() { return }
println!("a: {} b: {} c: {}", A_opt.unwrap(), B_opt.unwrap(), C_opt.unwrap());

This will at least not panic, but it is still kind of garbage as you are not really dealing with the edge cases.

A better solution is using match and pattern matching to be able to deal with different combinations

match (A_opt, B_opt, C_opt) {
    (Some(a), Some(b), Some(c)) => {
        println!("a: {} b: {} c: {}", a, b, c);
    }
    (a, b, c) => {
        if a.is_none() { eprintln!("A is none"); }
        if b.is_none() { eprintln!("B is none"); }
        if c.is_none() { eprintln!("C is none"); }
    } 
}

So the above code allows the different cases to be dealt with separately if needed and the happy case to be simply defined.

Be Careful of Syntactic Sugar

I was once using a convenient method to avoid having multiple constructors if only a subset of fields are to be set.

struct MyStruct {
    id: u64,
    a: f64,
    b: f64,
    c: f64,
}

To create an object if you maybe want to create a default object but set the id to a specific value

let obj = MyStruct{id: 007, ..Default::default() };

I originally thought the compiler would be smart enough to realize that if you are defaulting a value, it does not need to be created in the call to Default::default(). This is not the case. A complete default object is created, then the field(s) in question are overwritten. If there is a type that is not a primitive or the default is expensive, this can be expensive.

The alternative I use is to have specific constructors.

impl MyStruct {
    fn new_with_id(id: u64) -> Self {
        MyStruct { id, a: 0.0, b: 0.0, c: 0.0 }
    }
}

// create obj
let obj = MyStruct::new_with_id(007);

This avoids any duplication at runtime and can be much more efficient.

Error Chain

When doing a lot of parsing, database and other things, you will come across Result<T, E> a lot. This is used in cases where operations can fail. The database is not available, the json string is malformed etc. Results have two enum types like Option’s Some and None. Results types are called Ok and Err and the techniques above regarding if let can be used for results.

fn connect_to_database()->Results<Connection, DatabaseErr> { /*impl */}
if let Ok(conn) = connect_to_database {
    if Ok(results) = conn.run_sql("select * from who_let_the_dogs_out") {
         println!("{:?}", results);
    }
}

The thing that is tricky with Rust’s type systems is if you want to pass the errors out of the function. Say you want to connect to a database, parse the result and maybe do your own processing. There are 3 types of errors that can happen DatabaseError, ParserError, and MyError. So if the return type is Result<T, E> what is the type of E? This can be painful, but it is what the error_chain crate solves. Its solution is to map errors from other packages to a homogenous error type from your crate. Then no matter what error is created in the method, the Error type in results will be the Crate error.

With error_chain crate the example is below

use errors::*;
fn my_fun() -> Result(MyType) {
    let conn = get_connnection()?;     // ? passes the DatabaseError to the caller and error chain wraps it
    let res = conn.get_data()?;        // pass any DatabaseError to caller   (wrapped by error chain)
    let raw_value = parse(res)?;       // pass any ParserError  to caller  (wrapped by error chain)
    let value = process(raw_value)?;   // pass any crate error to caller   (wrapped by error chain)
    Ok(value)  // return result
}

// calling the function
match my_fun() {
    Ok(value) => { println!("{}", value);}
    Err(e) => { /* e is the wrapped type given by error chain */}
}

The above code is often so much neater than having to explicitly deal with all the reasons that the code could fail in the method. The error type is also only wrapped, so if you want to write error handling logic for the specific error reason, this is also easy to do.

Doc Tests

When writing documentation, it is normal to include code examples.

/// Squares a given number.
///
/// # Examples
///
/// ```
/// let x = 5;
/// let squared = square(x);
/// assert_eq!(squared, 25);
/// ```
fn square(x: i32) -> i32 {
    x * x
}

One thing that is very neat about cargo test is it will compile and run the examples in doc strings. Over time as you refactor often the examples get broken as method signatures change. So Rust doc tests prevent this by raising an error if the examples no longer work.

Clippy

Clippy is a linter that checks for common issues. Lints are grouped to make it easier to decide if they should be a compiler error or just a warning.

clippy::pedantic: This lint group includes all the lints that Clippy developers consider pedantic. These lints may catch potential issues that are less common or subjective, but they can help improve code quality.
clippy::style: This group includes lints that enforce coding style guidelines. It focuses on stylistic aspects of your code, such as naming conventions, formatting, and idiomatic patterns.
clippy::correctness: This group includes lints that check for correctness issues in your code. These lints help catch potential bugs, logic errors, or undefined behavior in your code.
clippy::complexity: This group includes lints that check for overly complex code or constructs that can be simplified for better readability and maintainability.
clippy::perf: This group includes lints that check for performance-related issues in your code. It helps identify inefficient code patterns or operations that can be optimized for better performance.
clippy::cargo: This group includes lints related to Cargo, Rust's package manager. It checks for issues related to dependencies, build configuration, and package metadata.
clippy::nursery: This group includes experimental or unstable lints that are still under development or evaluation. These lints may change or be removed in future releases of Clippy.

This can be added to lib.rs main.rs of the crate

#![deny(clippy::perf)]
#![warn(clippy::pedantic)]

It is also possible to set specific lints in case lower granularity is needed

// this has caught bugs where I did == instead of =
#![deny(clippy::bool_comparison)]

to run clippy you can use

cargo clippy --all-targets --all-features

No inheritance

Rust uses composition rather than inheritance. Inheritance if you come from languages like Java seems like a nice idea, but it is prone to bugs. It is better to explicitly define the behaviours of each struct to avoid code being magically added or changed as superclasses are modified. I worked on some java code once where the inheritance structure was about seven layers deep, and some methods were removed in one layer and added back in the next. Trying to work out what code was actually running in the seventh layer was pretty much impossible.

This is why Rust avoids abstract classes completely.

Rust is Explicit

This can seem annoying at first. It comes up a lot when casting types and working with f64 and f32 types. Rust makes you think about overflows and truncation errors when other languages might just swallow them. You should avoid using as if you can as it can mask conversion errors. Functions like into() or try_into() and checked_sub() provide safety when doing these types of operations.

Access Control and Visibility

Access Control is achieved via pub, pub(crate). The compiler also checks for leaking of information i.e. you cannot have a pub variable in a pub(crate) struct.

pub struct MyStruct {
    pub name: String,     // public name
    pub(crate) id: u64,   // only used withing the crate
    var: f64,             // can only be used in the module

    pub fn pub_fun() {}
    pub(crate) fn crate_fun() {}
    fn private_fun() {}
}

pub(crate) BadStruct {
    pub name: String,     // this is not allowed as the struct has lower access privileges
}

Tests are normally in the module i.e. file so that can make checking private variables easier. In other languages, this is more challenging. This can lead to the private variables being exposed so they can be checked within tests. This is a common testing anti-pattern. In general, private fields/methods should not be tested as it exposes the implementation and can make refactoring difficult.

Performance Benchmarking

So you have written some new code and want to know if it is awful. Or you just want to compare the performance of various libraries with a hand rolled solution to prove youre smarter than everyone else.

Criterion is a rust benchmarking library that makes this really simple to do. Just add this to cargo.toml

criterion = { version = "0.5.1", features = ["html_reports"] }

[profile.bench]
lto = true
opt-level = 3
codegen-units = 1    # improved benchmark reliability

[[bench]]
name = "hash_bench"
harness = false

This is an example benchmark test that can be placed in benches/hash_bench.rs.

use criterion::{black_box, Criterion, criterion_group, criterion_main};
use hashbrown::HashMap as HBHashMap;
use std::collections::HashMap as StdHashMap;
use fxhash::{FxHashMap};

fn map_fun1(map: &mut StdHashMap<i64, i64>) {
    map.insert(3, 42);
}

fn map_fun2(map: &mut FxHashMap<i64, i64>) {
    map.insert(3, 42);
}

fn map_fun3(map: &mut HBHashMap<i64, i64>) {
    map.insert(3, 42);
}

fn hash_brown_test(c: &mut Criterion) {
    let mut map: HBHashMap<i64, i64> = HBHashMap::default();
    c.bench_function("hashbrown map test", |b| b.iter(|| map_fun3(black_box(&mut map))));
}

fn fx_map_test(c: &mut Criterion) {
    let mut map: FxHashMap<i64, i64> = FxHashMap::default();
    c.bench_function("fxhash map test", |b| b.iter(|| map_fun2(black_box(&mut map))));
}

fn std_map_test(c: &mut Criterion) {
    let mut map: StdHashMap<i64, i64> = StdHashMap::default();
    c.bench_function("std map test", |b| b.iter(|| map_fun1(black_box(&mut map))));
}

criterion_group!(benches, std_map_test, fx_map_test, hash_brown_test);
criterion_main!(benches);

To run the benchmarking it is as simple as running cargo bench. When the html reports feature enabled produces reports like this.

Html reports can be found in the target/criterion directory. The reports are useful to dig deeper into the latency distribution of the code under test. Criterion also monitors the performance during refactoring to warn if the performance of a function has improved or regressed.

Lifetimes Strings and &str

This is something that is really important for performance. The borrow checker can be a pain. Using strings should be avoided in general, that said, when they are used they should be used once. Since &str is a reference and not an object, it cannot be moved. For it to be borrowed, the borrow check must make sure you don’t yeet it somewhere else while the function is running. This is done via lifetimes. When you pass a reference it is bound to a lifetime.

struct Instrument {
    instrument_id: u64,
    symbol: &'static str,
}

fn main() {
    // Create an instance of the Instrument struct
    let instrument = Instrument {
        instrument_id: 1,
        symbol: "BTCUSDT",
    };

    // Access fields of the Instrument struct
    println!("Instrument ID: {}", instrument.instrument_id);
    println!("Symbol: {}", instrument.symbol);
}

In the above example the 'static tells the compiler that the str value will live for the length of the program.

If the &str is passed as an argument the compiler can attach the lifetime of the variable so that it can check that it outlives whatever it is passed to.

struct Instrument<'a> {
instrument_id: u64,
symbol: &'a str,
}

fn main() {
    let symbol = "BTCUSDT";

    // Create an instance of the Instrument struct
    let instrument = Instrument {
        instrument_id: 1,
        symbol: symbol,
    };
}

In this example lifetime 'a is generated by the compiler and used to make sure that the symbol outlives instrument;

Derive

When creating structs, a lot of behaviour can be added via the derive macro. Objects can be de-serialised just by adding a single line of code. It is also worth adding Default to your classed should you want to use them in a data structure.

use serde::de::DeserializeOwned;

#[derive(Deserialize, Debug, Default)]
pub struct ApiConfig {
    pub key_map: HashMap<String, String>,
}

Macros

I actually am not a fan of these. I use simple ones as helper functions like

#[macro_export]
macro_rules! to_bps {
    ($x:expr) => {
        $x * 1_00_00.0
    };
}

let x_bps  = to_bps!(0.0001);

They are incredibly powerful (like the derive macro) and have the ability to write code themselves like any other pre-preprocessor. Just Like C++ templating, I am not sure how easy it is to debug/fix heavily macroed code. This is just a preference thing. I’m sure if you know what you are doing you can do some really cool things with it.

ChatGtp

If you are coming from a different language, chatgtp can be a good way to learn about how to do things in Rust. You can give it some example code and ask it to port it to rust. It is not perfect, but it is a good way to learn about some topics like iterators, data types, filters, etc. It is also good at suggesting popular libraries. This is how I originally found out about Rayon and some of its benefits for async vs. tokio.

Materials

A Half An Hour To Learn Rust

This is a great introduction into some of the key concepts into Rust. The main thing is that its readable and very quick to absorb if you come from other languages. Most programming books and incredible terse and frankly just boring to read. So this concept is really clever to take the same content and make it palatable.

Rust Exercises

This course will teach you Rust’s core concepts, one exercise at a time. You’ll learn about Rust’s syntax, its type system, its standard library, and its ecosystem.

The Rust Programming Language

This is kind of the bible of Rust. That said, it’s boring and more of a reference book. I have not completely read it, I have only picked through sections such as error handling etc. I would still say its worth having in case you need to look things up.

Rust Brain Teasers

This book covers some of the leaky abstractions and weird ways that Rust does not really behave the way you might expect. It is good additional reading as you become more familiar with the language.

Rust Atomics and Locks

This is a very detailed book not only about Rust but how memory architecture works in multithreaded systems. Its also additional learning if you want to get deep into interprocess communication techniques like SeqLocks.

Building Candlesticks in Rust

Sat, 29 Oct 2022 00:00:00 +0000

Candlesticks are a common way to represent price and volume of an asset over a period of time. There are various common types of bars such as time, volume, tick bars, hieken-ashi, renko to name a few. There is a lot of information about the implementations of these on the internet so their details will not be covered here. The aim of this article is to share some tips for implementation and also an implementation of Marcos Lopez de Prado’s volume imbalance bars from the book Advances in financial machine learning.

This article assumes that the reader is familiar with what a candlestick is, and the common types of methods of building candlesticks. If not then this Alternative Candlestick Types covers the basics.

There is also the question of why use volume bars or tick bars over time bars. The general idea is that due to volatility clustering asset returns are heteroskedastic so the alternative candlesticks aim to sample by information rate as opposed to time. This is a similar idea to stochastic volatility models such as the Heston Model in which the process for the volatility is separately modelled and then just assumes everything else is normal.

This article covers a lot of the information regarding not only these alternative bars but the impact on the statistical properties of the resulting return series Bars. TLDR; kurtosis is bad. You don’t want to find out the trend has changed, but waiting for a new candle stick left the position 20% underwater.

Let’s start by defining a bar

#[derive(Debug, Clone, Serialize)]
pub struct Bar {
    pub instrument_id: u32,        // instrument id
    pub open: f64,                 // open price
    pub high: f64,                 // high price
    pub low: f64,                  // low price
    pub close: f64,                // close price
    pub buy_volume: f64,           // total buy volume
    pub sell_volume: f64,          // total sell volume
    pub vwap_price: f64,           // volume weighted average price for bar
    pub open_ts: u64,              // timestamp in ns
    pub close_ts: u64,             // timestamp in ns
}

There a couple of extra variables of importance compared with OHLCV bars. Volume should be signed since there is information in the imbalance between buys and sells. Also volume weighted average price (vwap) can be tracked over the period since close-vwap contains information about momentum/commitment.

Time Bars

Define the BarBuilder trait ( i.e. interface).

pub trait BarBuilder {
    fn trade_update(&mut self, timestamp: &u64, price: &f64, qty: &f64) -> Option<Bar>;
}

Define TimeBarBuilder class. Notice in rust it doesn’t implement the BarBuilder interface.

pub struct TimeBarBuilder {
    time_window: u64,
    next_bar_time: u64,
    bar: Bar,
}

Constructor implementation

impl TimeBarBuilder {
    pub fn new(instrument_id: u32, time_window: u64) -> Self {
        TimeBarBuilder {
            time_window,
            next_bar_time: 0,
            bar: Bar::new(instrument_id),
        }
    }
}

Implement trade_update trait for the TimeBarBuilder

impl BarBuilder for TimeBarBuilder {
    fn trade_update(&mut self, timestamp: &u64, price: &f64, qty: &f64) -> Option<Bar> {
        let mut res: Option<Bar> = None;

        // check if there is a new bar
        if *timestamp > self.next_bar_time {
            // set the next time a bar should publish
            let rem: u64 = *timestamp % self.time_window;
            self.next_bar_time = *timestamp - rem + self.time_window;

            self.bar.close = *price;
            self.bar.close_ts = *timestamp;

            // compute the volume weighted average price for the bar
            self.bar.vwap_price /= self.bar.buy_volume - self.bar.sell_volume;

            if self.bar.open_ts != 0 {
                // take a copy of the bar information
                res = Some(self.bar.clone());
            }

            // reset the tracking values
            self.bar.reset(price, timestamp);
        }

        // update high/low prices
        if *price > self.bar.high { self.bar.high = *price; }
        else if *price < self.bar.low { self.bar.low = *price; }

        // update the vwap and quantity information
        self.bar.vwap_price += *price * (*qty).abs();
        if *qty > 0.0 { self.bar.buy_volume += *qty; }
        else { self.bar.sell_volume += *qty; }

        return res;
    }
}

Volume bars

Define VolumeBarBuilder class

pub struct VolumeBarBuilder {
    volume_threshold: f64,
    bar: Bar,
}

Constructor implementation

impl VolumeBarBuilder {
    pub fn new(instrument_id: u32, volume_threshold: f64) -> Self {
        VolumeBarBuilder {
            volume_threshold,
            bar: Bar::new(instrument_id),
        }
    }
}

Implement trade_update trait for the VolumeBarBuilder

impl BarBuilder for VolumeBarBuilder {
    fn trade_update(&mut self, timestamp: &u64, price: &f64, qty: &f64) -> Option<Bar> {
        let mut res: Option<Bar> = None;

        // a new bar is created when the volume threshold is crossed
        if (self.bar.buy_volume - self.bar.sell_volume) > self.volume_threshold {
            self.bar.close = *price;
            self.bar.close_ts = *timestamp;

            // compute the volume weighted average price for the bar
            self.bar.vwap_price /= self.bar.buy_volume - self.bar.sell_volume;

            if self.bar.open_ts != 0 {
                // take a copy of the bar information
                res = Some(self.bar.clone());
            }

            // reset the tracking values
            self.bar.reset(price, timestamp);
        }

        // update high/low prices
        if *price > self.bar.high { self.bar.high = *price; }
        else if *price < self.bar.low { self.bar.low = *price; }

        // update the vwap and quantity information
        self.bar.vwap_price += *price * (*qty).abs();
        if *qty > 0.0 { self.bar.buy_volume += *qty; }
        else { self.bar.sell_volume += *qty; }

        return res;
    }
}

Volume Time Bars

What is the correct threshold for a volume bar builder? Rather than set it to a single value it might be convenient to set it to a dynamic value based on the volume of recent X time bars. So a 5m volume time bar would create a new bar when the volume is larger than the recent average of 5 minute bars. This helps in two ways, one is that when working with different assets you don’t need to specify values for each individual asset, the second is that the average then naturally includes measure of seasonality. The aim of volume bars is to sample more frequently when “something unusual is happening” so it’s useful to have a benchmark of normal. It would be possible to go further in regard to seasonality modelling and if you are interested there is a review of using spline models to build seasonality models in my Masters Thesis.

This idea can be used to solve some hyper parameter problem associated with Marcos Lopez de Prado’s volume imbalance bars. The idea of a volume imbalance bar is that a new bar should be created if either a volume threshold is passed or the ratio of buys to sells exceeds a threshold. The implementation of this can be tricky since the thresholds need to be set. If the thresholds are calculated using the generated bars they can have degenerate behaviour where expectations tend towards zero or infinity.

My solution to this is to set the imbalance threshold to be a % of the expected volume so if the volume is greater than x or the imbalance threshold is greater than 20% then a new bar will be created. This means only parameters needed are the time aggregation, the decay for the volume ema for the time builder and the imbalance threshold percent. Since the threshold is not a function of the generated imbalance bars the system does not have issues with the expectations tending to zero or infinity.

The following is the implementation of a dynamic volume imbalance bar builder.

// implementation of a volume imbalance builder

pub struct DynVolumeImbBarBuilder<T: BarBuilder> {
    imb_threshold: f64,    // imbalance threashold [0,1]
    exp_volume: f64,       // volume threshold
    exp_imbalance: f64,    // imbalance threshold
    builder: T,            // bar bulider for generating samples for threshold
    bar: Bar,              // storage
    volume_info: Sma,      // moving average calulating recent volume
}

note the <T: BarBuilder> which means the builder uses another builder to be its “clock”. Normally a time bar is used but other types of bar could also be used such a range bar. This would give a new bar when an amount of volume trades that would normally move the market by a specified percentage.

impl<T: BarBuilder> DynVolumeBarBuilder<T> {
    pub fn new(instrument_id: u32, builder: T, imb_threshold: f64, vol_halflife: u16) -> Self {
        debug_assert!(0.0 <= imb_threshold && imb_threshold <= 1.0);

        DynVolumeImbBarBuilder {
            imb_threshold,
            exp_volume: 0.0,
            exp_imbalance: 0.0,
            builder,
            bar: Bar::new(instrument_id),
            volume_info: Sma::new(vol_halflife),
        }
    }

    pub fn trade_update(&mut self, timestamp: &u64, price: &f64, qty: &f64) -> Option<Bar> {
        let mut res: Option<Bar> = None;

        // call the driver bar builder to estimate the expected volume per bar
        match self.builder.trade_update(timestamp, price, qty) {
            Some(bar) => {
                if !is_close(bar.high, bar.low) {
                    let volume = bar.buy_volume + bar.sell_volume.abs();
                    self.exp_volume = self.volume_info.update(volume);
                    self.exp_imbalance = volume * self.imb_threshold;
                }
            }
            None => {}
        }

        // check the threshold is set
        if self.exp_volume == 0.0 {
            return None;
        }

        // n.b sell_volume is -ve
        let total_volume = self.bar.buy_volume - self.bar.sell_volume;
        let volume_imb = (self.bar.buy_volume + self.bar.sell_volume).abs();
        if total_volume > self.exp_volume || volume_imb > self.exp_imbalance {
            // set the next time a bar should publish
            self.bar.close = *price;
            self.bar.close_ts = *timestamp;

            // compute the volume weighted average price for the bar
            self.bar.vwap_price /= self.bar.buy_volume - self.bar.sell_volume;

            let mut res: Option<Bar> = None;
            if self.bar.open_ts != 0 {
                // take a copy of the bar information
                res = Some(self.bar.clone());
            }

            // reset the tracking values
            self.bar.reset(price, timestamp);
        }

        // update high/low prices
        if *price > self.bar.high { self.bar.high = *price; }
        else if *price < self.bar.low { self.bar.low = *price; }

        // update the vwap and quantity information
        self.bar.vwap_price += *price * (*qty).abs();
        if *qty > 0.0 { self.bar.buy_volume += *qty; }
        else { self.bar.sell_volume += *qty; }

        return res
    }
}

Multi Bar Volume Imbalance Bars

I need to write an article about correlation structure in cryptos since its fascinating. It’s mostly fascinating since there isn’t any! The first principal component of the correlation matrix is about 85% of variance and the first two components explained about 95% of the market. So what does that means in practical terms? When bitcoin moves, or etherium moves, so does everything else. There are good posts by tr8dr in regard to correlation clustering and transfer-entropy. When this is applied to cryptocurrencies the results is not a very complex graph, but you can see the structures between L0 coins, L1 coins DeFi/NFTs coins etc.

Conceptually I think volume should not be thought of as per asset, but per risk factor. So what we can do is make a volume bar builder which ticks not when the volume in asset X breaks a threshold but when the amount of risk in a certain risk factor breaks a threshold.

A very simple version of this can be done by monitoring multiple assets and aggregating their volume. A new set of candle sticks are generated when the aggregated volume crosses a threshold. This is basically assuming that correlations are 1 for all assets which in crypto is not a particularly bad assumption. n.b. The below implementation assumes all assets are in the same quoted currency.

Define the trait for a multi bar builder

pub trait MultiBarBuilder {
    fn trade_update( &mut self, timestamp: &u64, instrument_id: &u32,
        price: &f64, // price
        qty: &f64    // signed quantity
    ) -> Option<Vec<Bar>>;
}

Define the MultiDynVolumeImbBarBuilder class

pub struct MultiDynVolumeImbBarBuilder<T: MultiBarBuilder> {
    imb_threshold: f64,            // threshold parameter
    exp_volume: f64,               // expected volume
    exp_imbalance: f64,            // expected imbalance
    total_buy_volume: f64,         // total buys of all assets
    total_sell_volume: f64,        // total sells of all assets
    time_bar_counter: u32,         // number of time bars
    wait_time_bars: u32,           // number of bars to build threshold
    builder: T,
    bars: HashMap<u32, Bar>,       // collection of bars

    volume_info: Sma,
}

Implement the constructor and update methods

impl<T: MultiBarBuilder> MultiDynVolumeImbBarBuilder<T> {

    /* constructor */
    pub fn new(
        instrument_ids: &Vec<u32>,
        builder: T,
        imb_threshold: f64,
        vol_halflife: u16,
        wait_time_bars: u32,
    ) -> Self {
        debug_assert!(0.0 <= imb_threshold && imb_threshold <= 1.0);

        let mut bars: HashMap<u32, Bar> = HashMap::new();
        for iid in instrument_ids {
            bars.insert(*iid, Bar::new(*iid));
        }

        MultiDynVolumeImbBarBuilder {
            imb_threshold,
            exp_volume: 0.0,
            exp_imbalance: 0.0,
            total_buy_volume: 0.0,
            total_sell_volume: 0.0,
            time_bar_counter: 0,
            wait_time_bars,
            builder,
            bars,
            volume_info: Sma::new(vol_halflife),
        }
    }

    /* constructor */
    pub fn trade_update(&mut self, timestamp: &u64, instrument_id: &u32, price: &f64, qty: &f64) -> Option<Vec<Bar>> {
        let mut res: Option<Vec<Bar>> = None;

        // update normalisation bar builder to calculate average bar information
        match self
            .builder
            .trade_update(timestamp, instrument_id, price, qty)
        {
            Some(bars) => {
                let mut volume: f64 = 0.0;

                for bar in &bars {
                    if bar.buy_volume > 0.0 || bar.sell_volume > 0.0 {
                        volume += bar.buy_volume - bar.sell_volume;
                    }
                }

                // update average volume
                self.volume_info.update(volume);

                // wait until we have a few updates before publishing
                if self.time_bar_counter >= self.wait_time_bars {
                    self.exp_volume = self.volume_info.value;
                    self.exp_imbalance = self.volume_info.value * self.imb_threshold;
                }

                self.time_bar_counter += 1
            }
            None => {}
        }

        // n.b sell_volume is -ve
        let total_volume = self.total_buy_volume - self.total_sell_volume;
        let volume_imb = (self.total_buy_volume + self.total_sell_volume).abs();
        if self.exp_volume > 0.0 &&      // check the the expected volume is set up
            (total_volume > self.exp_volume || volume_imb > self.exp_imbalance)
        {
            // new bar to publish so record results
            let mut bar_res: Vec<Bar> = Vec::new();
            for (iid, tbar) in &self.bars..iter_mut() {

                if iid == *instrument_id { tbar.close = *price; }
                tbar.close_ts = *timestamp;
                tbar.vwap_price /= (tbar.buy_volume - tbar.sell_volume);

                // record bar and reset the tracking values
                bar_res.push(tbar.clone());
                tbar.reset(price, timestamp);
            }

            self.total_buy_volume = 0.0;
            self.total_sell_volume = 0.0;

            res = Some(bar_res);
        }

        // update bars
        let mut bar = self.bars.get_mut(instrument_id).unwrap();

        bar.close = *price;

        // update high/low prices
        bar.high = f64::max(bar.high, *price);
        bar.low = f64::min(bar.low, *price);

        // update the vwap and quantity information
        bar.vwap_price += *price * (*qty).abs();
        if *qty > 0.0 {
            bar.buy_volume += *qty;
            total_buy_volume += *qty;
        }
        else {
            bar.sell_volume += *qty;
            self.total_sell_volume += *qty;
        }

        return res;
    }
}

It should be noted that if a simple multi volume bar builder is needed then the imb_threshold can be set to 1.0. This will give the same behaviour as not having the volume imbalance check at all.

The performance of these implementations could be improved upon. This is from my research code so for the sake of code re-use one bar builder is embedded in another. Since only the time by volume needs to be tracked for the dynamic volume bars it would be possible to simply use an exponentially decaying sum of volume as the threshold. This would be more performant than generating complete time bars only to throw away almost all the information. Also since bars are cloned it would be better to have the builder store the bar and then return a boolean with true as there being a new bar. This would avoid any over head from memory allocation.

Conclusion

Using time bars to set the volume threshold is a neat way to avoid seasonality issues and also the complexity of defining multiple thresholds for different assets.
Using time bars to set a baseline for volume imbalance bars is a good way to avoid the problem of the threshold becoming degenerate and greatly simplifies the choosing of hyper parameters.
When multiple assets are concerned volume can be thought of as risk and its worth generating volume bars synchronously by looking at total traded risk.

I think these ideas are important specifically in cryptocurrency markets due to the kurtosis necessitating some for of informationally uniform sampling (volume or volume imbalance) and the correlation structure meaning volume traded on one asset cannot be viewed independently of the rest.

good luck, have fun.

Fast Logging for HFT In Rust

Sun, 23 Oct 2022 00:00:00 +0000

In this article, we’ll be discussing a fast way of logging in Rust and its application to high-frequency trading. The code presented here solves two problems, one is well-known, the latter less so. It is imperative to avoid using IO operations within the strategy thread, but logging operations can hide a lot of memory operations that should also be avoided. Logging from the strategy is useful for monitoring or debugging purposes. Logging can however be expensive if implemented inefficiently. I have known companies to run two builds in production, one with logging and the other without logging that trades. This set-up avoids any issues with logging affecting the strategies throughput, however, this in my opinion is a fairly extreme solution.

When discussing latency we also need to talk not only about the mean latency but also the full latency distribution. It’s important to be fast, but also to be consistently fast. HFT trading platforms are a form of asynchronous message systems and latency can be measured as the time between an input message such as an orderbook update and an output message such as an order. Message rate tends to be correlated with volume and volume tends to be correlated with market opportunity. It is quite normal for latency to spike when message rates spike. This is why not only the mean latency should be measured but the 99 and the 99.9 percentile latency. What is to be avoided is delaying the sending of orders since the system is stuffed with messages that it cannot process fast enough.

As discussed in the previous article, one of the main advantages of using languages such as C++ or rust because of their lack of garbage collection. Garbage tends to be created when there is a lot of activity and the garbage collectors (GC) make latency non-deterministic since other processes are blocked as it runs. I have not used Java in a long time so if there is now a decent parallel/concurrent GC then please forgive me. Even so, GCs are to be avoided even when using a GC language.

When trying to write efficient code there are a few important general rules. Memory operations tend to be quite expensive compared to arithmetic operations. Many high-frequency trading platforms tend to either not use data structures or to pre-allocate memory on startup. Other operations which are relatively expensive include those associated with IO or locking since those deal with the operating system. The issue with the operating system is not only that operations are relatively slow but because operations need to be synchronised across multiple processes, the latency of the operations is non-deterministic.

Let’s start with a naive implementation of logging.

/* simple logging using println! */
let date = Local::now();
println!("ts: {} volume: {} price: {} flag: {}", date.format("%Y-%m-%d %H:%M:%S"), 100.02, 20000.0, true);

One of the main issues with logging is that this involves a slow IO operation. The simple solution to this is to use asynchronous logging so the IO thread is separated from the strategy thread. This can significantly improve both the strategy latency and also the latency variance. A common implementation of this is to generate a log string and then serialise it to a separate async logging thread. This can be done via a lock-free queue of some sort to avoid the strategy thread needing to wait for locks.

The following is an improved implementation that uses a lock-free queue to communicate between threads.

/* implementation of logging on an asynchronous thread */
use lockfree::channel::spsc::{create, Sender, Receiver};

// set up async logging thread
let (mut sx, mut rx) = create::<String>();
let guard = thread::spawn(move || {
    let core_ids = core_affinity::get_core_ids().unwrap();
    core_affinity::set_for_current(*core_ids.last().unwrap());

    match (rx.recv()){
        Ok(msg) => { print("{}", msg); }
        Err(e) => { panic!("well this is not good"); }
    }
});

// strategy thread
let date = Local::now();
let log_msg = format!("ts: {} volume: {} price: {} flag: {}", date.format("%Y-%m-%d %H:%M:%S"), 100.02, 20000.0, true);
sx.send(log_msg).unwrap();   // should really handle errors

How expensive is the creation of the log string? The answer is a lot. Often timestamps are formatted to date string and the creation of the string itself involves dynamic memory allocation which should be avoided. It would be much better if it was possible to offload not only the IO operations but also the string building to a separate thread. Getting around this cost using an alternative implementation is pretty difficult to achieve in most languages.

A function is needed that takes not only a variable number of parameters but a variable number of parameters of arbitrary type. This is relatively complicated to do in strongly typed languages such as Rust and C++. One solution to this is to use some form of binary serialisation such as bincode, protobuffers or BSON to serialise the arguments to be logged as well as the format string. This is possible, but I don’t think it is the most efficient way to do it. For performance reasons, it would be better if most of this work could be done at compile time to avoid unnecessary memory operations.

Rust provides an alternative and very neat solution to this problem. In Rust closures can be serialised directly and then sent to another thread to be executed. So it is possible to create a closure which creates the log string and then outputs the log string. Instead of anything being executed by the strategy thread the function pointer and arguments are serialised and send via a channel. When this is received by the logging thread the closure can be deseralised and then invoked avoiding the strategy thread from having to do anything.

/* asynchronous execution of a serialised closure */
use lockfree::channel::spsc::{create, Sender, Receiver};

// struct for wrapping the closure so it can be serialised
struct RawFunc { data: Box<dyn Fn() + Send + 'static> }

impl RawFunc {
    fn new<T>(data: T) -> RawFunc
    where T: Fn() + Send + 'static,
    { return RawFunc { data: Box::new(data) }; }
    fn invoke(self) { (self.data)() }
}

// create async thread to execute logging closure
let (mut sx, mut rx) = create::<RawFunc>();  // lock free channel
let guard = thread::spawn(move | | {
    let core_ids = core_affinity::get_core_ids().unwrap();
    core_affinity::set_for_current(*core_ids.last().unwrap());

    match (rx.recv()){
        Ok(msg) => { output.invoke(); }
        Err(e) => { panic!("well this is not good"); }
    }
});

// strategy thread
let date = Local::now();   // the timestamp should be from the strategy thread
sx.send(RawFunc::new(move || {
       println!("ts: {} volume: {} price: {} flag: {}", date.format("%Y-%m-%d %H:%M:%S"), 100.02, 20000.0, true);
})).unwrap(); // should really handle errors

The relative performance of the three solutions on my local machine are logging directly takes on average 1769 nanoseconds, creation of the log string and serialising it takes roughly 1189 nanoseconds on average, and the solution to serialise the closure takes on average 120 nanoseconds. I think a reasonable target latency of a strategy should be in the single-digit microseconds. This serialised closure implementation of logging means this latency target is possible while still being able to log from the strategy thread. The real benefit of using the closures is that it doesn’t require any string or memory operations so there is also almost no impact in the 99% tail of the latency distribution. If we examine the latency distribution of each of these implementations this is easy to see. The closure version has much better behaviour.

When looking at the graphs the real improvement with regard to using serialised closures can be seen. Not only is the closure method faster on average but its latency distribution is much more predictable. I was somewhat surprised that the difference in results for string serialisation is so wild, but I put that down to the date formatting and string building.

It is possible to implement something similar in other languages. C++ has lambdas and I think these can be serialised. What I like about Rust is that the majority of the complexity is taken care of by the compiler for very little performance cost. You also don’t have to worry about concurrency issues since the compiler also makes sure that ownership is transferred to the logging thread. The more I learn about Rust the more I like its productivity as a function of programmer hours. I think this article is a cool example of that.

Bitcoin Elasticity and Volatility

Fri, 21 May 2021 00:00:00 +0000

So the crypto markets on May 19th were fun!? If you have been a part of these markets for any time, this volatility is not that surprising. That said it is still pretty amazing that the price can drop 30% in a single day. This is maybe more surprising since it is to the backdrop of an increase in institutional investors and a narrative of “this time it’s different”. I was asked a question with regards to “how is this level of volatility is possible?” I suspect that it is a problem related to some idiosyncrasies of crypto exchanges and the limited free float. I was curious to see what the data said and specifically what the market did during the massive pin bar that touched the 30k USD level.

Thankfully about a week ago I managed to complete a Rust market data collector for binance that lets me collect order book and trade data. This collector is much better than my previous python implementation since the old one often was overwhelmed when there was a large spike in the volume of market data. This led to the collected data having clock drift in timestamps since it was not possible to process the WebSocket feed fast enough. Rust is a monster and has no such issues as can be seen by the cpu load over the last 7 days.

python cpu 7-day load

rust cpu 7-day load

I think the spikes on the rust machine are from running gzip and moving data off the machine.

The Hypothesis

Since there is a limited amount of margin on each exchange at any one time this means the free float of cryptos are relatively low. I hypothesise that due to liquidations and issues with margin, prices become overly elastic and move on relatively low volume imbalances. An amount of volume which would normally move the market 1k could thus push the market 10k.

The Indicator

To measure order flow imbalance I use an event time indicator like Tr8dr’s BSI indicator described here. The indicator is the decaying cumulative sum of the signed volume. The labelling of the trades is something for debate. The binance “market maker is buyer” flag gives suspicious results, so I have discarded it in favour for using the price relative to mid.

pub struct BSIEma {
    pub value:f64,

    last_update_time:u64,
    alpha:f64,
}

impl BSIEma {

    pub fn new(half_life: u64) -> Self {
        BSIEma {
            value: 0.0,
            last_update_time:0,
            alpha: 2.0f64.powf(-1.0/(half_life as f64)),
        }
    }

    pub fn update(&mut self, ts: u64, qty: f64) -> f64 {

        // calculate the decay based on time passage
        let mut dt:f64 = (ts - self.last_update_time) as f64;
        let decay = self.alpha.powf(dt);
        self.last_update_time = ts;

        // decay existing values
        self.value *= decay;

        // update with new data
        self.value += qty;

        return self.value;
    }
}

Results

BTC Price vs 60 second BSI

During the day there were about 5-6 large sell events where the 60-second buy/sell imbalance exceeded 20 million USD. Interesting the first at about 1 am UTC only moved the market 1k USD. 20M USD at a price of 40k USD is equivalent to about 500 BTC or 0.002671888% of the total (assuming 18,713,356.25) in the world. I am surprised by these numbers since I expected them to be higher given the level of bloodshed in the market.

Zoom In Of Large Market Drop

During the times when the market moved down to 30k, there were three individual events each with a 60 second BSI imbalance of about 20M. The first two sell events moved the price about 3k together, and the final moved the price 3k USD on its own. During the whole day, the elasticity of the market was increasing meaning the amount of volume it took to move the market was much lower.

Conclusion

There is more formal analysis to do relating to a dynamic relationship between liquidity and volatility. That was not the aim of this post. I mainly wanted to show how significant the change in the markets’ ability to clear volume imbalances became on May 19th.

It would be better if I could include data from the whole aggregate market but at the moment I only have clean data for one exchange.

The price of assets when there is limited free float can be complicated. When there is a large amount of the asset off-exchange and held by people then price discovery can be difficult. There is in essence a “complete” limit order book in which all holder of the asset would be willing to move the assets to exchange given a certain price. However, these orders are invisible. I doubt the majority of holders are moving BTC on exchange to sell at these current prices, so the exchange order book data is in essence incomplete. This is why the on-chain analysis data is so compelling as it gives an insight into this wider “complete” order book.

To summarise bitcoin is an odd asset due to some nuances of its decentralised nature, lack of prime brokerage and the decentralised exchanges on which it transacts. It is volatile however its volatility is exacerbated by the inability to see the complete market which is where buyers and sellers would be willing to transact their whole holding as opposed to the thin slice they have on exchange.

This is not investment advice. Cryptos are highly risky assets.

Note

I noticed recently that this blog had been added to Quantocracy. It’s nice to see people reading my ramblings and it has motivated me to write more. To those of you who have written directly to me. Thank you, your questions are really helpful, both to clarify ideas of mine and also to think about topics to post about.