The Requests Stampede library is a wrapper around the Requests library that provides request retry logic and backoff delays. The goal of the project is to improve upstream service stability and time to recovery following an outage and influx of retry requests. The library provides various default configurations with sane parameters; however, behavior is configurable via both programmatic declarations and configuration files.

The Request Stampede library was largely inspired by the 2021-01-15 Signal messaging service outage wherein a service disruption was worsened by their mobile app retry-logic inflicting a self-induced DDoS attack on their systems. The project implementes many of the recommendations outlined in Google Cloud's blog post on the subject.

import RequestsStampede.horde

horde = RequestsStampede.horde.RetryRequest()

response = horde.get("https://www.example.com/")
print(response)

import RequestsStampede.horde

session = RequestsStampede.horde.RetrySession()

response = session.post(
    "https://www.example.com/login",
    data={
        'username': 'johndoe',
        'password': 'hunter2'
    }
)
print(response)

response = session.get("https://www.example.com/profile")
print(response)

import requests.session
import RequestsStampede.horde

session = RequestsStampede.horde.RetrySession(
    session=requests.session.Session(
        headers={
            "Authorization": "Bearer <TOKEN>"
        }
    )
)

response = session.post("https://www.example.com/resource")
print(response)

response = session.get("https://www.example.com/resource")
print(response)

stampede.yml

retry_config:
  retry_enabled: True
  retry_policy:
    type: infinite

  backoff_enabled: True
  backoff_policy:
    type: fibonacci
    initial_delay: 0.0
    maximum_delay: 144.0

import requests.session
import RequestsStampede.horde

session = RequestsStampede.horde.RetrySession(
    session=requests.session.Session(
        headers={
            "Authorization": "Bearer <TOKEN>"
        }
    )
)

response = session.post("https://www.example.com/resource")
print(response)

response = session.get("https://www.example.com/resource")
print(response)

The Requests Stampede library supports all HTTP methods currently supported by the Requests library:

• GET
• OPTIONS
• HEAD
• POST
• PUT
• PATCH
• DELETE

When an HTTP request is unsuccessful, that request may be retried depending on the configured retry policy. Following an unsuccessful request or retry attempt, a backoff delay may be introduced in an effort to reduce congestion and system load on the upstream resource.

The fixed retry policy will attempt to retransmit a request until the maximum number of attempts has been reached. Upon exhaustion of all retry attempts, the request will be discarded.

The infinite retry policy will indefinitely retransmit a request until a successful response is received.

Following a failed request, a fixed backoff policy will introduce a constant delay between the next attempted transmission. For example, a delay of 5 seconds may be introduced between request attempts.

Following a failed request, a random backoff policy will introduce a random, normally distributed, delay between two configurable bounds prior to the next attempted transmission. For example, a random delay between 5 and 15 seconds may be introduced between request attempts.

Following a failed request, a series of fibonacci delays will be introduced in between subsequent transmission attempts. For example, a retry attempt sequence may follow the pattern below:

No | Event
---|------
0  | Initial Request, Failure
1  | Backoff, 0 sec
2  | Retry Request, Failure
3  | Backoff, 1 sec
4  | Retry Request, Failure
5  | Backoff, 1 sec
6  | Retry Request, Failure
7  | Backoff, 2 sec
8  | Retry Request, Failure
9  | Backoff, 3 sec
...

The RetryRequest and RetrySession classes accept parameters to configure both the desired retry configuration and request session object.

If configuration parameters are not provided to RetryRequest or RetrySession, RequestsStampede will traverse the file-system for a configuration file, stampede.yml. For example, given an application running in /etc/application, RequestsStampede will traverse the following paths and select the first configuration file encountered:

1. /etc/application/stampede.yml
2. /etc/stampede.yml
3. /stampede.yml
4. $HOME/stampede.yml

If no configuration file is located, a default retry configuration will be utilized.

Requests Stampede exposes two interfaces within the RequestsStampede.horde sub-module, whether you choose RetryRequest or RetrySession is entirely dependent on whether or not you require session persistence across multiple requests.

For basic applications, such as unauthenticated web scraping, RetryRequest is likely suffient. The interface does not offer session persistence across multiple requests by default.

For more complex use-cases, your application may require session persistence to properly authenticate with an upstream resource. The RetrySession interface is best suited for this use case as it will use one session for all requests.

The logging module is used for all logging interfaces within the RequestsStampede module. Logs may be exposed by raising the logging level (i.e. logging.basicConfig(level=logging.DEBUG)).

virtualenv --python=python3 venv
source venv/bin/activate

pip3 install .

pip3 install -r dev-requirements.txt

black .
pylint RequestsStampede
pylint tests

pytest

1. Patrick Murray
2. Zachary Barden

1. The Requests Stampede icon and banner graphics use content created by Game-icons.net which was licensed under a Creative Commons 3.0 license. Minor modifications were made to the content for use by the Requests Stampede project.