Easily export your Sumologic log data.

NOTE: I no longer use Sumologic in my day-to-day life and am looking for a maintainer for this project. If you'd like to maintain this project, please get in touch with me via my email address: [email protected]. Thank you!

At Stormpath we use Sumologic to store our server logs.

We were preparing to export our logs for safe keeping, so I wrote this tool. It uses the Sumologic API and downloads every log stored over a predefined time period.

sumologic-export will grab every single Sumologic log you've ever written and store it in gzipped JSON files.

Installing sumologic-export is simple -- just use pip!

Once you have pip installed on your computer, you can run the following to install the latest release of sumologic-export:

$ pip install -U sumologic-export

That's it :)

NOTE: Since I first wrote this tool, Sumologic has changed the rules surrounding their search API. What this means is that you can only use this tool if you are either using a Trial or Enterprise account. All other types of Sumologic accounts do not have access to the underlying API that this tool uses. Thank you Tony Tovar for bringing this to my attention!

Before you can export all your Sumologic data, you'll need to configure sumologic-export and give it your Sumologic credentials. To do this, simply run:

$ sumologic-export configure

On the command line. This will prompt you for some basic information, then store your credentials in the local file ~/.sumo by default; you can pass the --config argument to specify a different location.

Next, to run a backup job, you can run:

$ sumologic-export

This will export all your Sumologic data for the past month and dump it into a new directory named exports, broken into files covering 60 minutes per file.

If you'd like to specify a custom date range, you can do so by adding a start and stop date, for instance:

$ sumologic-export 2014-01-01 2014-06-01

Or, if you'd like, you can just specify a start date, and the exporter will export all logs from the start date till the current day.

$ sumologic-export 2014-01-01

There are several date parsing formats supported:

• YYYY-MM-DD
• YYYY-MM-DDTHH:MM:SSZ
• YYYY-MM-DDTHH:MM:SS.SSSZ

You can also specify a time in Unix epoch timestamp format.

NOTE: Depending on how many logs you have, this process may take a while.

Once the process is finished, you'll have an exports directory populated with gziped JSON files. There will be one JSON file for each hour by default, for instance:

$ ls exports
2014-01-01T00-00-00.json.gz
2014-01-01T01-00-00.json.gz

To change the duration each file covers, you can set the --slice argument equal to the number of minutes a file should contain. For exmaple, you can set --slice=1440 to make each file cover 1440 minutes, or 24 hours.

SumoLogic has a maximum number of events a search job will return. If this limit is reached, the job will go into a FORCE PAUSED state. When this is detected, the job is split up in half (so if the slice is the default 60 minutes, the job will be converted into 2 jobs covering 30 minutes each). This will happen recursively until a slice is small enough to not run into the limit. For more information on limits, see their API documentatnion

For full usage information, run sumologic-export -h.

Need help? Can't figure something out? If you think you've found a bug, please open an issue on the GitHub issue tracker.

Otherwise, shoot me an email!

v0.9.0: 09-2020

- Updated syntax for Python 3
- Added logging to handle messages
- Added several arguments, including slice duration, config file location,
  and logging options
- Added recursion to handle when search job hits FORCE PAUSED limit
- Added delete job function to clean up when job is done
- General consistency updates

v0.0.2: 01-19-2015

- Fixing off-by-one error in pagination logic. This was causing us to NOT
  download the last page of logs :(  Thanks
  [@sumoway](https://github.com/sumoway) for the report!

v0.0.1: 06-25-2014

- First release!  Woo!