Skip to content

api.md

Latest commit

 

History

History
449 lines (357 loc) · 17.1 KB

api.md

File metadata and controls

449 lines (357 loc) · 17.1 KB

Devo API (Client)

Overview

This library performs queries to the Client API (Search rest api) of Devo.

USAGE

Constructor

  • key: The key of the domain
  • secret: The secret of the domain
  • url(optional): The url of the service. A static constants are provided with the commons clouds: can take several values, for example:
  • buffer: Buffer object if you modify the Devo Buffer class
from devo.api import Client

api = Client(key="myapikey",
              secret="myapisecret",
              url="https://api-eu.logtrust.com/search/query")

query() params

def query(self, **kwargs)

  • query: Query to perform
  • query_id: if not query, you can send the query id
  • dates: {'from': string, 'to': string} -> Date from and date to
  • format: Type of response from Client API
  • limit: Limits of rows returned
  • offset: Row number by which to start returning data
  • proccessor: Callback for process returned object/s
  • stream: Not wait for all response for return lines - real time mode if not "to" in dates
  • response: response type

return: Result of the query or Buffer object

buffer = api.query(query="from my.app.web.activityAll select * limit 10",
                     dates= {'from': "2018-02-06 12:42:00"},
                     response="json/simple/compact")

buffer.set_timeout(3) #Max wait for data, 0 is no timeout/wait forever
sleep(3)
try:
    while not buffer.is_empty():
        data = buffer.get(timeout=10) #You can send timeout for buffer or per query, if some queries need more timeout
        print(data)
except Exception as error:
    print(error)

Query by id has the same parameters as query (), changing the field "query" to "query_id", which is the ID of the query in Devo.

You need to verify that data has data, in addition, you can not use is_empty() when you are verifying queries in real time, when there is the possibility of spaces without data, you would have to use an infinite loop, for example.

close()

The Queue works on a secondary thread (Son) of the main program, so it will die when the program ends, but it is recommended to close the thread / queue if it is not used, to avoid problems.

The thread should close when using buffer.close ()

The "from" and "to", formats and other stuff...

Here we define the start and end of the query (query eventdate filter are secondary), those are the limits of the query.

From                                                                          To
|-----------------|OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO|------------------------|
                today()  <=    eventdate      <=    now()

Date Formats

  • Fixed format: As described on Official Python Docs. Accepted formats are:
    • '%Y-%m-%d %H:%M:%S'
    • '%Y-%m-%d', the time will be truncated to 00:00:00
  • Timestamp: From epoch in millis
  • Dynamic expression: Using the LinQ sintax we can use several functions
    • Relative functions:
      • now(): Current date and time
      • today(): Current date and time fixed to 00:00:00
      • yesterday(): Current date minus one day and time fixed to 00:00:00
    • Amount functions:
      • second(): Return 1
      • minute(): Return 60
      • hour(): Return 60 * 60
      • day(): Return 24 * 60 * 60
      • week(): Return 7 * 24 * 60 * 60
      • month(): Return 30 * 24 * 60 * 60

CLI USAGE

Usage: devo-api query [OPTIONS]

Perform query by query string

Options:
  --config PATH           JSON File with configuration, you can put all
                          options here
  -u, --url TEXT          Endpoint for the api.
  --apiKey TEXT           Key for the api.
  --apiSecret TEXT        Secret for the api.
  --query TEXT            Query.
  --stream / --no-stream  Flag for make streaming query or full query with
                          start and end. Default is true
  --proc                  if flag exists, dont return raw query reply. In
                          compact replies you receive proccessed lines.
  --output TEXT           File path to store query response if not want stdout
  -f, --format TEXT       The output format. Default is json/simple/compact
  --from TEXT             From date, and time for the query (YYYY-MM-DD
                          hh:mm:ss). For valid formats see devo.common README
  --to TEXT               To date, and time for the query (YYYY-MM-DD
                          hh:mm:ss). For valid formats see devo.common README
  --help                  Show this message and exit.

A configuration file does not have to have all the necessary keys, you can have the common values: url, port, certificates. And then send with the call the tag, file to upload, etc.

Both things are combined at runtime, prevailing the values that are sent as arguments of the call over the configuration file

Config file key: The CLI uses the "api" key to search for information. You can see one example in docs/common/config.example.json

  {
    "api": {
      "key": "MyAPIkeytoaccessdevo",
      "secret": "MyAPIsecrettoaccessdevo",
      "url": "https://api-us.logtrust.com/search/query"
    }
  }

You can use environment variables or a global configuration file for the KEY, SECRET and URL values

Priority order:

  1. -c configuration file option: if you use ite, CLI search key, secret and url, or token and url in the file
  2. params in CLI call: He can complete values not in configuration file, but not override it
  3. Environment vars: if you send key, secrey or token in config file or params cli, this option not be called
  4. ~/.devo.json: if you send key, secrey or token in other way, this option not be called

Environment vars are: DEVO_API_URL, DEVO_API_KEY and DEVO_API_SECRET

Choosing Fomat

The default response format (format) is json, to change the default value set for example:

api.format = 'json/compact'

To change the response format (format) of a query, just change the value of the attribute response of the query call

response = api.query(config['query'], 
                     date_from=config['from'], 
                     date_to=config['to'], 
                     format=config['format'])

Type allow the following values:

· json
· json/compact
· json/simple
· json/simple/compact
· msgpack
· csv (comma separated values)
· tsv (Tab separated Values)

Response type JSON

When format set to json, response is a Json Object with the following structure:

Field Name Type Description
success boolean True → ok
False → error
msg String Message Description in case of error
status Integer Numeric value that especify the error code.
0 - OK
1 - Invalid request
object Json Object The Query Result. The format of the content, depends on the Query data.

Example

    HTTP/1.1 200 OK
    Access-Control-Allow-Origin: *
    Content-Encoding: gzip
    Content-Type: application/json
    Date: Mon, 24 Oct 2016 14:39:48 GMT
    Server: nginx
    X-Content-Type-Options: nosniff
    X-XSS-Protection: 1; mode=block
    transfer-encoding: chunked
    Connection: keep-alive
    
    
    {
     "success": true,
     "status": 0,
     "msg": "valid request",
     "object": [
       {
         "eventdate": "2016-10-24 06:35:00.000",
         "host": "aws-apiodata-euw1-52-49-216-97",
         "memory_heap_used": "3.049341704E9",
         "memory_non_heap_used": "1.21090632E8"
       },
       {
         "eventdate": "2016-10-24 06:36:00.000",
         "host": "aws-apiodata-euw1-52-49-216-97",
         "memory_heap_used": "3.04991028E9",
         "memory_non_heap_used": "1.21090632E8"
       },
       {
         "eventdate": "2016-10-24 06:37:00.000",
         "host": "aws-apiodata-euw1-52-49-216-97",
         "memory_heap_used": "3.050472496E9",
         "memory_non_heap_used": "1.21090632E8"
       },
      ………
    }

Response type json/compact

When format set to json/compact, response is a Json Object with the following structure:

Field Name Type Description
success boolean True → ok
False → error
msg String Message Description in case of error
status Integer Numeric value that especify the error code.
0 - OK
1 - Invalid request
object Json Object
object.m Json Object Json Object with Metadata information, the key is the name of the field, and the value is an Object with the following information:.
  • type: type of the value returned:
    • timestamp: epoch value in milliseconds
    • str: string
    • int8: 8 byte integer
    • int4: 4 byt integer
    • bool: boolean
    • float8: 8 byte floating point.
  • index: integer value, that points to where in the array of values is the value of this field
object.d Json Object Array of Arrays with the values of the response of the query.

Example

{
    "msg": "",
    "status": 0,
    "object": {
        "m": {
            "eventdate": {
                "type": "timestamp",
                "index": 0
            },
            "domain": {
                "type": "str",
                "index": 1
            },
            "userEmail": {
                "type": "str",
                "index": 2
            },
            "country": {
                "type": "str",
                "index": 3
            },
            "count": {
                "type": "int8",
                "index": 4
            }
        },
"d": [
        [
            1506442210000,
            "self",
            "[email protected]",
            null,
            2
        ],
        [
            1506442220000,
            "self",
            "[email protected]",
            null,
            2
        ],
    .....
    ]
}

Response type json/simple

When format set to json/simple
The response is a stream of Json Objects with the following structure of the values that the Query generates, separated each registry CRLF.

When the query does not generate more information, the connection is closed by the server.

In case, no date to is requested, the connections is keeped alive.

Example

{"eventdate":1488369600000,"domain":"none","userEmail":"","country":null,"count":3}
{"eventdate":1488369600000,"domain":"[email protected]","userEmail":"0:0:0:0:0:0:0:1","country":null,"count":18}
{"eventdate":1488369600000,"domain":"none","userEmail":"[email protected]","country":null,"count":7}
{"eventdate":1488373200000,"domain":"[email protected]","userEmail":"127.0.0.1","country":null,"count":10}
{"eventdate":1488373200000,"domain":"[email protected]","userEmail":"0:0:0:0:0:0:0:1","country":null,"count":28}
{"eventdate":1488373200000,"domain":"self","userEmail":"[email protected]","country":null,"count":15}
{"eventdate":1488373200000,"domain":"self","userEmail":"[email protected]","country":null,"count":49}
{"eventdate":1488376800000,"domain":"[email protected]","userEmail":"127.0.0.1","country":null,"count":10}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":16}
{"eventdate":1488376800000,"domain":"[email protected]","userEmail":"0:0:0:0:0:0:0:1","country":null,"count":5}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":35}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":128}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":7}
{"eventdate":1488376800000,"domain":"self","userEmail":"[email protected]","country":null,"count":3}
{"eventdate":1488380400000,"domain":"[email protected]","userEmail":"127.0.0.1","country":null,"count":21}
{"eventdate":1488380400000,"domain":"self","userEmail":"[email protected]","country":null,"count":71}
{"eventdate":1488380400000,"domain":"[email protected]","userEmail":"0:0:0:0:0:0:0:1","country":null,"count":38}
{"eventdate":1488380400000,"domain":"self","userEmail":"[email protected]","country":null,"count":1}
...

Reponse type json/simple/compact

When format set to json/simple/compact
The response is a stream of Json Objects with the following structure each line is separated by CRLF:

The First Line is a JSON object map with the Metadata information, the key is the name of the field, and the value is a Object with the following information:

{"m":{"eventdate":{"type":"timestamp","index":0},"domain":{"type":"str","index":1},"userEmail":{"type":"str","index":2},"country":{"type":"str","index":3},"count":{"type":"int8","index":4}}}
  • type: type of the value returned:
    • timestamp: epoch value in milliseconds
    • str: string
    • int8: 8 byte integer
    • int4: 4 byt integer
    • bool: boolean
    • float8: 8 byte floating point.
  • index: integer value, that points to where in the array of values is the value of this field

The rest of the lines are data lines:

{"d":[1506439800000,"self","[email protected]",null,1]}

a field with name "d", gives access to the array of values with the information.

When the query does not generate more information, the connection is closed by the server.
In case, no date to is requested, the connections is keeped alive.

Example

{"m":{"eventdate":{"type":"timestamp","index":0},"domain":{"type":"str","index":1},"userEmail":{"type":"str","index":2},"country":{"type":"str","index":3},"count":{"type":"int8","index":4}}}
{"d":[1506439800000,"self","[email protected]",null,1]}
{"d":[1506439890000,"self","[email protected]",null,1]}
{"d":[1506439940000,"self","[email protected]",null,1]}
{"d":[1506439950000,"self","[email protected]",null,1]}
{"d":[1506440130000,"self","[email protected]",null,1]}
{"d":[1506440130000,"self","[email protected]",null,2]}
{"d":[1506440170000,"self","[email protected]",null,3]}
{"d":[1506440200000,"self","[email protected]",null,3]}
{"d":[1506440220000,"self","[email protected]",null,1]}
{"d":[1506440230000,"self","[email protected]",null,1]}
{"d":[1506440260000,"self","[email protected]",null,3]}
{"d":[1506440270000,"self","[email protected]",null,1]}
{"d":[1506440280000,"self","[email protected]",null,1]}
{"d":[1506440280000,"self","[email protected]",null,1]}
{"d":[1506440290000,"self","[email protected]",null,3]}
{"d":[1506440350000,"self","[email protected]",null,1]}

Response type msgpack

When response format set to msgpack
The response format is the same that the Json format, but encoded using MsgPack an efficient binary serialization format (http://msgpack.org/)

Response type csv

When format set to csv
The system returns the information in CSV(Comma Separated Values) format, as follows

Example

eventdate,domain,userEmail,country,count
2017-03-01 12:00:00.000,none,,,3
2017-03-01 12:00:00.000,[email protected],0:0:0:0:0:0:0:1,,18
2017-03-01 12:00:00.000,none,[email protected],,7
2017-03-01 13:00:00.000,[email protected],127.0.0.1,,10
2017-03-01 13:00:00.000,[email protected],0:0:0:0:0:0:0:1,,28
2017-03-01 13:00:00.000,self,[email protected],,15
2017-03-01 13:00:00.000,nombre,[email protected],,49
2017-03-01 14:00:00.000,[email protected],127.0.0.1,,10
2017-03-01 14:00:00.000,self,[email protected],,16
2017-03-01 14:00:00.000,[email protected],0:0:0:0:0:0:0:1,,5
2017-03-01 14:00:00.000,self,[email protected],,35
2017-03-01 14:00:00.000,nombre,[email protected],,128
2017-03-01 14:00:00.000,self,[email protected],,7
2017-03-01 14:00:00.000,self,[email protected],,3
2017-03-01 15:00:00.000,[email protected],127.0.0.1,,21
2017-03-01 15:00:00.000,self,[email protected],,71
2017-03-01 15:00:00.000,[email protected],0:0:0:0:0:0:0:1,,38
2017-03-01 15:00:00.000,self,[email protected],,1
2017-03-01 15:00:00.000,nombre,[email protected],,80

Response type tsv

When format set to tsv
The system returns the information in TSV (Tab Separated Values) format as follows

Example

eventdate   domain  userEmail   country count
2017-03-01 12:00:00.000 none            3
2017-03-01 12:00:00.000 [email protected]    0:0:0:0:0:0:0:1     18
2017-03-01 12:00:00.000 none    [email protected]        7
2017-03-01 13:00:00.000 [email protected]    127.0.0.1       10
2017-03-01 13:00:00.000 [email protected]    0:0:0:0:0:0:0:1     28
2017-03-01 13:00:00.000 self    [email protected]     15
2017-03-01 13:00:00.000 nombre [email protected]       49
2017-03-01 14:00:00.000 [email protected]    127.0.0.1       10
2017-03-01 14:00:00.000 self    [email protected]        16
2017-03-01 14:00:00.000 [email protected]    0:0:0:0:0:0:0:1     5
2017-03-01 14:00:00.000 self    [email protected]     35
2017-03-01 14:00:00.000 nombre [email protected]       128
2017-03-01 14:00:00.000 self    [email protected]       7
2017-03-01 14:00:00.000 self    [email protected]     3
2017-03-01 15:00:00.000 [email protected]    127.0.0.1       21
2017-03-01 15:00:00.000 self    [email protected]        71
2017-03-01 15:00:00.000 [email protected]    0:0:0:0:0:0:0:1     38