This library allows you to send logs or lookups to the Devo platform.

• Data and lookups sending merged in one procedure
• Allows to send real time data
• Logger integration and logging handler capacity for Sender

• Tested compatibility between python 2.x and 3.x series
• Unit tests for both python 2.x and 3.x

Before sending the lookup information it is necessary to initialize the collector configuration

There are different ways (and types) to initialize the collector configuration

Variable descriptions

• address (string): host name to send data
• port (int): port
• cert_reqs (boolean_): indicates if certificate is required
• key (string): key file path
• cert (string): cert file path
• chain (string): chain file path
• tag (string): remote table name

• With certificates:

   engine_config = SenderConfigSSL(address=SERVER, port=PORT,key=KEY, cert=CERT,chain=CHAIN)
   con = Sender(engine_config)

• Without certificates SSL

   engine_config = SenderConfigSSL(address=SERVER, port=PORT, cert_reqs=False)
   con = Sender(engine_config)

• Without certificates TCP

   engine_config = SenderConfigTCP(address=SERVER, port=PORT)
   con = Sender(engine_config)

• From config function - TCP example

    con = Sender.from_config({"address": "relayurl", "port": 443, "type": "TCP"})

• From config function - SSL example

    con = Sender.from_config({"address": "relayurl", "port": 443, "key": "/tmp/key.key", "cert": "/tmp/cert.cert", "chain": "/tmp/chain.crt"})

• From a file

The file must contain a json format structure with the values into sender variable. The variables will depend of certificate type.

This is a example:

{   
    "sender": {
	        "address":"devo-relay",
	        "port": 443,
	        "key": "/devo/certs/key.key",
	        "cert": "/devo/certs/cert.crt",
	        "chain": "/devo/certs/chain.crt"
	    },
}

To initialize the collector configuration from a file we need to import Configuration class

from devo.common import Configuration

conf = Configuration()
conf.load_json("./config.json.example", 'sender')
config = conf.get()
engine_config = Sender.from_config(config)

In order to use Sender as an Handler, for logging instances, the tag property must be set either through the constructor or using the object method: set_logger_tag(tag).

The regular use of the handler can be observed in this 3 examples:

######Setting up configuration variables.

tag = 'test.dump.free'
engine_config = SenderConfigSSL(address=server, port=port,
                                        key=key, cert=cert,
                                        chain=chain)
                    

######First example: Setting up tag after Sender is created

logger = logging.getLogger('DEVO_logger')
# tag added after Sender is created
con = Sender(engine_config)
con.set_logger_tag(tag)
logger.addHandler(con)

######Second example: Setting up a Sender with tag

logger = logging.getLogger('DEVO_logger')
#Sender created ready to be used
con = Sender(engine_config, tag)
logger.addHandler(con)

######Third example: Setting up a static Sender

engine_config = {"address": server, "port": port,
                             "key": key, "cert": cert,
                             "chain": chain, "type": "SSL", "cert_regs": True}
logger = logging.getLogger('DEVO_logger')
#Static Sender
con = Sender.for_logging(engine_config, "SSL", tag)
logger.addHandler(con)

• After we use the configuration class, we will now be able to send events to the collector

con = Sender(engine_config)   

• send logs to the collector,

con.send(tag="test.drop.actors", msg='Hasselhoff vs Cage')

• Send raw log to collector

con.send_raw('<14>Jan  1 00:00:00 Nice-MacBook-Pro.local'
        'test.drop.actors: Testing this random tool')

• log_format (string): Log format to send
• facility (int): facility user
• severity (int): severity info
• hostname (string): set hostname machine
• multiline (bool): Default False. For multiline msg
• zip (bool): Default False. For send data zipped

With the Devo Sender you can make a compressed delivery to optimize data transfer, with the restriction that you have to work with bytes (default type for text strings in Python 3) and not with str.

con = Sender(engine_config) 
con.send(tag=b"test.drop.actors", msg=b'Hasselhoff vs Cage', zip=True)
con.flush_buffer()

The compressed delivery will store the messages in a buffer that, when it is filled, will compress and send, so you have to take into account that you have to execute the flush_buffer function at the end of the data transfer loop, to empty the possible data that have been left uncompressed and send.

Its important flush the buffer when you're done using it.

The default buffer length its 19500 and you can change it with:

con.max_zip_buffer = 19500

You can change the default compression level with:

con.compression_level = 6

compression_level is an integer from 0 to 9 or -1 controlling the level of compression; 1 (Z_BEST_SPEED) is the fastest and produces the lower compression, 9 (Z_BEST_COMPRESSION) is the slowest and produces the highest compression. 0 (Z_NO_COMPRESSION) has no compression. The default value is -1 (Z_DEFAULT_COMPRESSION). Z_DEFAULT_COMPRESSION represents a default compromise between speed and compression (currently equivalent to level 6).

###Lookup

Just like the send events case, to create a new lookup or send data to existent lookup table we need to initialize the collector configuration (as previously shown).

In case to initialize the collector configuration from a json file, you must include a new object into the lookup variable with the next parameters:

• name: lookup table name
• file: CSV file path
• lkey: lookup column key

Example:

{   
    "lookup": {
        "name": "Test_Lookup_of_180306_02",
        "file": "test_lookup.csv",
        "lkey": "KEY"
    }
}

After initializing the colletor, you must initialize the lookup class.

• name (string): lookup table name
• historic_tag (boolean): save historical
• con (LtSender): Sender conection

    lookup = Lookup(name=config['name'], historic_tag=None, con=con)

After initializing the lookup, you can upload a CSV file with the lookup data by send_csv method from LtLookup class.

Params

• path (string required): CSV file path
• has_header (boolean default: True): CSV has header
• delimiter (string default: ','): CSV delimiter
• quotechar (string default: '"'): CSV quote char
• headers (list default: []): header array
• key (string default: 'KEY'): lookup key
• historic_tag (string default: None): tag

Example

    lookup.send_csv(config['file'], headers=['KEY', 'COLOR', 'HEX'], key=config['lkey'])

Complete example

conf = Configuration()
conf.load_json("./config.json.example", 'sender')
conf.load_json("./config.json.example", 'lookup')
config = conf.get()
con = Sender.from_config(config)
lookup = Lookup(name=config['name'], historic_tag=None, con=con)
with open(config['file']) as f:
    line = f.readline()

lookup.send_csv(config['file'], headers=line.rstrip().split(","), key=config['lkey'])

con.socket.shutdown(0)

After initializing the lookup, you can send data to the lookup. There are two ways to do this.

The first option is to generate a string with the headers structure and then send a control instruction to indicate the start or the end of operation over the lookup. Between those control instructions must be the operations over every row of the lookup.

The header structure is an object list with values and data types of the lookup data.

Example:

[{"KEY":{"type":"str","key":true}},{"HEX":{"type":"str"}},{"COLOR":{"type":"str"}}]

To facilitate the creation of this string we can call list_to_headers method of Lookup class.

Params

• lst (list required): column names list of the lookup
• key (string required): key column name
• type (string default: 'str'): column data type

Example:

pHeaders = Lookup.list_to_headers(['KEY','HEX', 'COLOR'], 'KEY')

With this string we can call send_control method of Sender class to send the control instruction.

Params

• type (string required 'START'|'END'): header type
  • START: start of header
  • END: end of header
• headers (string required): header structure
• action (string required 'FULL'|'INC'): action type
  • FULL: delete previous lookup data and then add the new
  • INC: add new row to lookup table

Example:

lookup.send_control('START', p_headers, 'INC')

The other option is basically the same operations but with less instructions. With send_headers method of LtLookup class we can unify two instructions.

The relevant difference is that we louse control over data types of lookup data. The data type will be a string.

Params

• headers (list default: [] ): columna name list of lookup
• key (string default 'KEY'): column name of key
• event (string default: 'START'): header event
  • START: start of header
  • END: end of header
• headers (string required): header structure
• action (string required 'FULL'|'INC'): action type
  • FULL: delete previous lookup data and then add the new
  • INC: add new row to lookup table

Example:

lookup.send_headers(headers=['KEY', 'HEX', 'COLOR'], key='KEY', event='START', action='FULL')

Finally, to send a new row we can use send_data_line method from LtLooup class.

Params

• key (string default:'key'): key value
• fields (list default: []): values list
• delete (boolean default: False): row must be deleted

Example:

lookup.send_data_line(key="11", fields=["11", "HEX11", "COLOR11" ])

A complete example to send a lookup row is:

conf = Configuration()
conf.load_json("./config.json.example", 'sender')
conf.load_json("./config.json.example", 'lookup')
config = conf.get()
con = Sender.from_config(config)
lookup = Lookup(name=config['name'], historic_tag=None, con=con)

pHeaders = Lookup.list_to_headers(['KEY','HEX', 'COLOR'], 'KEY')
lookup.send_control('START', pHeaders, 'INC')
lookup.send_data_line(key="11", fields=["11", "HEX11", "COLOR11" ])
lookup.send_control('END', pHeaders, 'INC')

con.socket.shutdown(0)

A simplify complete example to send a row of lookup is:

conf = Configuration()
conf.load_json("./config.json.example", 'sender')
conf.load_json("./config.json.example", 'lookup')
config = conf.get()
con = Sender.from_config(config)
lookup = Lookup(name=config['name'], historic_tag=None, con=con)

lookup.send_headers(headers=['KEY', 'HEX', 'COLOR'], key='KEY', event='START')
lookup.send_data_line(key="11", fields=["11", "HEX12", "COLOR12"], delete=True)
lookup.send_headers(headers=['KEY', 'HEX', 'COLOR'], key='KEY', event='END')

con.socket.shutdown(0)

NOTE:

• The start and end control instructions should have the list of the names of the columns in the same order in which the lookup was created.
• Keep in mind that the sockets must be closed at the end

You can use one optional configuration file in the client commands

To send info us the "sender" key, with information to send to Devo. If you want to add lookup info, you need use the "lookup" key.

A configuration file does not require all the keys, you can pass the common values: url, port, certificates. After that you can send the tag, the upload file, and so on, along with the function call.

Both things are combined at runtime, prevailing the values that are sent as arguments of the call over the configuration file

Priority order:

1. -c configuration file option: if you use ite, CLI search key, secret and url, or token and url in the file
2. params in CLI call: He can complete values not in configuration file, but does not overrides it
3. Environment vars: if you send the key, secrkey or token in config file or params cli, this option will not be called
4. ~/.devo.json: if you send the key, secrey or token in other ways, this option will not be called

Config file example:

  {
    "sender": {
      "address":"devo-relay",
      "port": 443,
      "key": "/devo/certs/key.key",
      "cert": "/devo/certs/cert.crt",
      "chain": "/devo/certs/chain.crt"
    },
    "lookup": {
      "name": "Test lookup",
      "file": "/lookups/lookup.csv",
      "lkey": "KEY"
    }
  }

You can see another example in docs/common/config.example.json

data command is used to send logs to Devo

Usage: devo-sender data [OPTIONS]

  Send to devo

Options:
  -c, --config PATH   Optional JSON File with configuration info.
  -a, --address TEXT  Devo relay address
  -p, --port TEXT     Devo relay address port
  --key TEXT          Devo user key cert file.
  --cert TEXT         Devo user cert file.
  --chain TEXT        Devo chain.crt file.
  --cert_reqs/
  --no-cert_reqs BOOL   Boolean to indicate if the shipment is done using security certificates or not.
  --multiline/
  --no-multiline BOOL Flag for multiline (With break-line in msg). Default is False.
  --type TEXT         Connection type: SSL or TCP
  -t, --tag TEXT      Tag / Table to which the data will be sent in Devo.
  -l, --line TEXT     For shipments of only one line, the text you want to
                      send.
  -f, --file TEXT     The file that you want to send to Devo, which will
                      be sent line by line.
  -h, --header TEXT   This option is used to indicate if the file has headers
                      or not, they will not be send.
  --help              Show help message and exit.

Examples

#Send test line to table "test.drop.ltsender"
devo-sender data -c ~/certs/config.json

#Send line to table "unknown.unknown"
devo-sender data -c ~/certs/config.json -l "True Survivor - https://www.youtube.com/watch?v=ZTidn2dBYbY"

#Send all file malware.csv (With header) to table "my.app.test.malware"
devo-sender data -c ~/certs/config.json -t my.app.test.films -f "/SecureInfo/my-favorite-disney-films.csv" -h True

#Send file malware.csv (Without header) to table "my.app.test.malware" without config file, using the call to put all info directly
devo-sender data -a app.devo.com -p 10000 --key ~/certs/key.key --cert ~/certs/cert.crt --chain ~/certs/chain.crt  -t my.app.test.films -f "/SecureInfo/my-favorite-disney-films.csv" -h True

You have example file in the "tests" folder of the project for a simple, and most useful example). All the values must be at the same level and without "-"

lookup command is used to send lookups to Devo

Usage: devo-sender lookup [OPTIONS]

  Send csv lookups to devo

Options:
  -c, --config PATH      Optional JSON File with configuration info.
  -a, --address TEXT     Devo relay address
  -p, --port TEXT        Devo relay address port
  --key TEXT             Devo user key cert file.
  --cert TEXT            Devo user cert file.
  --chain TEXT           Devo chain.crt file.
  --cert_reqs/
  --no-cert_reqs BOOL   Boolean to indicate if the shipment is done using security certificates or not.
  --type TEXT            Connection type: SSL or TCP
  -n, --name TEXT        Name for Lookup.
  -f, --file TEXT        The file that you want to send to Devo, which
                         will be sent line by line.
  -lk, --lkey TEXT       Name of the column that contains the Lookup key. It 
                         has to be the exact name that appears in the header.
  -d, --delimiter TEXT   CSV Delimiter char.
  -qc, --quotechar TEXT  CSV Quote char.
  --help                 Show this message and exit.

Example

#Send lookup when all Devo data is in config file
devo-sender lookup -c ~/certs/config.json -n "Test Lookup" -f "~/tests/test_lookup.csv -lk "KEY"