Skip to content

Commit b0a2d93

Browse files
lironcohen272DanielTal87
lironcohen272
and
DanielTal87
authored
CIAC-15150 create new ElasticsearchEventCollector integration (demisto#42549)
* DRAFT: create new ElasticsearchEventCollector integration * add logs * after pre-commit * fix tests and delete commands from yml * edit README file * bump rn * implement es-get-event command * fix tests * fix yml * fix after pre commit * fixing after ai-reviewer * add fetch and get event tests * fix duplication events and add traceback * fix after pre-commit * fix after doc review * add get_event to test module * fix test module * change "ElasticsearchEventCollector" to "Elasticsearch Event Collector" * change fetch_size from 50 to 5000 * change fetch_size default value * fix --------- Co-authored-by: Daniel Tal <[email protected]>
1 parent 6f50aa2 commit b0a2d93

10 files changed

Lines changed: 2415 additions & 12 deletions

‎Packs/Elasticsearch/Integrations/ElasticsearchEventCollector/ElasticsearchEventCollector.py‎

Lines changed: 892 additions & 0 deletions
Large diffs are not rendered by default.

‎Packs/Elasticsearch/Integrations/ElasticsearchEventCollector/ElasticsearchEventCollector.yml‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
category: Database
2+
sectionorder:
3+
- Connect
4+
- Collect
5+
commonfields:
6+
id: ElasticsearchEventCollector
7+
version: -1
8+
configuration:
9+
- name: url
10+
display: Server URL
11+
required: true
12+
type: 0
13+
additionalinfo: The Elasticsearch server to which the integration connects. Ensure that the URL includes the correct Elasticsearch port. The default port for Elasticsearch v7 and below is 9200. Use the Server URL for on-premises deployments.
14+
section: Connect
15+
- name: auth_type
16+
display: Authorization type
17+
defaultvalue: Basic auth
18+
additionalinfo: |-
19+
Select the authentication type and enter the appropriate credentials:
20+
- Basic Auth: Enter Username and Password.
21+
- Bearer Auth: Enter Username and Password.
22+
- API Key Auth: Enter the API Key ID and API Key.
23+
options:
24+
- Basic auth
25+
- Bearer auth
26+
- API key auth
27+
type: 15
28+
section: Connect
29+
required: true
30+
- name: api_key_auth_credentials
31+
additionalinfo: Use for API key auth
32+
display: API key ID
33+
displaypassword: API Key
34+
type: 9
35+
section: Connect
36+
required: false
37+
- name: credentials
38+
additionalinfo: Use for Basic auth. Optionally you can use Username as an API key ID and Password as an API key for API Key auth.
39+
display: Username
40+
type: 9
41+
section: Connect
42+
required: false
43+
- name: insecure
44+
display: Trust any certificate (not secure)
45+
type: 8
46+
section: Connect
47+
advanced: true
48+
required: false
49+
- name: proxy
50+
display: Use system proxy settings
51+
type: 8
52+
section: Connect
53+
advanced: true
54+
required: false
55+
- name: client_type
56+
defaultvalue: Elasticsearch
57+
additionalinfo: In some hosted ElasticSearch environments, the standard ElasticSearch client is not supported. If you encounter any related client issues, please consider using the OpenSearch client type.
58+
display: Client type
59+
options:
60+
- Elasticsearch
61+
- OpenSearch
62+
- Elasticsearch_v8
63+
- Elasticsearch_v9
64+
type: 15
65+
section: Connect
66+
advanced: true
67+
required: false
68+
- name: fetch_index
69+
display: Index to fetch incidents from
70+
type: 0
71+
section: Collect
72+
required: false
73+
additionalinfo: CSV
74+
- name: fetch_query
75+
display: Query String
76+
type: 0
77+
additionalinfo: |-
78+
Query string uses the Lucene syntax.
79+
For more information about the Lucene syntax, see the link in the help section.
80+
The query will be used when fetching incidents. Index time field will be used as a filter in the query.
81+
section: Collect
82+
required: false
83+
- name: fetch_time_field
84+
display: Index time field
85+
type: 0
86+
section: Collect
87+
required: false
88+
additionalinfo: The time field used for sorting and limiting results. If using a nested field, separate field names with dot notation.
89+
- name: raw_query
90+
display: Raw Query
91+
type: 12
92+
additionalinfo: |-
93+
Raw Query allows raw DSL queries and will override the 'Query String' Lucene syntax string.
94+
Results will not be filtered or sorted. Use additional parameters in the raw query for these purposes.
95+
For more information see the link in the help section.
96+
section: Collect
97+
advanced: true
98+
required: false
99+
- name: time_method
100+
display: Time field type
101+
defaultvalue: 'Simple-Date'
102+
type: 15
103+
options:
104+
- Simple-Date
105+
- Timestamp-Seconds
106+
- Timestamp-Milliseconds
107+
section: Collect
108+
advanced: true
109+
required: false
110+
additionalinfo: For more information see the explanation in the help section.
111+
- name: map_labels
112+
defaultvalue: 'true'
113+
display: Map JSON fields into labels
114+
type: 8
115+
section: Collect
116+
advanced: true
117+
required: false
118+
- name: fetch_size
119+
defaultvalue: '5000'
120+
display: The maximum number of results per fetch
121+
type: 0
122+
section: Collect
123+
required: false
124+
- name: timeout
125+
display: Request timeout (in seconds).
126+
type: 0
127+
defaultvalue: '60'
128+
section: Connect
129+
advanced: true
130+
required: false
131+
- name: isFetchEvents
132+
display: Fetch events
133+
type: 8
134+
section: Collect
135+
required: false
136+
description: "Search for and analyze data in real time. \n Supports version 6 and later."
137+
display: Elasticsearch Event Collector
138+
name: ElasticsearchEventCollector
139+
script:
140+
commands:
141+
- arguments:
142+
- name: start_time
143+
description: Start time for fetching events. Supports ISO format ("2023-01-01T23:59:59") or natural language ("2 hours ago", "now").
144+
required: true
145+
- name: end_time
146+
description: End time for fetching events. Supports ISO format ("2023-01-01T23:59:59") or natural language ("2 hours ago", "now").
147+
required: false
148+
- name: time_method
149+
auto: PREDEFINED
150+
defaultValue: 'Simple-Date'
151+
predefined:
152+
- Simple-Date
153+
- Timestamp-Seconds
154+
- Timestamp-Milliseconds
155+
required: true
156+
description: For more information see the explanation in the help section.
157+
- name: fetch_size
158+
defaultValue: '10'
159+
description: The maximum number of results per fetch.
160+
required: false
161+
- name: fetch_index
162+
required: false
163+
description: CSV.
164+
- name: fetch_time_field
165+
required: true
166+
description: The time field used for sorting and limiting results. If using a nested field, separate field names with dot notation.
167+
- name: fetch_query
168+
description: Query string uses the Lucene syntax.
169+
required: false
170+
- name: raw_query
171+
type: 12
172+
description: Raw Query allows raw DSL queries and will override the 'Query String' Lucene syntax string.
173+
required: false
174+
description: This command is used for developing/ debugging and is to be used with caution, as it can cause the API request limit to be exceeded.
175+
name: es-get-events
176+
dockerimage: demisto/elasticsearch:1.0.0.5954979
177+
isfetchevents: true
178+
runonce: false
179+
script: '-'
180+
subtype: python3
181+
type: python
182+
marketplaces:
183+
- marketplacev2
184+
- platform
185+
fromversion: 8.4.0
186+
tests:
187+
- No tests (auto formatted)

‎Packs/Elasticsearch/Integrations/ElasticsearchEventCollector/ElasticsearchEventCollector_dark.svg‎

Lines changed: 28 additions & 0 deletions
Loading

‎Packs/Elasticsearch/Integrations/ElasticsearchEventCollector/ElasticsearchEventCollector_description.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
The Elasticsearch Event Collector integration supports Elasticsearch 6.0.0 and later.
2+
This integration was integrated and tested with versions 6.6.2, 7.3, 8.4.1 of Elasticsearch.
3+
4+
## Authentication
5+
6+
There are 3 different authentication [methods](https://www.elastic.co/docs/api/doc/elasticsearch#doc-authentication)
7+
8+
### Basic Auth (http)
9+
10+
To use **Basic Authentication**:
11+
12+
* Select **Basic Auth** from the *Authorization type* dropdown.
13+
* Enter your **Username** in the *Username* field.
14+
* Enter your **Password** in the *Password* field.
15+
16+
### API Key Auth (http_api_key)
17+
18+
To use **API Key Authentication**:
19+
20+
* Select **API Key Auth** from the *Authorization type* dropdown.
21+
* Enter your **API key ID** in the *API key ID* field.
22+
* Enter your **API key** in the *API key* field.
23+
24+
For more info about API Key management see [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/security-api-create-api-key.html)
25+
26+
**Note:** Alternatively, you can select the **Basic Auth** type and enter the API key ID in the *Username* field and the API key in the *Password* field.
27+
Example:
28+
for *API Key ID* kQme5aOx enter: _api_key_id:kQme5aOx
29+
for *API Key* ui2lp2axT enter: ui2lp2axT
30+
31+
### Bearer Auth (http)
32+
33+
To use **Bearer Authentication**:
34+
35+
* Select **Bearer Auth** type from the *Authorization type* dropdown.
36+
* Enter your **Username** in the *Username* field.
37+
* Enter your **Password** in the *Password* field.
38+
39+
For more info see [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/security-api-get-token.html#security-api-get-token-prereqs)
40+
41+
## Notes
42+
43+
* Not all fields can be used for sorting in Elasticsearch. Sorting is only supported for fields of the following types: **boolean**, **numeric**, **date**, and **keyword**.
44+
* The "Test" button does not fully validate the fetch events functionality.
45+
46+
## Additional Configuration Parameters Details
47+
48+
Fetch events requires:
49+
50+
* Index
51+
* Index time field
52+
* Query String or Raw Query
53+
54+
For further information about type mapping, see [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/mapping.html#mapping-type).
55+
56+
**Query String**
57+
Query String is queried using the Lucene syntax. For more information about Lucene syntax see [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/query-dsl-query-string-query.html#query-string-syntax).
58+
59+
**Raw Query**
60+
Allows raw DSL queries. For more information about Query DSL see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html).
61+
62+
**Time field type**
63+
3 formats supported:
64+
65+
* Simple-Date - A plain date string. You must specify the format in which the date is stored. For more information about time formatting, see [here](http://strftime.org/).
66+
* Timestamp-Second - A numeric value representing the number of seconds since the Unix epoch (00:00:00 UTC on 1 January 1970). Example: '1572164838'
67+
* Timestamp-Milliseconds - A numeric value representing the number of milliseconds since the Unix epoch. Example: '1572164838123'

‎Packs/Elasticsearch/Integrations/ElasticsearchEventCollector/ElasticsearchEventCollector_image.png‎

3.59 KB
Loading

0 commit comments

Comments
 (0)