Skip to content

Commit 0078ea5

Browse files
lcawllcawl
authored
Adds APM NodeJS and RUM Javascript machine learning examples
2 parents 58a1b3e + 5cdae79 commit 0078ea5

10 files changed

+454
-0
lines changed

‎Machine Learning/Anomaly Detection/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Anomaly Detection
2+
3+
This directory contains example anomaly detection job configurations.
4+
5+
TIP: Kibana can also recognize certain types of data and provide specialized
6+
wizards for that context. For more details, refer to
7+
[supplied anomaly detection configurations](https://www.elastic.co/guide/en/machine-learning/8.0/ootb-ml-jobs.html).

‎Machine Learning/Anomaly Detection/apm_jsbase/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
## APM: RUM Javascript
2+
3+
Detect problematic spans and identify user agents that are potentially causing issues.
4+
These jobs are applicable to data from Elastic APM RUM JavaScript Agents (where
5+
`agent.name` is `js-base`).
6+
7+
### Create anomaly detection jobs and datafeeds
8+
9+
Copy the contents of the appropriate *.json file into the
10+
[create anomaly detection jobs API](https://www.elastic.co/guide/en/elasticsearch/reference/8.0/ml-put-job.html) in the Kibana Dev Console. For example:
11+
12+
```
13+
PUT _ml/anomaly_detectors/abnormal_span_durations_jsbase
14+
{
15+
...
16+
}
17+
```
18+
19+
* `abnormal_span_durations_jsbase.json`: Models the duration of spans. Detects spans that are taking longer than usual to process.
20+
21+
* `anomalous_error_rate_for_user_agents_jsbase.json`: Models the error rate of user agents. Detects user agents that are encountering errors at an above normal rate. This job can help detect browser compatibility issues.
22+
23+
* `decreased_throughput_jsbase.json`: Models the transaction rate of the application. Detects periods during which the application is processing fewer requests than normal.
24+
25+
* `high_count_by_user_agent_jsbase.json`: Models the request rate of user agents. Detects user agents that are making requests at a suspiciously high rate. This job is useful in identifying bots.
26+
27+
For more information about anomaly detection and running machine learning jobs,
28+
refer to [Finding anomalies](https://www.elastic.co/guide/en/machine-learning/8.0/ml-ad-finding-anomalies.html).

‎Machine Learning/Anomaly Detection/apm_jsbase/abnormal_span_durations_jsbase.json‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
{
2+
"groups": [
3+
"apm"
4+
],
5+
"description": "APM JSBase: Looks for spans that are taking longer than usual to process.",
6+
"analysis_config": {
7+
"bucket_span": "15m",
8+
"detectors": [
9+
{
10+
"detector_description": "increased span duration",
11+
"function": "high_mean",
12+
"field_name": "span.duration.us",
13+
"partition_field_name": "span.type"
14+
}
15+
],
16+
"influencers": [
17+
"span.type",
18+
"trace.id",
19+
"span.name",
20+
"service.name"
21+
]
22+
},
23+
"allow_lazy_open": true,
24+
"analysis_limits": {
25+
"model_memory_limit": "128mb"
26+
},
27+
"data_description": {
28+
"time_field": "@timestamp"
29+
},
30+
"custom_settings": {
31+
"created_by": "ml-module-apm-jsbase",
32+
"custom_urls": [
33+
{
34+
"url_name": "APM",
35+
"time_range": "2h",
36+
"url_value": "apm#/traces?rangeFrom=$earliest$&rangeTo=$latest$&kuery=trace.id:\"$trace.id$\"&_g=()"
37+
}
38+
]
39+
},
40+
"datafeed_config": {
41+
"indices": [
42+
"apm-*"
43+
],
44+
"max_empty_searches": 10,
45+
"query": {
46+
"bool": {
47+
"must": [
48+
{ "bool": { "filter": { "term": { "agent.name": "js-base" } } } },
49+
{ "bool": { "filter": { "term": { "processor.event": "span" } } } }
50+
]
51+
}
52+
}
53+
}
54+
}

‎Machine Learning/Anomaly Detection/apm_jsbase/anomalous_error_rate_for_user_agents_jsbase.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"groups": [
3+
"apm"
4+
],
5+
"description": "APM JSBase: Detects user agents that are encountering errors at an above normal rate. This can help detect browser compatibility issues.",
6+
"analysis_config": {
7+
"bucket_span": "15m",
8+
"detectors": [
9+
{
10+
"detector_description": "high error rate for user agent",
11+
"function": "high_non_zero_count",
12+
"partition_field_name": "user_agent.name"
13+
}
14+
],
15+
"influencers": [
16+
"user_agent.name",
17+
"error.exception.message.keyword",
18+
"error.page.url",
19+
"service.name"
20+
]
21+
},
22+
"allow_lazy_open": true,
23+
"analysis_limits": {
24+
"model_memory_limit": "32mb"
25+
},
26+
"data_description": {
27+
"time_field": "@timestamp"
28+
},
29+
"custom_settings": {
30+
"created_by": "ml-module-apm-jsbase",
31+
"custom_urls": [
32+
{
33+
"url_name": "APM",
34+
"time_range": "2h",
35+
"url_value": "apm#/services/$service.name$/errors?rangeFrom=$earliest$&rangeTo=$latest$&refreshPaused=true&refreshInterval=0&kuery=user_agent.name:\"$user_agent.name$\"&_g=()"
36+
}
37+
]
38+
},
39+
"datafeed_config": {
40+
"indices": [
41+
"apm-*"
42+
],
43+
"max_empty_searches": 10,
44+
"query": {
45+
"bool": {
46+
"must": [
47+
{ "bool": { "filter": { "term": { "agent.name": "js-base" } } } },
48+
{ "exists": { "field": "user_agent.name" } }
49+
]
50+
}
51+
}
52+
}
53+
}

‎Machine Learning/Anomaly Detection/apm_jsbase/decreased_throughput_jsbase.json‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
{
2+
"groups": [
3+
"apm"
4+
],
5+
"description": "APM JSBase: Identifies periods during which the application is processing fewer requests than normal.",
6+
"analysis_config": {
7+
"summary_count_field_name": "doc_count",
8+
"bucket_span": "15m",
9+
"detectors": [
10+
{
11+
"detector_description": "low throughput",
12+
"function": "low_count"
13+
}
14+
],
15+
"influencers": [
16+
"service.name"
17+
]
18+
},
19+
"allow_lazy_open": true,
20+
"analysis_limits": {
21+
"model_memory_limit": "10mb"
22+
},
23+
"data_description": {
24+
"time_field": "@timestamp"
25+
},
26+
"custom_settings": {
27+
"created_by": "ml-module-apm-jsbase",
28+
"custom_urls": [
29+
{
30+
"url_name": "APM",
31+
"time_range": "2h",
32+
"url_value": "apm#/services?rangeFrom=$earliest$&rangeTo=$latest$&refreshPaused=true&refreshInterval=0&kuery=&transactionType=request"
33+
}
34+
]
35+
},
36+
"datafeed_config": {
37+
"indices": [
38+
"apm-*"
39+
],
40+
"max_empty_searches": 10,
41+
"query": {
42+
"bool": {
43+
"filter": { "term": { "agent.name": "js-base" } }
44+
}
45+
},
46+
"aggregations": {
47+
"buckets": {
48+
"date_histogram": {
49+
"field": "@timestamp",
50+
"fixed_interval": "900000ms"
51+
},
52+
"aggregations": {
53+
"@timestamp": {
54+
"max": {
55+
"field": "@timestamp"
56+
}
57+
}
58+
}
59+
}
60+
}
61+
}
62+
}

‎Machine Learning/Anomaly Detection/apm_jsbase/high_count_by_user_agent_jsbase.json‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
{
2+
"groups": [
3+
"apm"
4+
],
5+
"description": "APM JSBase: Detects user agents that are making requests at a suspiciously high rate. This is useful in identifying bots.",
6+
"analysis_config": {
7+
"bucket_span": "15m",
8+
"detectors": [
9+
{
10+
"detector_description": "high request rate for user agent",
11+
"function": "high_non_zero_count",
12+
"partition_field_name": "user_agent.name"
13+
}
14+
],
15+
"influencers": [
16+
"user_agent.name",
17+
"service.name"
18+
]
19+
},
20+
"allow_lazy_open": true,
21+
"analysis_limits": {
22+
"model_memory_limit": "32mb"
23+
},
24+
"data_description": {
25+
"time_field": "@timestamp"
26+
},
27+
"custom_settings": {
28+
"created_by": "ml-module-apm-jsbase",
29+
"custom_urls": [
30+
{
31+
"url_name": "APM",
32+
"time_range": "2h",
33+
"url_value": "apm#/services/$service.name$/transactions?rangeFrom=$earliest$&rangeTo=$latest$&refreshPaused=true&refreshInterval=0&kuery=user_agent.name:\"$user_agent.name$\"&_g=()"
34+
}
35+
]
36+
},
37+
"datafeed_config": {
38+
"indices": [
39+
"apm-*"
40+
],
41+
"max_empty_searches": 10,
42+
"query": {
43+
"bool": {
44+
"must": [
45+
{ "bool": { "filter": { "term": { "agent.name": "js-base" } } } },
46+
{ "bool": { "filter": [{ "exists": { "field": "user_agent.name" } }] } },
47+
{ "bool": { "filter": { "term": { "processor.event": "transaction" } } } }
48+
]
49+
}
50+
}
51+
}
52+
}

‎Machine Learning/Anomaly Detection/apm_nodejs/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
## APM: NodeJS
2+
3+
Detect abnormal traces, anomalous spans, and identify periods of decreased
4+
throughput. These jobs are applicable to data from Elastic APM Node.js Agents
5+
(where `agent.name` is `nodejs`).
6+
7+
### Create anomaly detection jobs and datafeeds
8+
9+
Copy the contents of the appropriate *.json file into the
10+
[create anomaly detection jobs API](https://www.elastic.co/guide/en/elasticsearch/reference/8.0/ml-put-job.html) in the Kibana Dev Console. For example:
11+
12+
```
13+
PUT _ml/anomaly_detectors/abnormal_span_durations_nodejs
14+
{
15+
...
16+
}
17+
```
18+
19+
* `abnormal_span_durations_nodejs.json`: Models the duration of spans. Detects spans that are taking longer than usual to process.
20+
* `abnormal_trace_durations_nodejs.json`: Models the duration of trace transactions. Detects trace transactions that are processing slower than usual.
21+
* `decreased_throughput_nodejs.json`: Models the transaction rate of the application.
22+
Detects periods during which the application is processing fewer requests than normal.
23+
24+
25+
For more information about anomaly detection and running machine learning jobs,
26+
refer to [Finding anomalies](https://www.elastic.co/guide/en/machine-learning/8.0/ml-ad-finding-anomalies.html).

‎Machine Learning/Anomaly Detection/apm_nodejs/abnormal_span_durations_nodejs.json‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
{
2+
"groups": [
3+
"apm"
4+
],
5+
"description": "APM NodeJS: Looks for spans that are taking longer than usual to process.",
6+
"analysis_config": {
7+
"bucket_span": "15m",
8+
"detectors": [
9+
{
10+
"detector_description": "increased span duration",
11+
"function": "high_mean",
12+
"field_name": "span.duration.us",
13+
"partition_field_name": "span.type"
14+
}
15+
],
16+
"influencers": [
17+
"span.type",
18+
"trace.id",
19+
"span.name",
20+
"service.name"
21+
]
22+
},
23+
"allow_lazy_open": true,
24+
"analysis_limits": {
25+
"model_memory_limit": "128mb"
26+
},
27+
"data_description": {
28+
"time_field": "@timestamp"
29+
},
30+
"custom_settings": {
31+
"created_by": "ml-module-apm-nodejs",
32+
"custom_urls": [
33+
{
34+
"url_name": "APM",
35+
"time_range": "2h",
36+
"url_value": "apm#/traces?rangeFrom=$earliest$&rangeTo=$latest$&kuery=trace.id:\"$trace.id$\"&_g=()"
37+
}
38+
]
39+
},
40+
"datafeed_config":
41+
{
42+
"indices": [
43+
"apm-*"
44+
],
45+
"max_empty_searches": 10,
46+
"query": {
47+
"bool": {
48+
"must": [
49+
{ "bool": { "filter": { "term": { "agent.name": "nodejs" } } } },
50+
{ "bool": { "filter": { "term": { "processor.event": "span" } } } }
51+
]
52+
}
53+
}
54+
}
55+
}

0 commit comments

Comments
 (0)