We need two types of guard rails.
- Global memory guard rails.
- File level data guard rails
File level guard rails
If a file has too many lines we stop at an upper limit of entries.
See store the skipped/cut files in a shared location
We have the following properties:
- PROP_LOGPARSER_FILEENTRY_LIMIT
- PROP_LOGPARSER_FILESIZE_LIMIT
- PROP_LOGPARSER_HEAP_LIMIT
- PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE
- PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT
| Property |
Description |
Affects |
Scale |
Default Value |
| PROP_LOGPARSER_FILEENTRY_LIMIT |
Maximum number of entries to parse per file |
File parsing |
Count |
-1 (disabled) |
| PROP_LOGPARSER_FILESIZE_LIMIT |
Maximum file size in MB to parse |
File parsing |
Megabytes |
-1 (disabled) |
| PROP_LOGPARSER_HEAP_LIMIT |
Maximum heap size increase in MB before warning |
File parsing, FilterBy, Search, enrich, groupBy |
Megabytes |
-1 (disabled) |
| PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE |
Maximum percentage of memory usage before warning |
File parsing, FilterBy, Search, enrich, groupBy |
Percentage |
-1 (disabled) |
| PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT |
Whether to throw exception when memory limits exceeded |
Error handling |
Boolean |
false |
We need two types of guard rails.
File level guard rails
If a file has too many lines we stop at an upper limit of entries.
See store the skipped/cut files in a shared location
We have the following properties: