add automated log transfer, AC-Hunter issue 135#62
Merged
lisaSW merged 10 commits intoconfig_updatefrom Jul 9, 2025
Merged
Conversation
Co-Authored-By: William Stearns <[email protected]>
Contributor
|
FYI I'm going to commit a light linting/cleaning pass over the YAML files into this PR branch to make local review/testing process easier. |
0x6d6f7468
requested changes
Jul 8, 2025
Contributor
0x6d6f7468
left a comment
0x6d6f7468
left a comment
There was a problem hiding this comment.
Testing this on fresh Digital Ocean droplets (using the latest pushed commit, before any linting/cleanup), I'm getting this error:
TASK [Zeek Install: Transfer zeek_log_transport.sh shell script to target system.] ***********************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the remote, see the remote_src option
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "Could not find or access './opt/zeek_log_transport.sh'\nSearched in:\n\t/home/acm/rita-v5.0.8-installer/files/./opt/zeek_log_transport.sh\n\t/home/acm/rita-v5.0.8-installer/./opt/zeek_log_transport.sh\n\t/home/acm/rita-v5.0.8-installer/files/./opt/zeek_log_transport.sh\n\t/home/acm/rita-v5.0.8-installer/./opt/zeek_log_transport.sh on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
127.0.0.1 : ok=23 changed=1 unreachable=0 failed=1 skipped=21 rescued=0 ignored=0
Testing methodology (behavior recreated on all three supported distros):
- Create fresh droplets
- Upload repo contents to droplet
- Run
./generate_installer.shon droplet - Move (or upload) and decompress created installer archive
- Run installer with
./install_rita.sh 127.0.0.1
It seems like the installer is referencing a script named zeek_log_transport.sh, but that script isn't retrieved by RITA's installer. I will investigate further.
Contributor
|
Ok on further inspection, I think we can add script retrieval logic to the installer when That said, I'm also wondering if we want to flip this and have zeek installation turned off by default, but that might be a discussion for another time. |
Contributor
|
Doing a final round of testing, then I think it'll be good to go. |
Download zeek_log_transport.sh to send to the sensor.
Contributor
|
Made the following changes:
Worth noting, a condition of Tested on all permutations of Ubuntu, CentOS, and Rocky (both local and remote installations). Testing included #58's changes just to get CentOS/Rocky -> Ubuntu installations to succeed. |
lisaSW
approved these changes
Jul 9, 2025
caffeinatedpixel
added a commit
that referenced
this pull request
Sep 22, 2025
* Update on config structure, functionality, and tests Co-Authored-By: Naomi Kramer <[email protected]> * Extend subnet type to read/write from db, update tests Co-Authored-By: Liza Tsibur <[email protected]> * updated read file config test and subnet tests * fixed config and util tests, updated subnet related functions Co-Authored-By: Naomi Kramer <[email protected]> * Remove error return from GetDefaultConfig Co-Authored-By: Liza Tsibur <[email protected]> * added json tags to database struct * Updated beacon weights validation for config * updates to score thresholds validation tags * changes to config subnet validation and testing * Update subnet.go * Write missing host entries to http to populate http_proto * Updating some fields to uint64 * WIP update some field types * Update zeek count types and fix tests * Add clickhouse credentials * Misc fixes * Update pointer * Add ability to mark datasets as sample datasets * fix column name * Fix datasets exiting import if hour is empty * Fix zeek count parsing from TSV files * Remove storing dns conns in arrays, Fix historical first seen dns lag * Remove unused columns * Update config.hjson * Update config.hjson * updated impact category score functions to use float64 Co-Authored-By: Naomi Kramer <[email protected]> * Update subnet.go * Store import version in imports table * Fix duplicated SNI/IP long connections * Update subnet_test.go * Cleanup output * Rolling files updates (#39) * Limit number of days to import for rolling datasets * Fix breaking imports when import was interrupted * Remove debug output --------- Co-authored-by: Naomi Kramer <[email protected]> * Omit parts of env from output * Set max for threat intel datasize * Remove SELinux neutering for QA * Add network size column * Fix http_proto for missing host, update tests for missing host fixes * Add online feeds to default config * Update sshprep (#45) * Update sshprep Co-Authored-By: William Stearns <[email protected]> * Update sshprep Add Bradley's suggestion of using head -1 to limit to a single address. --------- Co-authored-by: Naomi Kramer <[email protected]> Co-authored-by: William Stearns <[email protected]> * Installer Behavior Tweaks (#41) * Add --yes flag to add-apt-repository command * Add missing sudo flags, make sure we're using the SUDO variable instead * Add ability to perform zone transfers (#48) * Store zone transfer records Co-Authored-By: moth <[email protected]> * Update config * Add tests * Tests, connectivity test * Update tests --------- Co-authored-by: moth <[email protected]> * Support RedHat/RHEL as a valid target (#47) * Update sshprep Co-Authored-By: William Stearns <[email protected]> * Supporrt RedHat/RHEL as a valid target --------- Co-authored-by: Naomi Kramer <[email protected]> Co-authored-by: William Stearns <[email protected]> Co-authored-by: moth <[email protected]> * Fix tests (#49) * Fix tests * Update WalkFiles to use UTC * fixed issue with rolling datasets over 24hours old not getting historical first seen timestamp set (#52) * Change values from float32 to float64 (#50) * Switch float32 to float64 * Update threat category calculation to match CalculateBucketedScore (#51) --------- Co-authored-by: Liza Tsibur <[email protected]> * Bump max query execution time default value * Use string instead of error for ZoneTransferConnectivityErrors struct fields (#61) * Upgrade Golang to version 1.24 (#59) (#60) * Replace get_url with shell and curl (#58) * Update sshprep Co-Authored-By: William Stearns <[email protected]> * Replace get_url with shell and curl * Use get_url by default, fall back to curl if it fails --------- Co-authored-by: Naomi Kramer <[email protected]> Co-authored-by: William Stearns <[email protected]> Co-authored-by: moth <[email protected]> * add automated log transfer, AC-Hunter issue 135 (#62) * Update sshprep Co-Authored-By: William Stearns <[email protected]> * add automated log transfer, PR135 * cron requires non-executable permission * Specify suggested YAML plugin and config in VSCode workspace * Linting and light cleanup * Update generate_installer.sh Download zeek_log_transport.sh to send to the sensor. * Create cron file if remote zeek installation * Only run zeek log import steps for remote sensor installations --------- Co-authored-by: Naomi Kramer <[email protected]> Co-authored-by: William Stearns <[email protected]> Co-authored-by: moth <[email protected]> * Temporarily disable RITA/Zeek log transport until installer is modular (#66) * Uniform -y flag usage for repo management/package installation; Uniform SUDO variable usage (#68) * Resolve Installer Side Effects and Formalize RHEL Support (#73) * Add missing necessary wildcards for RHEL versions * Remove Ansible task replacing python3-requests to avoid RHEL distro installation side effects * Update supported distros in README * Update scoring defaults * Resolve Ansible Reboot Errors (#75) * Clean up conditionals; Fix reboot step for Ubuntu * Suppress erroneous error output on RPM systems, ignore errors on reboot necessity checks * Ignore missing host rows for openhttp (#76) * Fix integration tests due to prevalence (#77) --------- Co-authored-by: Liza Tsibur <[email protected]> Co-authored-by: moth <[email protected]> Co-authored-by: William Stearns <[email protected]> Co-authored-by: William Stearns <[email protected]> Co-authored-by: moth <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is one half of the resolution of https://github.com/activecm/AC-Hunter/issues/135 . The other PR will be in the AC-Hunter develop branch.