Skip to content

Commit 898d08b

Browse files
fxdgearfxdgear
authored
Don't Use PW in ENV vars and default SSL on (elastic#39)
* Don't use passwords in ENV vars * upgrade docker-compose file version * upgrade elasticstack TAG * use secrets instead of PW in ENV vars * Make it easy to use secrets * Don't teach people to use passwords in ENV vars * just as easy to use a docker secret as it is to use a bind mounted volume * adding setup.yml to create 1. passwords 2. keystores 3. certs adding health checks update the readme turn off ssl for kibana endbpoint to avoid confusion for new users * .env password is needed to seed the whole process, remove old docker image -platinum * remove https from kibana, makes stack a bit easier to get started * remove zip file and allow setp.yml to work with windows * if env ELASTIC_PASSWORD not provided by user generate a pw. Also updated to echo the pw for the user when the script finishes. * undo changes to make file as they aren't needed anymore update docs on readme for how to use and get the elastic password * Make the makefile clean up more things. * remove keystores * do proper docker-compose down * remove volumes and networks make it fresh * updated readme with more windows instructions re env vars * use openssl to generate password * Fix bug that causes logstash and kibana setup to get stuck. Possible fixes: Use curl's `-u` flag for user:pass, or use hex instead of base64 for `openssl rand`. I chose `curl -u` The bug is that base64 includes `/`, so the `openssl rand -base64 16` password could contain a `/` and make curl think that the username is really the hostname: ``` curl 'https://elastic:nfAvKVigT7Bd7R60/o+1OQ==@elasticsearch:9200/' curl: (6) Could not resolve host: elastic; Unknown error ``` The bug manifests as setup_logstash and setup_kibana getting stuck waiting for Elasticsearch to be online, but never succeeds because the curl invocation is wonky. * force recreation of es keystore when running setup * update readme with ways to set env vars in powershell * more notes for windows users regarding path structure
1 parent d52aac2 commit 898d08b

27 files changed

Lines changed: 894 additions & 77 deletions

‎.gitignore‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,23 @@
11
**/*~
2+
# We wanna ignore any config/ssl/etc.. type file. As the setup container will create these for us.
3+
4+
config/elasticsearch/elasticsearch.crt
5+
config/elasticsearch/elasticsearch.key
6+
config/elasticsearch/elasticsearch.keystore
7+
config/kibana/kibana.crt
8+
config/kibana/kibana.key
9+
config/kibana/kibana.keystore
10+
config/logstash/logstash.crt
11+
config/logstash/logstash.key
12+
config/logstash/logstash.keystore
13+
config/ssl/ca/ca.crt
14+
config/ssl/ca/ca.key
15+
config/ssl/docker-cluster-ca.zip
16+
config/ssl/docker-cluster.zip
17+
18+
config/auditbeat/auditbeat.keystore
19+
config/filebeat/filebeat.keystore
20+
config/heartbeat/heartbeat.keystore
21+
config/metricbeat/metricbeat.keystore
22+
config/packetbeat/packetbeat.keystore
23+
config/apm-server/apm-server.keystore

‎Makefile‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,4 +27,8 @@ $(TARGETS:%=%-checkout):
2727
(cd stack/$(@:%-checkout=%) && git fetch && git reset --hard && git checkout origin/$(GIT_BRANCH))
2828

2929
$(TARGETS:%=%-clean):
30-
rm -rf stack/$(@:%-clean=%)
30+
rm -rf stack/$(@:%-clean=%) && find . -name "*.keystore" -exec rm -f {} \; && \
31+
docker-compose -f docker-compose.setup.yml -f docker-compose.yml down --remove-orphans && \
32+
docker-compose -f setup.yml down --remove-orphans && \
33+
docker volume rm stack-docker_es_data
34+

‎README.md‎

Lines changed: 40 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,21 +3,55 @@ This example Docker Compose configuration demonstrates many components of the
33
Elastic Stack, all running on a single machine under Docker.
44

55
## Prerequisites
6-
- Docker and Compose. Windows and Mac users get Compose installed automatically
7-
with Docker. Linux users can:
6+
- Docker and Docker Compose.
7+
* Windows and Mac users get Compose installed automatically
8+
with Docker for Windows/Mac.
9+
10+
* Linux users can read the [install instructions](https://docs.docker.com/compose/install/#install-compose) or can install via pip:
811
```
912
pip install docker-compose
1013
```
1114

12-
- At least 4GiB of RAM for the containers. Windows and Mac users _must_
15+
* Windows Users must set the following 2 ENV vars:
16+
* `COMPOSE_CONVERT_WINDOWS_PATHS=1`
17+
* `PWD=/path/to/checkout/for/stack-docker`
18+
* for example I use the path: `/c/Users/nick/elastic/stack-docker`
19+
* Note: you're paths must be in the form of `/c/path/to/place` using `C:\path\to\place` will not work
20+
* You can set these two ways:
21+
1. Temporarily add an env var in powershell use: `$Env:COMPOSE_CONVERT_WINDOWS_PATHS=1`
22+
2. Permanently add an env var in powershell use: `[Environment]::SetEnvironmentVariable("COMPOSE_CONVERT_WINDOWS_PATHS", "1", "Machine")`
23+
> Note: you will need to refresh or create a new powershell for this env var to take effect
24+
3. in System Properties add the environment variables.
25+
26+
27+
* At least 4GiB of RAM for the containers. Windows and Mac users _must_
1328
configure their Docker virtual machine to have more than the default 2 GiB of
1429
RAM:
1530

1631
![Docker VM memory settings](screenshots/docker-vm-memory-settings.png)
1732

1833
## Starting the stack
19-
Try `docker-compose up` to create a demonstration Elastic Stack with
20-
Elasticsearch, Kibana, Logstash, APM Server, Auditbeat, Metricbeat, Filebeat,
21-
Packetbeat, and Heartbeat.
34+
35+
First we need to:
36+
37+
1. set default password
38+
2. create keystores to store passwords
39+
3. install dashboards, index patterns, etc.. for beats and apm
40+
41+
This is accomplished using the setup.yml file:
42+
```
43+
docker-compose -f setup.yml up
44+
```
45+
46+
Please take note after the setup completes it will output the password
47+
that is used for the `elastic` login.
48+
49+
Now we can launch the stack with `docker-compose up -d` to create a demonstration Elastic Stack with
50+
Elasticsearch, Kibana, Logstash, Auditbeat, Metricbeat, Filebeat, Packetbeat,
51+
and Heartbeat.
2252

2353
Point a browser at [`http://localhost:5601`](http://localhost:5601) to see the results.
54+
> *NOTE*: Elasticsearch is now setup with self-signed certs.
55+
56+
Log in with `elastic` and what ever your auto generated elastic password is from the
57+
setup.

‎config/apm-server/apm-server.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apm-server.frontend.enabled: true
2+
3+
output.elasticsearch:
4+
hosts: ['elasticsearch:9200']
5+
protocol: "https"
6+
username: elastic
7+
# Read PW from apm-server.keystore
8+
password: "${ELASTIC_PASSWORD}"
9+
ssl.certificate_authorities: ["/usr/share/apm-server/certs/ca/ca.crt"]
10+
11+
setup.kibana:
12+
host: "http://kibana:5601"
13+
username: elastic
14+
password: "${ELASTIC_PASSWORD}"
15+
protocol: "http"
16+
ssl.enabled: false
17+
ssl.certificate_authorities: ["/usr/share/apm-server/certs/ca/ca.crt"]

‎config/auditbeat/auditbeat.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
auditbeat.modules:
2+
3+
- module: auditd
4+
audit_rules: |
5+
-w /etc/passwd -p wa -k identity
6+
-a always,exit -F arch=b32 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
7+
8+
- module: file_integrity
9+
paths:
10+
- /bin
11+
- /usr/bin
12+
- /sbin
13+
- /usr/sbin
14+
- /etc
15+
16+
output.elasticsearch:
17+
hosts: ['elasticsearch:9200']
18+
protocol: "https"
19+
username: elastic
20+
# Read PW from auditbeat.keystore
21+
password: "${ELASTIC_PASSWORD}"
22+
ssl.certificate_authorities: ["/usr/share/auditbeat/certs/ca/ca.crt"]
23+
24+
setup.kibana:
25+
host: "http://kibana:5601"
26+
username: elastic
27+
password: "${ELASTIC_PASSWORD}"
28+
protocol: "http"
29+
ssl.enabled: false
30+
ssl.certificate_authorities: ["/usr/share/auditbeat/certs/ca/ca.crt"]

‎config/elasticsearch/elasticsearch.p12‎

3.43 KB
Binary file not shown.

‎config/elasticsearch/elasticsearch.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
cluster.name: "docker-cluster"
2+
network.host: 0.0.0.0
3+
transport.host: 0.0.0.0
4+
5+
# minimum_master_nodes need to be explicitly set when bound on a public IP
6+
# set to 1 to allow single node clusters
7+
# Details: https://github.com/elastic/elasticsearch/pull/17288
8+
discovery.zen.minimum_master_nodes: 1
9+
xpack.license.self_generated.type: trial
10+
xpack.security.enabled: true
11+
xpack.security.http.ssl.enabled: true
12+
xpack.security.http.ssl.verification_mode: certificate
13+
xpack.security.http.ssl.key: certs/elasticsearch/elasticsearch.key
14+
xpack.security.http.ssl.certificate: certs/elasticsearch/elasticsearch.crt
15+
xpack.security.http.ssl.certificate_authorities: [ "certs/ca/ca.crt" ]
16+
17+
xpack.security.transport.ssl.enabled: true
18+
xpack.security.transport.ssl.key: certs/elasticsearch/elasticsearch.key
19+
xpack.security.transport.ssl.certificate: certs/elasticsearch/elasticsearch.crt
20+
xpack.security.transport.ssl.certificate_authorities: [ "certs/ca/ca.crt" ]

‎config/filebeat/filebeat.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
filebeat.config:
2+
prospectors:
3+
path: ${path.config}/prospectors.d/*.yml
4+
reload.enabled: false
5+
modules:
6+
path: ${path.config}/modules.d/*.yml
7+
reload.enabled: false
8+
9+
processors:
10+
- add_cloud_metadata:
11+
12+
output.elasticsearch:
13+
hosts: ['elasticsearch:9200']
14+
protocol: "https"
15+
username: elastic
16+
# Read PW from filebeat.keystore
17+
password: "${ELASTIC_PASSWORD}"
18+
ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca/ca.crt"]
19+
20+
setup.kibana:
21+
host: "http://kibana:5601"
22+
username: elastic
23+
password: "${ELASTIC_PASSWORD}"
24+
protocol: "http"
25+
ssl.enabled: false
26+
ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca/ca.crt"]

‎config/heartbeat/heartbeat.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
heartbeat.monitors:
2+
- type: http
3+
schedule: '@every 5s'
4+
urls:
5+
- https://elasticsearch:9200
6+
- http://kibana:5601
7+
ssl:
8+
certificate_authorities: ["/usr/share/heartbeat/certs/ca/ca.crt"]
9+
10+
- type: icmp
11+
schedule: '@every 5s'
12+
hosts:
13+
- elasticsearch
14+
- kibana
15+
16+
processors:
17+
- add_cloud_metadata:
18+
19+
output.elasticsearch:
20+
hosts: ['elasticsearch:9200']
21+
protocol: "https"
22+
username: elastic
23+
# Read PW from heartbeat.keystore
24+
password: "${ELASTIC_PASSWORD}"
25+
ssl.certificate_authorities: ["/usr/share/heartbeat/certs/ca/ca.crt"]
26+
27+
setup.kibana:
28+
host: "http://kibana:5601"
29+
username: elastic
30+
password: "${ELASTIC_PASSWORD}"
31+
protocol: "http"
32+
ssl.enabled: false
33+
ssl.certificate_authorities: ["/usr/share/heartbeat/certs/ca/ca.crt"]

‎config/kibana/kibana.p12‎

3.39 KB
Binary file not shown.

0 commit comments

Comments
 (0)