implement KubernetesExecutor by adwk67 · Pull Request #311 · stackabletech/airflow-operator

adwk67 · 2023-08-01T16:14:52Z

Description

Implements KubernetesExecutor.
Fixes #2.
Jenkins: 🟢 https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/99/

Implementation summary:

the executor field has been removed from the top-level cluster config
the workers role has been replaced with a complex enum specifying one of:
- celery: this contains the worker config/roles etc.
- kubernetes: this contains just the logging/resources of the config, plus env-overrides (has to be included as we don't have a role object)
when executor = kubernetes, a pod template is deployed as config map that airflow will use to spawn pods for the DAG tasks

Tests on Openshift:

--- PASS: kuttl (3580.00s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/logging_airflow-2.4.1_openshift-true (282.85s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-insecure-tls_openshift-true_executor-celery (321.02s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-2.6.1_openshift-true_executor-kubernetes (215.58s)
        --- PASS: kuttl/harness/logging_airflow-2.2.5_openshift-true (305.99s)
        --- PASS: kuttl/harness/logging_airflow-2.2.4_openshift-true (328.94s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-2.6.1_openshift-true_executor-kubernetes (202.57s)
        --- PASS: kuttl/harness/mount-dags-configmap_airflow-latest-2.6.1_openshift-true_executor-celery (290.25s)
        --- PASS: kuttl/harness/orphaned-resources_airflow-latest-2.6.1_openshift-true (318.29s)
        --- PASS: kuttl/harness/smoke_airflow-2.2.5_openshift-true_executor-kubernetes (292.73s)
        --- PASS: kuttl/harness/smoke_airflow-2.4.1_openshift-true_executor-kubernetes (267.12s)
        --- PASS: kuttl/harness/smoke_airflow-2.6.1_openshift-true_executor-celery (284.75s)
        --- PASS: kuttl/harness/smoke_airflow-2.4.1_openshift-true_executor-celery (335.22s)
        --- PASS: kuttl/harness/smoke_airflow-2.6.1_openshift-true_executor-kubernetes (226.21s)
        --- PASS: kuttl/harness/cluster-operation_airflow-latest-2.6.1_openshift-true (303.22s)
        --- PASS: kuttl/harness/resources_airflow-latest-2.6.1_openshift-true (209.99s)
        --- PASS: kuttl/harness/smoke_airflow-2.2.4_openshift-true_executor-celery (343.50s)
        --- PASS: kuttl/harness/smoke_airflow-2.2.5_openshift-true_executor-celery (395.81s)
        --- PASS: kuttl/harness/smoke_airflow-2.2.4_openshift-true_executor-kubernetes (314.30s)
        --- PASS: kuttl/harness/logging_airflow-2.6.1_openshift-true (239.35s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-server-verification-tls_openshift-true_executor-celery (311.55s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-no-tls_openshift-true_executor-kubernetes (202.59s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-no-tls_openshift-true_executor-celery (296.19s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-insecure-tls_openshift-true_executor-kubernetes (246.54s)
        --- PASS: kuttl/harness/mount-dags-gitsync_airflow-latest-2.6.1_openshift-true_executor-celery (251.68s)
        --- PASS: kuttl/harness/ldap_airflow-latest-2.6.1_ldap-authentication-server-verification-tls_openshift-true_executor-kubernetes (200.25s)

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

# Author
- [x] Changes are OpenShift compatible
- [x] Helm chart can be installed and deployed operator works
- [x] Integration tests passed (add new ones for the executor)
- [x] Documentation
- [x] Logging works for ephemeral worker pods (not working: covered by https://github.com/stackabletech/airflow-operator/issues/318)
- [x] Check tini invocation
- [x] CRD changes are approved
- [x] Ensure vector container is shutdown on the worker pods used by the kubernetes executor. See https://github.com/stackabletech/docker-images/pull/435

# Reviewer
- [x] Code contains useful comments
- [x] (Integration-)Test cases added
- [x] Documentation added or updated
- [x] Changelog updated
- [x] Cargo.toml only contains references to git tags (not specific commits or branches)

# Acceptance
- [ ] Feature Tracker has been updated
- [ ] Proper release label has been added

rust/operator-binary/src/airflow_controller.rs

adwk67 · 2023-08-18T10:42:37Z

Jenkins after refactoring CRD structure: 🟢 https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/100/

razvan

It works and it's ok but IMO:

ExecutorConfig vs AirflowConfig - the names are confusing don't reflect the intent.
Not sure if it's wise to introduce so much complication just because of the affinity issue of executors. But fortunately it's an implementation detail that we can hopefully change later without breaking the CRD.

tests/templates/kuttl/mount-dags-gitsync/03-assert.yaml.j2

rust/operator-binary/src/airflow_controller.rs

docs/modules/airflow/pages/index.adoc

docs/modules/airflow/pages/getting_started/first_steps.adoc

razvan · 2023-08-22T15:48:03Z

This is almost adding support for complete new product. ~~It may be worth adding an explicit integration test for kubernetes workers and actually run a some tasks with it.~~ This is already done.

Co-authored-by: Razvan-Daniel Mihai <[email protected]>

rust/operator-binary/src/airflow_controller.rs

Co-authored-by: Razvan-Daniel Mihai <[email protected]>

adwk67 added 4 commits July 28, 2023 17:59

wip: using template mounted as config map

9762dfa

wip: build config map

0172890

wip: initial working configmap

3ffbb2f

wip: cleanup

94065da

github-actions bot reviewed Aug 1, 2023

View reviewed changes

rust/operator-binary/src/airflow_controller.rs Show resolved Hide resolved

adwk67 linked an issue Aug 1, 2023 that may be closed by this pull request

Extend Airflow operator by implementing KubernetesExecutor #2

Closed

adwk67 added 24 commits August 1, 2023 18:21

wip: clippy allow

6f8b14e

rebuild charts

4f91708

merged main

e0b9730

fixed merged conflict

4321e77

fixed merged conflict

4623cbd

added provisional comment re. worker pod usage

0cd4824

updated changelog

3e3a304

pin serde/serde_derive at 1.0.171

0fa8dec

fixed typo

d6b448b

reverted executor change

cfa37a2

added namespace for k8s executors

14bec71

add alternative env-var for older versions

f89eca9

simplified config map

27814b9

wip: replace role with enum

21d71ff

flatten celery executor config

965b255

wip: adapting tests

6d01971

wip: added templating

e2105ee

working smoke/celery test

b6e2efb

working smoke/kubernetes test

41966b8

linting

6ed22bf

fix: added pre-2.5 name of env-var

1f4800f

added env overrides for config map test

ecfd1cb

added gitysnc elements to pod template and adapted tests

c34c244

adapted ldap tests

a892133

adwk67 marked this pull request as ready for review August 14, 2023 09:52

adwk67 requested a review from a team August 14, 2023 09:53

adwk67 added 11 commits August 14, 2023 13:26

move constants

b628846

specified the resources for workers

cc7f695

make executor non-optional, remove enum-discriminant usage

7ed321a

regenerate charts

70cd55a

added note on image name and shutdown hook

ca331ab

added shell capture for statefulsets

0d2613e

wip:crd refactoring

e08a07c

fixed tests, rebuilt charts

23983db

rebuilt charts, adapted integration tests

d3ab0e5

further integration test fixes

d612a59

updated docs

aa88c1d

corrected docs

b35966a

adwk67 requested review from razvan and sbernauer August 18, 2023 14:23

razvan reviewed Aug 22, 2023

View reviewed changes

adwk67 and others added 2 commits August 23, 2023 09:45

Update rust/operator-binary/src/airflow_controller.rs

eaec70b

Co-authored-by: Razvan-Daniel Mihai <[email protected]>

Update docs/modules/airflow/pages/index.adoc

86d9493

Co-authored-by: Razvan-Daniel Mihai <[email protected]>

github-actions bot reviewed Aug 23, 2023

View reviewed changes

rust/operator-binary/src/airflow_controller.rs Show resolved Hide resolved

adwk67 and others added 2 commits August 23, 2023 09:49

Update docs/modules/airflow/pages/getting_started/first_steps.adoc

8412573

Co-authored-by: Razvan-Daniel Mihai <[email protected]>

input from review

c1f43d9

razvan approved these changes Aug 23, 2023

View reviewed changes

adwk67 added this pull request to the merge queue Aug 23, 2023

Merged via the queue into main with commit 8164085 Aug 23, 2023

adwk67 deleted the feat/kubernetes-executor branch August 23, 2023 11:02

siegfriedweber mentioned this pull request Jul 18, 2024

Logs from the airflow container are logged twice #473

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

implement KubernetesExecutor#311

implement KubernetesExecutor#311
adwk67 merged 54 commits intomainfrom
feat/kubernetes-executor

adwk67 commented Aug 1, 2023 •

edited by razvan

Loading

Uh oh!

Uh oh!

adwk67 commented Aug 18, 2023

Uh oh!

razvan left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

razvan commented Aug 22, 2023 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

adwk67 commented Aug 1, 2023 • edited by razvan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Uh oh!

Uh oh!

adwk67 commented Aug 18, 2023

Uh oh!

razvan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

razvan commented Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adwk67 commented Aug 1, 2023 •

edited by razvan

Loading

razvan commented Aug 22, 2023 •

edited

Loading