I'll see what Duo can do 👌, but let us get one function before the other 👌

These are just to the chart changes. I am fairly certain there's some expansion required to the CRD handler, for the Runway definitions.

@skundapur The problem iwth SubPaths, is that any content updates do not populate when there are changes made, without a full restart of the pod. You can account for that, but there are concerns.

A container using a ConfigMap as a subPath volume mount will not receive updates when the ConfigMap changes.

Are you desring to populate base.yaml in the container itself, and then populate further configuration into the same directory where this base configuration is stored? If this is the case, you may want to consider allowing for a config.d type of behavior, where the application can read from several directories, rather than a single explicit location.

├── base.yaml
└── config.d/
    ├── staging.yaml
    └── pre.yaml

There are a few ways around this, but SubPath has known and distinct drawbacks.

Jason Plum (a93adbe6) at 17 Mar 13:03

... and 2 more commits

@GitLabDuo Sensical. I'll tell the editor to do that on the next pass.

Jason Plum (226ae005) at 17 Mar 09:28

... and 1 more commit

Jason Plum (8ae101fd) at 17 Mar 08:03

... and 2 more commits

What

feat(projected-volumes): Support render of projectedVolume in Deployment

Support the rendering of a projectedVolume in k8s Deployment.

Notice: This intentionally renders only Secret and ConfigMap items. This is in order to limit exposure of sensitive materials.

• Added unittest, and included into Makefile.

Why

Support mounting multiple complex projected volumes, for applications that require multiple specific keys mounted from different sources, into a given location.

cc @mkaeppler @skundapur regarding Slack discussion.

Jason Plum (6a5326b4) at 17 Mar 00:13

Jason Plum (fba8a4de) at 16 Mar 21:35

Jason Plum (4aaf5eb7) at 16 Mar 21:23

... and 3 more commits

I have not had a chance to test CNG with your dns+tls:// scheme. Perhaps since this might be the preferred way going forward is there someone on the reference architecture team that can validate this?

@stanhu That would be a standard Cloud Native Hybrid RA + TLS @ Gitaly, with any Consul involved for resolution, if we wanted to match how this seems to be expected to behave at scale.

That said, you could also try this a supplemented item: a Service object of type: ExternalName (docs). This would then have Gitaly ask for external-gitaly.namespace.svc and resolve to gitaly.somewhere.else, matching the TLS/SSL name. In theory:

global:
  gitaly:
    enabled: false
    external:
    - name: default
      hostname: gitaly.somewhere.else
      address: dns:external-gitaly
      tlsEnabled: true

With this merged, please open the backport MRs as appropriate. Keep an eye out for any supplemental change that need to be pulled backwards as well.

@vespian_gl @suleimiahmed Confirm: Is this meant for %18.10 or %18.11 ?

This MR contains the following updates:

Notes for Reviewer

• For security reasons, our Renovate fork cannot trigger chart pipelines
  • ⚠️ Reviewer needs to initiate new MR pipeline, subsequently triggering chart pipeline from it. ⚠️

Release Notes

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻️ Rebasing: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this MR and you won't be reminded about this update again.

• If you want to rebase/retry this MR, check this box

This MR has been generated by Renovate Bot.

Jason Plum (66d636e7) at 16 Mar 19:45

Jason Plum (9ea0c343) at 16 Mar 19:45

... and 1 more commit

What does this MR do?

Migrate check-packages pipeline to download packages from Pulp

Change the Omnibus package checking pipeline to use pulp.gitlab.com instead of packages.gitlab.com.

• Add configurable package_server to install_package.sh
• Pass Pulp basic auth credentials (username/password) to install scripts
• Fetch PULP_PASSWORD_RO and PULP_USER_RO from https://dev.gitlab.org/gitlab/omnibus-gitlab/-/settings/ci_cd#js-cicd-variables-settings
• Set check-packages-functionality to use pre-release repo on Pulp

Ref: gitlab-org/build/team-tasks#162

Testing

Start a Docker container using the same image used by Omnibus, e.g. https://dev.gitlab.org/gitlab/omnibus-gitlab/-/jobs/34303624/viewer

docker run -it --rm registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/ubuntu_20.04_arm64:5.50.0 bash

Clone and checkout the branch

apt update && apt install git vim -y
git clone https://gitlab.com/gitlab-org/omnibus-gitlab.git && cd omnibus-gitlab && git checkout pre-release-check-pulp 
bash scripts/ci/prepare_bundle.sh

In install_package.sh , manually change package_name_version_dist='gitlab-ee=18.9.0-ee.0' . On the pipeline, it is set by the commit tag.

Run

PULP_URL=pulp.gitlab.com PULP_USER=omnibus-downloader PULP_PASSWORD='password' package_type=deb package_manager=apt package_repository=pre-release bash scripts/ci/install_package.sh

All works well. GitLab is installed. We can see that pulp.gitlab.com is used:

Installing gitlab-ce using https://pulp.gitlab.com/install/repositories/gitlab/pre-release/script.deb.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    10  100    10    0     0     36      0 --:--:-- --:--:-- --:--:--    36
100  5287  100  5287    0     0   4569      0  0:00:01  0:00:01 --:--:--  4569
Hit:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease
Hit:5 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
Hit:4 https://pulp.gitlab.com/gitlab/pre-release/ubuntu/focal focal InRelease
Reading package lists... Done
N: Usage of apt_auth.conf(5) should be preferred over embedding login information directly in the sources.list(5) entry for 'https://pulp.gitlab.com/gitlab/pre-release/ubuntu/focal'
Repository configured successfully.
Ready to install packages.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  gitlab-ee
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 1463 MB of archives.
After this operation, 4581 MB of additional disk space will be used.
Get:1 https://pulp.gitlab.com/gitlab/pre-release/ubuntu/focal focal/main arm64 gitlab-ee arm64 18.9.0-ee.0 [1463 MB]
Fetched 1463 MB in 1min 40s (14.6 MB/s)                                                                                                                      
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package gitlab-ee.
(Reading database ... 23046 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_18.9.0-ee.0_arm64.deb ...
Unpacking gitlab-ee (18.9.0-ee.0) ...

Progress: [ 20%] [###########################.............................................................................................................] 

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes.
• Documentation created/updated.
• Tests added.
• Integration tests added to GitLab QA.
• Equivalent MR/issue for the GitLab Chart opened.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.

What does this MR do?

Migrate check-packages pipeline to download packages from Pulp

Change the Omnibus package checking pipeline to use pulp.gitlab.com instead of packages.gitlab.com.

• Add configurable package_server to install_package.sh
• Pass Pulp basic auth credentials (username/password) to install scripts
• Fetch PULP_PASSWORD_RO and PULP_USER_RO from https://dev.gitlab.org/gitlab/omnibus-gitlab/-/settings/ci_cd#js-cicd-variables-settings
• Set check-packages-functionality to use pre-release repo on Pulp

Ref: gitlab-org/build/team-tasks#162

Testing

Start a Docker container using the same image used by Omnibus, e.g. https://dev.gitlab.org/gitlab/omnibus-gitlab/-/jobs/34303624/viewer

docker run -it --rm registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/ubuntu_20.04_arm64:5.50.0 bash

Clone and checkout the branch

apt update && apt install git vim -y
git clone https://gitlab.com/gitlab-org/omnibus-gitlab.git && cd omnibus-gitlab && git checkout pre-release-check-pulp 
bash scripts/ci/prepare_bundle.sh

In install_package.sh , manually change package_name_version_dist='gitlab-ee=18.9.0-ee.0' . On the pipeline, it is set by the commit tag.

Run

PULP_URL=pulp.gitlab.com PULP_USER=omnibus-downloader PULP_PASSWORD='password' package_type=deb package_manager=apt package_repository=pre-release bash scripts/ci/install_package.sh

All works well. GitLab is installed. We can see that pulp.gitlab.com is used:

Installing gitlab-ce using https://pulp.gitlab.com/install/repositories/gitlab/pre-release/script.deb.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    10  100    10    0     0     36      0 --:--:-- --:--:-- --:--:--    36
100  5287  100  5287    0     0   4569      0  0:00:01  0:00:01 --:--:--  4569
Hit:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease
Hit:5 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
Hit:4 https://pulp.gitlab.com/gitlab/pre-release/ubuntu/focal focal InRelease
Reading package lists... Done
N: Usage of apt_auth.conf(5) should be preferred over embedding login information directly in the sources.list(5) entry for 'https://pulp.gitlab.com/gitlab/pre-release/ubuntu/focal'
Repository configured successfully.
Ready to install packages.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  gitlab-ee
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 1463 MB of archives.
After this operation, 4581 MB of additional disk space will be used.
Get:1 https://pulp.gitlab.com/gitlab/pre-release/ubuntu/focal focal/main arm64 gitlab-ee arm64 18.9.0-ee.0 [1463 MB]
Fetched 1463 MB in 1min 40s (14.6 MB/s)                                                                                                                      
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package gitlab-ee.
(Reading database ... 23046 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_18.9.0-ee.0_arm64.deb ...
Unpacking gitlab-ee (18.9.0-ee.0) ...

Progress: [ 20%] [###########################.............................................................................................................] 

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• MR title and description are up to date, accurate, and descriptive.
• MR targeting the appropriate branch.
• Latest Merge Result pipeline is green.
• When ready for review, MR is labeled workflowready for review per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

• The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
• If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
• If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
• If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes.
• Documentation created/updated.
• Tests added.
• Integration tests added to GitLab QA.
• Equivalent MR/issue for the GitLab Chart opened.
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.