done

Michael Kazakov (9eacb142) at 19 Mar 08:42

... and 1 more commit

I have 3 options

• Remove к8s_proxy_read_timeout, k8s_proxy_connect_timeout from nginx-kas config. In this case the global configuration will be used
• Define defaults as constants. The first variant I did.
• Define defaults dynamically using global values

I do not know which I have to choose

Michael Kazakov (f4f08d90) at 17 Mar 16:28

@victorprete

It is brake tests

NoMethodError: undefined method `command' for Chef::Resource::RunitService

Michael Kazakov (88a1b3e4) at 17 Mar 11:45

Michael Kazakov (35ef37b3) at 17 Mar 08:11

... and 1 more commit

ok, will do the change tomorrow

The code is not written by me. It is suggested by @mkozono. See #9235 (comment 3077578194)

@luciezhao Hi, Lucie! MR already done !9133

we can remove them or define using the defaults in this case you can have them differ. But I can't imagine reason to do so.

I took the values from the sources

It may be related to gitlab-org/omnibus-gitlab#9638

@ash2k

In my infrastructure I use consul service which will go down automatically after fix gitlab-org/cluster-integration/gitlab-agent#813

@stanhu

Can You review the MR?

Current omnibus nginx service is configured to serve external (agent requests). While rails requests are going directly to KAS api port. All works file when KAS and rails are on the same host. When we move KAS on separated host we have to route rails requests to that host. Also we need add TLS to the connection to male it secure. This setup can be configured using gilab.rb.

gitlab.rb on rais/sidekiq

gitlab_rails['gitlab_kas_internal_url'] = 'grpcs://kas-host:8153'

gitlab.rb on kas node

gitlab_kas['internal_api_listen_address'] = '0.0.0.0:8153'
gitlab_kas['internal_api_certificate_file'] = '<path to SSL certificate file>'
gitlab_kas['internal_api_key_file'] = '<path to SSL certificate key file>'

So, we can leave omnibus as it is. Just put this block in documentation

@ash2k

After deep look into omnibus code I fund that the best way to add proxy for KAS API is create a new nginx service. Also I found the way to expose API port with TLS configured and connect rails directly to it using gitlab.rb. I've tested the config and it works fine.

So, question is. Shall we need proxy between Rails and KAS API?

Working workaround

gitlab.rb on rais/sidekiq

gitlab_rails['gitlab_kas_internal_url'] = 'grpcs://kas-host:8153'

gitlab.rb on kas node

gitlab_kas['internal_api_listen_address'] = '0.0.0.0:8153'
gitlab_kas['internal_api_certificate_file'] = '<path to SSL certificate file>'
gitlab_kas['internal_api_key_file'] = '<path to SSL certificate key file>'

@ash2k Implementation plan:

• Duplicate nginx-gitlab-kas.conf.erb into a separate file. Suggest nginx-gitlab-kas-external.conf.erb (port 443) and nginx-gitlab-kas-internal.conf.erb (port 8153)
• Introduce an opt-in flag to enable it.
• There should be a way to specify IPs of all kas instances in the config. They can be grouped with upstream. Let's call it gitlab_kas_internal_upstream.
• We only need a single location that sends everything to 8153, similar to the existing /gitlab.agent - grpc_pass <%= node['gitlab_kas']['upstream_grpc_scheme'] %>://gitlab_kas_internal_upstream;
• Add some tests.
• Add documentation to https://docs.gitlab.com/administration/clusters/kas/#use-a-load-balancer-or-reverse-proxy-with-multiple-kas-instances.

Based on gitlab-org/cluster-integration/gitlab-agent#810 (comment 3094216212) Locations will be:

• /gitlab.agent. -> port 8153
• /gitlab.agent.(agent_configuration|agent_registrar). -> port 8150