Also, there are no errors in the keycloak or wildfly container logs indicating a problem.

We had the same issue as well. It seems to be a proxy buffer problem in nginx, header sent by Keycloak exceed the default buffer size, which causes nginx to let the request fail with 502.

After adding the following settings to our server block in nginx config, the keycloak requests worked as before:

proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

interesting. I always use minikube as a way to test before I deploy in GKE. I guess I'll have to deploy in my staging env on GKE to see if anything needs to be done there. I'll give this a try locally by trying to patch the nginx-ingress-controller with the settings. Thanks!

How did you apply the settings.

I tried adding your settings to my existing annotation in the ingress which I added to turn on sticky sessions for vaadin.

inginx.ingress.kubernetes.io/configuration-snippet:
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

it doesn't seem to fix the 502 with 20.0.2 or 20.0.3

Thanks,
Stephen

Hi @ssill2,

we applied this configuration directly in the nginx configuration file, as in our case it was a plain local installation, no kubernetes ingress.

But for Kubernetes ingress my guess would be as described here, using Ingress annotations nginx.org/proxy-buffers and nginx.org/proxy-buffer-size or the mentioned config map keys proxy-buffers and proxy-buffer-size should do the job.

Seems like ingress does not support configuration of proxy_busy_buffer_size, I guess it could work either.

Greetings,
Marcus

@Persi

I actually tried the specific annotations first, before adding them to the snippet annotation. It didn't seem to solve the issue either. I found exactly the same thing that there was no specific annotation for the busy buffer size. I'll give it another try though. I may just have to try to upgrade our staging keycloak, which is in GCP/GKE and uses the google ingress, not the nginx-ingress that minikube does.

I'll let you know how it turns out.

Thanks for the response!,
Stephen

Hi @ssill2,

we applied this configuration directly in the nginx configuration file, as in our case it was a plain local installation, no kubernetes ingress.

But for Kubernetes ingress my guess would be as described here, using Ingress annotations nginx.org/proxy-buffers and nginx.org/proxy-buffer-size or the mentioned config map keys proxy-buffers and proxy-buffer-size should do the job.

Seems like ingress does not support configuration of proxy_busy_buffer_size, I guess it could work either.

Greetings, Marcus

So I tried the following annotation configuration in my development ingress in minikube. The behavior didn't change.

  annotations:
    kubernetes.io/ingress.allow-http: "true"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.org/proxy-buffers: "4 256k"
    nginx.org/proxy-buffer-size: "128k"
    inginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Forwarded-For $proxy_protocol_addr;
      proxy_set_header X-Forwarded-Proto  $scheme;
      proxy_set_header Host $host;
      proxy_busy_buffers_size 256k;

I was really hopeful that your settings would just fix the problem lol.

The problem with everything being in kubernetes is I don't have the tools like tcpdump to look at what's actually happening in the http conversations between my app's pod and the keycloak pod. But I find it fascinating that .1->.2 made all the difference though.

Ah probably you have to use the standard kubernetes Ingress annotations, so instead of nginx.org/proxy-buffer-size try to use nginx.ingress.kubernetes.io/proxy-buffer-size and instead of nginx.org/proxy-buffers try nginx.ingress.kubernetes.io/proxy-buffers-number.

ah cool, I'll give that a try here in a bit :) Thanks!

hey! I can confirm this fixed it!

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dev-apps-ingress
  namespace: {{ .Values.namespace }}
  annotations:
    kubernetes.io/ingress.allow-http: "true"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/proxy-buffers: "4 256k"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
    inginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Forwarded-For $proxy_protocol_addr;
      proxy_set_header X-Forwarded-Proto  $scheme;
      proxy_set_header Host $host;
      proxy_busy_buffers_size 256k;

changing those annotations to be nginx.ingress.kubernetes.io/* made them take effect. the 502 problem went away!

Thanks Marcus! 😄