Changing nginx.ingress.kubernetes.io/whitelist-source-range error code

[DerekTBrown]

Context

• ingress-nginx supports an annotation, nginx.ingress.kubernetes.io/whitelist-source-range (docs), which allows the operator to require requests to come from a specific set of IP addresses.
• If a request comes from outside this range, ingress-nginx returns a 403 status code (ref).
  • Under the hood, this is implemented via the ngx_http_access_module (ref).

Why is this problematic?

• A 403 error code communicates that the client lacks permissions for a "particular resource, or action" (ref). However, clients will continue to operate under the assumption that requests for a different resource or action may succeed.
• This may lead to unexpected behavior:
  1. Clients might continually retry their requests, mistakenly believing that with additional credentials or different parameters they might eventually gain access, even though the underlying IP block will persist.
     2. Clients may mishandle/misreport these error messages. For example, a multi-tenant service may treat 4xx errors as being caused by its clients, rather than a misconfiguration of the service itself.
     3. Operators may have a difficult time investigating such issues, since it is ambiguous whether the error is caused by the proxy layer or the service itself.

What is the proper status code?

I believe the correct behavior is for nginx to return a 511 code, which is tailored for this case:

The HTTP 511 Network Authentication Required server error response status code indicates that the client needs to authenticate to gain network access. This status is not generated by origin servers, but by intercepting proxies that control access to a network.

Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/511

How could this fix be applied?

Changing from 403 -> 511 is doubtless a breaking change for clients. Thus, I propose allowing this to be user-configurable, defaulting to existing behavior:

• Step 1: Extend ngx_http_access_module to support custom code:

deny all;              # defaults to 403 Forbidden
deny 192.168.1.0/24 code=511;  # returns 511 Unauthorized for this range

• Step 2: Make this option configurable within ingress-nginx:

annotations:
       nginx.ingress.kubernetes.io/whitelist-source-range-deny-code: 511

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

/remove-kind bug
/kind feature

[Gacko]

The ngx_http_access_module is maintained by NGINX project, not here. We only use NGINX as the underlying webserver and do not change its functionality. I'd recommend discussing this with their community first.

[DerekTBrown]

Apparently this is supported on their end, just not in ingress-nginx:

nginx/nginx#610 (comment)

[DerekTBrown]

@Gacko can you re-open?