tower_http::compression::CompressionLayer broken in lambda function URL

[fluxth]

Summary

After the update to lambda_http v0.11.2, some response headers no longer propagate in lambda function URL environment. See below for how to reproduce.

Maybe related PR #852

How to reproduce

Using the following code, deploy to a lambda function URL and access it:

use axum::{response::Json, routing::get, Router};
use lambda_http::{run, tracing, Error};
use serde_json::{json, Value};
use std::env::set_var;
use tower_http::compression::CompressionLayer;

async fn root() -> Json<Value> {
    Json(json!(
        {
            "msg": "headers are broken!",
            "data": [
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer",
                "long data to trigger compression layer"
            ]
        }
    ))
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    set_var("AWS_LAMBDA_HTTP_IGNORE_STAGE_IN_PATH", "true");

    tracing::init_default_subscriber();

    let compression_layer: CompressionLayer = CompressionLayer::new()
        .br(true)
        .deflate(true)
        .gzip(true)
        .zstd(true);

    let app = Router::new().route("/", get(root)).layer(compression_layer);

    run(app).await
}

Behavior in lambda_http v0.11.1

$ curl -v -s --compressed https://i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws/
*   Trying [2406:da18:8b4:9a02:f373:cbfa:ca80:c9e9]:443...
* Connected to i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws (2406:da18:8b4:9a02:f373:cbfa:ca80:c9e9) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=*.lambda-url.ap-southeast-1.on.aws
*  start date: Dec 27 00:00:00 2023 GMT
*  expire date: Jan 25 23:59:59 2025 GMT
*  subjectAltName: host "i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws" matched cert's "*.lambda-url.ap-southeast-1.on.aws"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M03
*  SSL certificate verify ok.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws
> User-Agent: curl/8.4.0
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
< Date: Wed, 08 May 2024 15:17:13 GMT
< Content-Type: application/json
< Content-Length: 98
< Connection: keep-alive
< x-amzn-RequestId: 478464ff-1e97-45e0-8ce0-5ff3dadf6073
< content-encoding: gzip
< vary: accept-encoding
< X-Amzn-Trace-Id: root=1-663b9779-61725605046a4fb97132244a;parent=75fe143b03d7642a;sampled=0;lineage=b073041a:0
<
* Connection #0 to host i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws left intact
{"data":["long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer"],"msg":"headers are broken!"}%

Note the content-encoding: gzip and vary: accept-encoding headers in the response above.

After updated to lambda_http v0.11.2

$ curl -v -s --compressed https://i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws/ --output - | xxd
*   Trying [2406:da18:8b4:9a03:5ec7:9962:6e3e:b3df]:443...
* Connected to i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws (2406:da18:8b4:9a03:5ec7:9962:6e3e:b3df) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [370 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [10 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [4991 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=*.lambda-url.ap-southeast-1.on.aws
*  start date: Dec 27 00:00:00 2023 GMT
*  expire date: Jan 25 23:59:59 2025 GMT
*  subjectAltName: host "i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws" matched cert's "*.lambda-url.ap-southeast-1.on.aws"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M03
*  SSL certificate verify ok.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws
> User-Agent: curl/8.4.0
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
< Date: Wed, 08 May 2024 15:10:07 GMT
< Content-Type: application/json
< Content-Length: 98
< Connection: keep-alive
< x-amzn-RequestId: 1539f8a2-4b94-4727-a7f6-33dfa1cfd0d5
< X-Amzn-Trace-Id: root=1-663b95cf-35d447f776de2f4476fcb225;parent=495893fde4e46190;sampled=0;lineage=b073041a:0
<
{ [98 bytes data]
* Connection #0 to host i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws left intact
00000000: 1f8b 0800 0000 0000 00ff edca c109 8030  ...............0
00000010: 0c05 d055 e23f 3b41 5711 0fd1 865a 6c1b  ...U.?;AW....Zl.
00000020: 497a 1171 7771 0e3d 3e78 1722 7746 9850  Iz.qwq.=>x."wF.P
00000030: b425 7a41 5da9 5b4e 498c 56ad 8789 7bd6  .%zA].[NI.V...{.
00000040: 4685 4f31 8c7f fc4c 9c47 544f 08d8 84a3  F.O1...L.GTO....
00000050: 9813 9bd0 62ba 4b1b 703f fb92 e0a3 3b02  ....b.K.p?....;.
00000060: 0000                                     ..

These bytes are indeed gzipped contents but both content-encoding: gzip and vary: accept-encoding headers present in v0.11.1 are now missing. If I pipe the response bytes into gunzip we now get the expected data:

$ curl -v -s --compressed https://i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws/ --output - | gunzip
*   Trying [2406:da18:8b4:9a03:b860:65b5:983e:f182]:443...
* Connected to i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws (2406:da18:8b4:9a03:b860:65b5:983e:f182) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [370 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [10 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [4991 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=*.lambda-url.ap-southeast-1.on.aws
*  start date: Dec 27 00:00:00 2023 GMT
*  expire date: Jan 25 23:59:59 2025 GMT
*  subjectAltName: host "i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws" matched cert's "*.lambda-url.ap-southeast-1.on.aws"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M03
*  SSL certificate verify ok.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws
> User-Agent: curl/8.4.0
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
< Date: Wed, 08 May 2024 15:10:11 GMT
< Content-Type: application/json
< Content-Length: 98
< Connection: keep-alive
< x-amzn-RequestId: 7e52d846-97e1-445e-aa12-10ddbdbb2934
< X-Amzn-Trace-Id: root=1-663b95d3-0c67e94d4827a09c3601d3a3;parent=4704ebce03d0e894;sampled=0;lineage=b073041a:0
<
{ [98 bytes data]
* Connection #0 to host i46lp737huic6dh57chtwj72oa0gcsnd.lambda-url.ap-southeast-1.on.aws left intact
{"data":["long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer","long data to trigger compression layer"],"msg":"headers are broken!"}%

[calavera]

Ugh, ok. Sorry about that. I yanked that version from crates.io. You're right, the problem was introduced in that PR. We need to move Lambda URLs to their own payloads instead of reusing the Alb payloads: https://github.com/awslabs/aws-lambda-rust-runtime/blob/main/lambda-events/src/event/lambda_function_urls/mod.rs

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.