H264 decoder behaves incorrectly when reconfigured with different aspect ratio

[vladikadiroff]

Version

Media3 main branch

More version details

Media3 1.1.x
Media3 1.3.x
Media3 1.4.x
Media3 1.5.x
Media3 1.6.0-alpha1

Devices that reproduce the issue

Devices with c2.exynos.h264.decoder that reproduce the issue

• Pixel 7 Pro (Android 14 - 15)
• Pixel 8 Pro (Android 14 - 15)

Devices with c2.android.avc.decoder that reproduce the issue

• Pixel 4A (Android 14)
• Pixel 6 Pro (Android 14)
• Pixel 7 Pro (Android 14 - 15)
• Pixel 8 Pro (Android 14 - 15)
• Poco F3 (Android 13)
• Poco M5 (Android 13)
• Poco X6 Pro (Android 14)
• Xiaomi A3 (Android 11)
• Xiaomi 13T (Android 14)
• Xiaomi Redmi 8 Pro (Android 14)
• Xiaomi Redmi Note 9s (Android 12)
• Vivo Y33s (Android 13)
• Vivo x Flip (Android 14)
• One Plus 7 pro (Android 11)
• Nothing Phone 1 (Android 13)
• Samsung A50 (Android 11)
• Samsung S23 Ultra (Android 14)
• Motorola Razr 50 Ultra (Android 14)
• Android Studio Emulators (Android 12 - 15, also maybe less 12)

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Yes

Reproduction steps

It is easier to test using the sample app: https://github.com/vladikadiroff/media3-issue,
but you can also reproduce this in "demo-session" app by replacing the source links and attach MediaCodecSelector from https://gist.github.com/vladikadiroff/45daf4d68d8bcb4b1baffed63a433cd2.

1. Open the app
2. Click next button until the 3rd video appears
3. Playback failed

Expected result

Playlist is playing normally

with_fix.mp4

Actual result

Playback failed

issue.mp4

Media

Sample App:
media3-h264-broken.apk.zip

MediaCodecSelector for Reproduction:
https://gist.github.com/vladikadiroff/45daf4d68d8bcb4b1baffed63a433cd2

Media URLs:

https://bl.rutube.ru/route/2ff1f14bfebbe5634d278c648cc6de86.m3u8?guids=b90d44fe-f06e-4c7f-a581-c7116d1a473c_608x1080_1519012_D39667_B1314619A192000_F30A44100_avc1.640029_mp4a.40.2,0ce1b425-a61f-40af-8910-963a9b184e93_136x240_361044_D39667_B289700A64200_F30A44100_avc1.42c01f_mp4a.40.2,9617f7f1-55cf-492e-b587-f422553a6522_408x720_940485_D39667_B736087A192000_F30A44100_avc1.640029_mp4a.40.2,88b6838a-9bcb-46f0-b722-4f9b593ffbd7_208x360_501465_D39667_B430120A64200_F30A44100_avc1.42c01f_mp4a.40.2,60f244aa-3163-435b-b8cf-74d35be010f5_816x1440_2249244_D39667_B2044856A192000_F30A44100_avc1.640034_mp4a.40.2,f22360d9-dd6c-4c35-a936-b359c08f96e6_88x144_172858_D39667_B101511A64200_F30A44100_avc1.42c01f_mp4a.40.2,a1f7dca1-eb55-4827-9e2d-2ed3450e1605_272x480_568444_D39667_B431229A128119_F30A44100_avc1.4d401f_mp4a.40.2&sign=vP2Vz8QlYBIODz-wdpuvlQ&expire=2020699397&guarantee=7&scheme=https&minimal_long_temperature=300000

https://bl.rutube.ru/route/c2cf2d92119ccab132ad699458451fce.m3u8?guids=87e6d49e-1696-49bb-a453-47f02e3828b3_256x144_371448_D6765360_B236316A64000_F25A44100_avc1.42c01f_mp4a.40.2,ef7dfe4e-f9b0-40d2-b551-fc11a6aa90a8_432x232_642613_D6765360_B507481A64000_F25A44100_avc1.42c01f_mp4a.40.2,d42dab1b-3865-4519-9417-f878d196f742_640x360_1084340_D6765360_B949208A64000_F25A44100_avc1.42c01f_mp4a.40.2,548fa999-fce5-4c71-a37b-ca533f36b05b_856x480_1232930_D6765360_B968230A128000_F25A44100_avc1.4d401f_mp4a.40.2,519407aa-7bf6-490b-aa2f-d70c242466f5_1280x720_2070581_D6765360_B1677879A192000_F25A44100_avc1.640029_mp4a.40.2,14da8ad5-f497-42e9-9ab4-5b8c873e53e8_1920x1080_3890152_D6765360_B3497450A192000_F25A44100_avc1.640029_mp4a.40.2&sign=rq6XzKCy5PYfIyXf_9_8tA&expire=2020699377&guarantee=6&scheme=https&minimal_long_temperature=300000

https://bl.rutube.ru/route/192d0fe316aa7a2ab770ce578f5a7540.m3u8?guids=ddc15d68-e53e-48f9-bae6-e031da021cc2_256x144_568311_D5503120_B433178A64000_F25A44100_avc1.42c01f_mp4a.40.2,00192241-2eda-422b-b84d-3438cb624c72_432x232_1045583_D5503120_B910450A64000_F25A44100_avc1.42c01f_mp4a.40.2,62c27499-607b-48d2-9689-feb56c82e354_640x360_1776586_D5503120_B1641453A64000_F25A44100_avc1.42c01f_mp4a.40.2,6b0e22b1-7b74-4332-95a4-1bf526f5f902_1920x1080_5616400_D5503120_B5223696A192000_F25A44100_avc1.640029_mp4a.40.2,9963f111-6b1a-4fb5-86ae-2b715b9a6a08_856x480_1862389_D5503120_B1597687A128000_F25A44100_avc1.4d401f_mp4a.40.2,57d0d863-19e5-4db1-bc75-8b85da6a4741_1280x720_3024962_D5503120_B2632259A192000_F25A44100_avc1.640029_mp4a.40.2&sign=hEvSYkYZIJPQaUvRW3XFDA&expire=2020699353&guarantee=6&scheme=https&minimal_long_temperature=300000

https://bl.rutube.ru/route/4c9da41e277d036cb6bffc17a01ed1d9.m3u8?guids=058d4f53-8f85-4ffe-8857-16ea89398fca_272x480_488270_D33025_B385337A96000_F30A44100_avc1.4d401f_mp4a.40.2,47efff58-a2ff-458d-a339-9e4f0b0af650_88x144_173302_D33025_B102546A64155_F30A44100_avc1.42c01f_mp4a.40.2,461c6f6f-d208-45c2-b598-8ec62f2f5cbe_208x360_502295_D33025_B433457A64155_F30A44100_avc1.42c01f_mp4a.40.2,6e745cc6-105b-4219-886f-bcebf012870b_408x720_845200_D33025_B744348A96000_F30A44100_avc1.640029_mp4a.40.2,a7f732b6-9422-40fd-9789-1e1fd5e55f87_608x1080_1625210_D33025_B1528911A96000_F30A44100_avc1.640029_mp4a.40.2,dbf26a41-ec66-4f6b-a54f-a2bfc4c01d0a_136x240_376925_D33025_B307357A64155_F30A44100_avc1.42c01f_mp4a.40.2&sign=cqHoZkCOPkFCmkua5GP6eg&expire=2020699329&guarantee=6&scheme=https&minimal_long_temperature=300000

Bug Report

• You will email the zip file produced by adb bugreport to [email protected] after filing this issue.

[vladikadiroff]

I have investigated this problem and found the following:

When changing media content, the MediaCodecVideoRenderer calls the method canReuseCodec, which in turn calls MediaCodecInfo.canReuseCodec, checking if the codec can be reused or not.

MediaCodecInfo.canReuseCodec() performs four checks:

1. Rotation changes
2. Changes in the size of the new content if the codec is non-adaptive
3. Changes in ColorInfo
4. needsAdaptationReconfigureWorkaround

If these checks pass successfully, the method initializationDataEquals is called, which determines the codec reuse strategy - with or without reconfiguration (flags REUSE_RESULT_YES_WITH_RECONFIGURATION and REUSE_RESULT_YES_WITHOUT_RECONFIGURATION).

If the checks do not pass, REUSE_RESULT_NO is returned, indicating that the codec cannot be reused and needs recreated.

After obtaining the result from MediaCodecInfo.canReuseCodec(), in MediaCodecVideoRenderer we additionally check that the new content size has not increased and that the maxInputSize of the new content is not greater than that of the old content.

If the additional checks pass successfully, we return the result obtained from MediaCodecInfo.canReuseCodec(); otherwise, we return REUSE_RESULT_NO.

---

How will the codecs c2.exynos.h264.decoder and c2.android.avc.decoder behave if we try to switch from content with a resolution of 1280x720 to content with a resolution of 408x720?

1. The method MediaCodecVideoRenderer.canReuseCodec() will be called, which will invoke MediaCodecInfo.canReuseCodec().
2. MediaCodecInfo.canReuseCodec() will perform the four checks described above, which will pass successfully.
3. MediaCodecInfo.canReuseCodec() will call the method initializationDataEquals, which will return the result REUSE_RESULT_YES_WITH_RECONFIGURATION.
4. After receiving the result in MediaCodecVideoRenderer.canReuseCodec(), methods will be called to check that the current content size is not larger than the previous one, which will pass successfully.
5. The method MediaCodecVideoRenderer.canReuseCodec() will return the result REUSE_RESULT_YES_WITH_RECONFIGURATION.
6. Then the method updateCodecOperatingRate() will be called, which will return a successful result, leading to the assumption that the reconfiguration was successful.
7. The render method will be called, but drainOutputBuffer will always return false.

---

How to fix:
Option 1: Add an additional check to the MediaCodecInfo.canReuseCodec() method (see #2004)
Option 2: Use a modified RenderersFactory class https://gist.github.com/vladikadiroff/a63a0e5e55891dbed9d3b4a6e74a0cd1

[microkatz]

@vladikadiroff

Thank you for reporting the issue and providing the media content as well as the potential fix in #2004!

I've created an internal bug, [b/387267733 for our reference] to discuss with the codec team about this issue before moving forward with your suggested solution. Before creating the workaround I want to pinpoint the exact issue. I'll keep you updated for when a resolution is decided.

[vladikadiroff]

Hi @microkatz, maybe you have any updates?

[microkatz]

@vladikadiroff

There is not an update yet but thank you for your patience.