feat(StreamsExtractor): generate PoToken #7069

FineFindus · 2025-02-03T20:07:30Z

Implements support for locally generating PoTokens using the device web view. This is a direct port of
TeamNewPipe/NewPipe#11955 to native Kotlin.
However, there are some notable differences:

NewPipe use an additional PoTokenGenerator interface. As there is only one implementation right now, this is skipped
NewPipe handles broken web views on older devices. Since we're increasing minSdk in BREAKING CHANGE: bump minSdk to 26 #7047, I'm hoping we can get away without

~~It is marked as a draft, since while it does generate valid PoTokens, playback does not work (Source error: open failed: ENOENT; help is appreciated).~~

Closes: #7065

FineFindus · 2025-02-04T07:58:00Z

Please don't add unnecessary comments. Linking an issue to be closed is fine, but the rest just adds noise. I'm already well aware of the NewPipe PR (see description above).

Figim · 2025-02-04T08:25:33Z

Please don't add unnecessary comments. Linking an issue to be closed is fine, but the rest just adds noise. I'm already well aware of the NewPipe PR (see description above).

Sorry.

"Source Error" only occurs if "HLS" is enabled. DASH works great. tested
Live streams also play smoothly on HLS

Bnyro · 2025-02-04T08:44:46Z

The crash log is a temporary DNS issue on your side, that's not LibreTube related.

Figim · 2025-02-04T14:35:33Z

A source error still occurs when starting a video in HLS.

XRecorder_20250205_01.mp4

FineFindus · 2025-02-05T14:56:55Z

A few questions:

Since HLS does not work with potokens either, should it be removed/disabled?
JavaScriptUtil.kt uses nanojson, should it be ported as well? I left it for now, as it would make it easier to follow future changes. There isn't any noticable impact on filesize, since it is already included in NewPipeExtractor.

YT-Advanced · 2025-02-05T15:06:55Z

Since HLS does not work with potokens either, should it be removed/disabled?

This issues seem to be NewPipeExtractor bug itself, since HLS don't need pot but it still be appended in streaming url

JavaScriptUtil.kt uses nanojson, should it be ported as well? I left it for now, as it would make it easier to follow future changes.

Uhm u should port it also

gechoto · 2025-02-05T15:47:21Z

With the current code the purpose of JavaScriptUtil.kt is not perfectly clear at first look and it might take a while to notice that this is for po tokens.
The class and function names are pretty generic.
For example in the first function it says something about some "challenge data". Yeah cool, which challenge data? Someone new to the project won't know.

I suggest to do one of these to make it more obvious it is about po tokens:

Move JavaScriptUtil.kt and PoTokenWebView.kt to a sub-package named potoken
Rename JavaScriptUtil.kt to something like PoTokenJavaScriptUtil.kt
Rename the functions and extend the documentation in JavaScriptUtil.kt to make it clear this is for po tokens

NewPipe did the first option here and put everything po token related into a sub package:
https://github.com/TeamNewPipe/NewPipe/tree/dev/app/src/main/java/org/schabi/newpipe/util/potoken

Bnyro · 2025-02-15T14:15:20Z

Since HLS does not work with potokens either, should it be removed/disabled?

It shouldn't be completely removed because it might still work with some private Piped instances, however disabling it by default is possible.

JavaScriptUtil.kt uses nanojson, should it be ported as well? I left it for now, as it would make it easier to follow future changes.

Yes, I would like to keep the used libraries consistent across the code. Since the NanoJSON code part is only very small and the code differs already due to the fact that it's ported to Kotlin, I think it's reasonable to port it.

I don't have much time to look into the problems you mentioned at the moment (and the next few weeks) unfortunately due to tons of exams at university. I'll see if I can find some time to allocate for reviewing soon, but no guarantees.

HLS does not work by default, leading to a bad user expierence, as videos won't play, disable it. Ref: libre-tube#7069 (comment)

FineFindus · 2025-02-15T21:36:56Z

Disabled and ported.

I don't have much time to look into the problems you mentioned at the moment (and the next few weeks) unfortunately due to tons of exams at university. I'll see if I can find some time to allocate for reviewing soon, but no guarantees.

Sure, no pressure. I would also like to clean up the commits before merging. Good luck with your exams.

app/src/main/java/com/github/libretube/api/local/JavaScriptUtil.kt

gechoto · 2025-02-17T14:45:32Z

There should be some better error handling.
Currently if something in parseChallengeData or parseIntegrityTokenData fails the app crashes.

gechoto · 2025-02-17T18:16:26Z

There is a NewPipe PR which ports po token code to coroutines

TeamNewPipe/NewPipe#12028

Maybe this could be used as a base.
With the code in the linked PR the app does not crash.
It does handle exceptions better.

FineFindus · 2025-02-17T19:18:47Z

Currently if something in parseChallengeData or parseIntegrityTokenData fails the app crashes.

Please attach a crash log/reproducible case and don't post vague comments about 'something' crashing.

gechoto · 2025-02-17T19:29:34Z

Currently if something in parseChallengeData or parseIntegrityTokenData fails the app crashes.

Please attach a crash log/reproducible case and don't post vague comments about 'something' crashing.

One specific crash log won't be useful. It crashes on any error.
Just do something obviously wrong in parseChallengeData, could be corrupted challenge data or for example try to access something which does not exist:

val messageId = challengeData[10].jsonPrimitive.content

HLS does not work by default, leading to a bad user expierence, as videos won't play, disable it. Ref: libre-tube#7069 (comment)

app/src/main/java/com/github/libretube/api/local/PoTokenWebView.kt

Implements support for locally generating PoTokens using the device webview. This is a direct port of TeamNewPipe/NewPipe#11955 to native Kotlin. Closes: libre-tube#7065

gechoto · 2025-03-25T12:45:35Z

app/src/main/java/com/github/libretube/api/poToken/JavaScriptUtil.kt

+    val privateDoNotAccessOrElseSafeScriptWrappedValue = challengeData[1]
+        .takeIf { it !is JsonNull }
+        ?.jsonArray
+        ?.find { it.jsonPrimitive.isString }
+
+    val privateDoNotAccessOrElseTrustedResourceUrlWrappedValue = challengeData[2]
+        .takeIf { it !is JsonNull }
+        ?.jsonArray
+        ?.find { it.jsonPrimitive.isString }


since you access challengeData null-safe now
ae23ba3

this could also be applied to these two

but I don't think this is a perfect solution
because there are still other things which can go wrong and make the app crash

accessing something on challengeData out of bounds was only one example

there should be better error handling in general instead
maybe this whole parseChallengeData should be ran in try/catch

there is this NewPipe PR now which seems to handle errors better:
TeamNewPipe/NewPipe#12028

gechoto · 2025-03-25T13:10:42Z

app/src/main/java/com/github/libretube/api/poToken/JavaScriptUtil.kt

+    val messageId = challengeData.getOrNull(0)?.jsonPrimitive?.content
+    val interpreterHash = challengeData.getOrNull(3)?.jsonPrimitive?.content
+    val program = challengeData.getOrNull(4)?.jsonPrimitive?.content
+    val globalName = challengeData.getOrNull(5)?.jsonPrimitive?.content
+    val clientExperimentsStateBlob = challengeData.getOrNull(7)?.jsonPrimitive?.content


idk maybe it is better to not use getOrNull here and instead log an exception

problem is that currently LibreTube crashes instead of logging the exception and showing the user an error message
all of this stuff should be wrapped in an error handler
not just these few lines

from my experience integrating po token support in OuterTune error handling works better with the changes from this PR:
TeamNewPipe/NewPipe#12028

gechoto mentioned this pull request Feb 3, 2025

PO token support OuterTune/OuterTune#283

Closed

This comment was marked as spam.

Sign in to view

FineFindus force-pushed the feat/potoken branch from 3a637c3 to 98a0bf1 Compare February 4, 2025 14:15

Figim mentioned this pull request Feb 5, 2025

The media doesn't play fast4x/RiMusic#5615

Closed

2 tasks

FineFindus force-pushed the feat/potoken branch from c0c210f to 817521a Compare February 5, 2025 14:56

gechoto mentioned this pull request Feb 5, 2025

Add support for poTokens OuterTune/OuterTune#294

Merged

7 tasks

FineFindus force-pushed the feat/potoken branch from 5b9b297 to 6ecad37 Compare February 15, 2025 18:09

FineFindus added a commit to FineFindus/LibreTube that referenced this pull request Feb 15, 2025

feat(Preferences): disable HLS by default

6ecad37

HLS does not work by default, leading to a bad user expierence, as videos won't play, disable it. Ref: libre-tube#7069 (comment)

FineFindus added a commit to FineFindus/LibreTube that referenced this pull request Feb 15, 2025

feat(Preferences): disable HLS by default

d3eafde

HLS does not work by default, leading to a bad user expierence, as videos won't play, disable it. Ref: libre-tube#7069 (comment)

gechoto reviewed Feb 17, 2025

View reviewed changes

app/src/main/java/com/github/libretube/api/local/JavaScriptUtil.kt Outdated Show resolved Hide resolved

FineFindus added a commit to FineFindus/LibreTube that referenced this pull request Feb 18, 2025

feat(Preferences): disable HLS by default

d7b1ace

HLS does not work by default, leading to a bad user expierence, as videos won't play, disable it. Ref: libre-tube#7069 (comment)

gechoto reviewed Feb 21, 2025

View reviewed changes

app/src/main/java/com/github/libretube/api/local/PoTokenWebView.kt Outdated Show resolved Hide resolved

FineFindus mentioned this pull request Feb 23, 2025

network error when using bg-helper #7114

Open

3 tasks

This was referenced Feb 24, 2025

"IOS Player response is not valid" error with Local Extraction enabled (Android 13, v0.27.0) #7119

Closed

Source Error When Playing Any Video #7146

Closed

Bnyro mentioned this pull request Mar 1, 2025

IOS player response is not valid #6982

Closed

3 tasks

Figim mentioned this pull request Mar 20, 2025

ExoPlayer "Response code: 403" (Source error). Video middle/start. (Full local mode/DASH) #7227

Open

Figim mentioned this pull request Mar 20, 2025

Sign in to confirm you're not a bot #7219

Closed

3 tasks

FineFindus added 5 commits March 23, 2025 21:58

feat(StreamsExtractor): generate PoToken

26babb9

Implements support for locally generating PoTokens using the device webview. This is a direct port of TeamNewPipe/NewPipe#11955 to native Kotlin. Closes: libre-tube#7065

fix(potoken): access challengeData null-safe

ae23ba3

refactor(potoken): use null over empty string for Json data

b2787d9

refactor(potoken): remove outdated build configuration

3bbf9c3

style(potoken): add empty line

34de9f9

FineFindus force-pushed the feat/potoken branch from 6ecad37 to 34de9f9 Compare March 23, 2025 21:22

FineFindus marked this pull request as ready for review March 23, 2025 21:22

gechoto reviewed Mar 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(StreamsExtractor): generate PoToken #7069

feat(StreamsExtractor): generate PoToken #7069

FineFindus commented Feb 3, 2025 •

edited

Loading

This comment was marked as spam.

FineFindus commented Feb 4, 2025

Figim commented Feb 4, 2025 •

edited

Loading

Bnyro commented Feb 4, 2025

Figim commented Feb 4, 2025 •

edited

Loading

FineFindus commented Feb 5, 2025 •

edited

Loading

YT-Advanced commented Feb 5, 2025

gechoto commented Feb 5, 2025 •

edited

Loading

Bnyro commented Feb 15, 2025

FineFindus commented Feb 15, 2025

gechoto commented Feb 17, 2025 •

edited

Loading

gechoto commented Feb 17, 2025

FineFindus commented Feb 17, 2025

gechoto commented Feb 17, 2025 •

edited

Loading

gechoto Mar 25, 2025 •

edited

Loading

gechoto Mar 25, 2025

feat(StreamsExtractor): generate PoToken #7069

Are you sure you want to change the base?

feat(StreamsExtractor): generate PoToken #7069

Conversation

FineFindus commented Feb 3, 2025 • edited Loading

This comment was marked as spam.

FineFindus commented Feb 4, 2025

Figim commented Feb 4, 2025 • edited Loading

Bnyro commented Feb 4, 2025

Figim commented Feb 4, 2025 • edited Loading

A source error still occurs when starting a video in HLS.

FineFindus commented Feb 5, 2025 • edited Loading

YT-Advanced commented Feb 5, 2025

gechoto commented Feb 5, 2025 • edited Loading

Bnyro commented Feb 15, 2025

FineFindus commented Feb 15, 2025

gechoto commented Feb 17, 2025 • edited Loading

gechoto commented Feb 17, 2025

FineFindus commented Feb 17, 2025

gechoto commented Feb 17, 2025 • edited Loading

gechoto Mar 25, 2025 • edited Loading

Choose a reason for hiding this comment

gechoto Mar 25, 2025

Choose a reason for hiding this comment

FineFindus commented Feb 3, 2025 •

edited

Loading

Figim commented Feb 4, 2025 •

edited

Loading

Figim commented Feb 4, 2025 •

edited

Loading

FineFindus commented Feb 5, 2025 •

edited

Loading

gechoto commented Feb 5, 2025 •

edited

Loading

gechoto commented Feb 17, 2025 •

edited

Loading

gechoto commented Feb 17, 2025 •

edited

Loading

gechoto Mar 25, 2025 •

edited

Loading