Add ability to add proxy domains via ENV

[89Q12]

This PR adds the ability to add a list of proxy domains used for proxying videos, the domain is randomly chosen. If no domains are configured the HOST_URL is used.
The domains can be configured using in either the config.yaml or ENV
proxy_domains: ["..","...",".."]

Also this implementation assumes https to be available on the proxy domains.

[SamantazFox]

When using an external proxy URL, it would be better to redirect /lastest_version and /api/manifest/* to the given proxy domain, rather than the individual streams. That way, you ensure that the proxy server will fetch the video streams by itself and make sure that they're bound to its own IP (I'm not entirely sure if the streams are geo-restricted to a country, or to a smaller region).

[89Q12]

I'm not sure what do you exactly mean by that, but http3-ytproxy or piped-proxy have obviously no logic for /lastest_version. Sure invidious could be used itself when compiled with api only basically but its not as lightweight or fast as the a fore mentioned proxy.
I get the problem though, converting to draft for that reason.

[unixfox]

Well once invidious will have redis support I think compiling invidious in API only will become a viable solution for running proxies.

Or one can also run alternatives software of Invidious that support the same endpoints like:

• https://github.com/unixfox/invidious-stripped-down
• https://github.com/11Tuvork28/rusted-invidious
• their own custom implementation?

Technically with this PR someone has the choice of running what they want.

---

Like @SamantazFox said, I would also like to have as a feature the redirection to another domain for the endpoints /latest_version and /api/manifest/*.

[89Q12]

Okay kinda get it, well then the discussion on matrix was useless but sure will do that.

[SamantazFox]

@11Tuvork28 What do you think of the following?

Mode 1 - default, use invidious' integrated proxy (current situation)

video_proxy:
  redirect_mode: local
  # domains: <HOST_URL>

Mode 2 - redirects only the /videoplayback endpoint (e.g use Piped's http3-ytproxy)

video_proxy:
  redirect_mode: partial
  domains:
    - a.example.com
    - b.example.com

Mode 3 - redirects all video endpoints (implies that the proxy server supports /latest_version and can provide DASH/HLS manifests)

video_proxy:
  redirect_mode: full
  domains:
    - a.example.com
    - b.example.com

---

As an alternative:

media_proxy:
  # Redirect for `/videoplayback`, and possibly `/latest_version` and `/api/manifest/*`
  external_video_proxy: # enabled/disabled
  video_proxy_mode: # like above
  video_proxy_domains:
    - a.example.com

  # Redirect for `/vi/*`, `/vi_webp/*`, `/sb/*`, `/ggpht/*`, `/s_p/*` and `/yts/img/*`
  external_image_proxy: # enabled/disabled
  image_proxy_domains:
    - b.example.com

[unixfox]

I don't think there is a point for maintaining two kinds of redirect because you need to serve the content from the same IP address that generated the request to youtube servers.

People who will use this functionality will want to serve a working experience to their users, so it doesn't make sense to redirect to the incorrect proxy resulting in a playback error.

[89Q12]

@SamantazFox
That's a nice solution, I like it and would implement it that way including the image part.
I would however change the structure:

media_proxy:
  # Redirect for `/videoplayback`, and possibly `/latest_version` and `/api/manifest/*`
  external_video_proxy: 
    enabled: true/false
    proxy_mode: # like above
    proxy_domains:
      - a.example.com

  # Redirect for `/vi/*`, `/vi_webp/*`, `/sb/*`, `/ggpht/*`, `/s_p/*` and `/yts/img/*`
  external_image_proxy:
    enabled: true/false
    append_host_to_uri: true/false
    image_proxy_domains:
      - b.example.com

Reason behind this is that the file structure is clearer in my opinion and append_host_to_uri would be needed for Piped's proxy implementations.

---

@unixfox

No you don't, my instance produces requests from one of those IP's:

• 167.235.242.98(GER)
• 159.69.152.204(GER)
• 65.108.249.34(HEL)
• 167.235.202.120(GER)
• 116.203.183.22(GER)
• 95.217.189.191(HEL)

While the proxy server is on 45.83.107.127(GER), it works great. Those IP's aren't even in the same country but it somehow works.
Here proof:

What you are seeing is a port forward to the pod in Helsinki with IP 65.108.249.34 making requests to 45.83.107.127, using dash that go through just fine.

[unixfox]

No you don't, my instance produces requests from one of those IP's:

Yes you need to serve it from the same IP, it's for encrypted videos (music videos and more) like this one: https://www.youtube.com/watch?v=pRpeEdMmmQ0.

Related: #2850

[89Q12]

I get that(Have read the issue before) and I tried this and a few others I found in ytdl issues. They all work fine what can I say?
https://watch.thekitty.zone/watch?v=pRpeEdMmmQ0&quality=dash&local=1
https://watch.thekitty.zone/watch?v=qOtgxpDyGPw&quality=dash&local=1
https://watch.thekitty.zone/watch?v=AAbokV76tkU&quality=dash&local=1

In any case I still see the image_proxy as relevant, because invidious isn't MT and static content serving is better done via performant proxies.

[SamantazFox]

I don't think there is a point for maintaining two kinds of redirect because you need to serve the content from the same IP address that generated the request to youtube servers.

People who will use this functionality will want to serve a working experience to their users, so it doesn't make sense to redirect to the incorrect proxy resulting in a playback error.

Imo, both redirect options make sense. I see the following use cases:

• Invidious + http3-ytproxy on the same machine, only to benefit from HTTP/3
• invidious + multiple servers running http3-ytproxy (or similar) in the same rack/DC
• invidious with a bunch of distant video servers, maybe in another country, likely managed by another pice of software (each server provides its own stream URLs)

I would however change the structure:

Ah, sure, of course, it was only a quick draft. I definitely prefers your solution!

Edit: maybe some simple host rewrite rules, rather than append_host_to_uri? e.g:

• include_host_domain: no -> no domain
• include_host_domain: param -> regular invidious way
• include_host_domain: subdomain -> what unixfox does on yewtube for video

[unixfox]

• Invidious + http3-ytproxy on the same machine, only to benefit from HTTP/3

Why would you want to do that? HTTP3 is literally useful for high latency connections between a client and a server. That doesn't make any sense here.

• invidious + multiple servers running http3-ytproxy (or similar) in the same rack/DC

In this configuration you can always have a reverse proxy that serve the requests for the endpoints /videoplayback, /latest_version, dash manifest (and more) to random servers.

Anyway, off to my real suggestion:

I would very much prefer, as a first batch, to keep this functionality very simple, simple redirect for both endpoints /latest_version and dash manifest to randomly chosen domains. We set the setting parameters so that they can be expanded later for much more advanced usages, like the suggestion written in #3658 (comment)

This way we can provide a functionality that for the moment suit everyone and because it will be "simple" to code we can ship this functionality in the coming weeks.

[github-actions]

This pull request has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely abandoned or outdated. If you think this pull request is still relevant and applicable, you just have to post a comment and it will be unmarked.

[github-actions]

This pull request has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely abandoned or outdated. If you think this pull request is still relevant and applicable, you just have to post a comment and it will be unmarked.

[89Q12]

FYI: Not sure when if or when I will come around to actually implement it.

[github-actions]

This pull request has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely abandoned or outdated. If you think this pull request is still relevant and applicable, you just have to post a comment and it will be unmarked.