-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] Index out of bounds (IndexError) when importing playlist #4069
Comments
This comment was marked as duplicate.
This comment was marked as duplicate.
Check if this solves your issue: #4048 (comment) |
That seems to be an unrelated issue, they were trying to import subscriptions using the playlist dialog and got the error. I’m trying to import playlists through the playlist dialog and I’m getting that error. |
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
Same issue, maybe youtube has changed the export csv ? I don't have a way to verify it, but I looked at the code. In invidious/src/invidious/user/imports.cr Lines 33 to 42 in 069e91d
it seems that this function expects these data points to be in the CSV:
which are not inside the csv. The title is in the filename, but other fields are not present anywhere.
Which signifies a discrepancy and possible export csv schema change. Update: youtube definitely changed the schema. Now the csvs exported are as follows. For
Do note that I am translating titles and schema from polish. I'll try later today to write a PR for this bug, but I know of Crystal since today, so I may give up during env setup :/ Additionaly, since the schema changed a PR that changes import function is necessary anyway - I think it'll be very useful to implement importing every playlist at once as a directory. Since the data is split in two types of files anyway, users will need to upload at least two files even when importing one playlist and it would be useful to not have to upload these files for each playlist to import. There are two problems still: Since the files with video ids do not contain playlist id it may be difficult to join the data between the two types of CSVs youtube gives - the only identifiers are playlist title, which is obviously unreliable and may also be escaped since playlist title is part of the filename and playlist creation timestamp which then can be used to search for the CSV file with a video where The second problem is that users may not want to upload all playlist, but then they can simply delete the csv files of unwanted playlist. |
This comment was marked as spam.
This comment was marked as spam.
I wrote a small script that can be used as a starting point for transforming the new CSV format into the old format, as a temporary workaround until we can patch Invidious to parse the new format directly. It's a bit of a hack and you'll probably have to fix some things. You give as arguments the path of the Script text#!/usr/bin/env python3
import argparse
import csv
import os
import pathlib
import shutil
parser = argparse.ArgumentParser()
parser.add_argument("input_dir")
parser.add_argument("output_dir")
args = parser.parse_args()
input_dir = pathlib.Path(args.input_dir).resolve()
output_dir = pathlib.Path(args.output_dir).resolve()
os.chdir(input_dir)
playlists = {}
with open("playlists.csv") as f:
for record in csv.DictReader(f):
playlist_name = record["Playlist Title (Original)"]
playlists[playlist_name] = record
videos_by_playlist = {}
mangled_names = {
"Winter _23": "Winter '23",
"New Year_s Day": "New Year's Day",
}
for fname in os.listdir():
if not fname.endswith(".csv"):
continue
if fname == "playlists.csv":
continue
assert fname.endswith("-videos.csv"), fname
playlist_name = fname.removesuffix("-videos.csv")
playlist_name = mangled_names.get(playlist_name, playlist_name)
videos = []
with open(fname) as f:
for record in csv.DictReader(f):
videos.append(record)
playlists[playlist_name]["videos"] = videos
try:
shutil.rmtree(output_dir)
except FileNotFoundError:
pass
output_dir.mkdir()
for playlist_name, playlist in playlists.items():
visibility = playlist["Playlist Visibility"]
header = "Video Id,Unused 1,Unused 2,Unused 3,Title,Description,Visibility"
lines = []
lines.append(header)
lines.append(f",,,,{playlist_name},,{visibility}")
lines.append("")
lines.append(header)
for video in playlist["videos"]:
video_id = video["Video ID"].strip()
lines.append(f"{video_id},,,,,,")
with open(output_dir / f"{playlist_name}.csv", "w") as f:
f.writelines(line + "\n" for line in lines) |
How to add this in? On the webpage? |
The script is for transforming the Google Takeout CSV files before import. |
I see, so I add this to a file of text document? Then place this in takeout folder? |
You'll need some Python knowledge to be able to properly use and adapt the script to your use case. I would recommend seeking advice elsewhere, the issue tracker is primarily for technical discussion rather than general support. |
This code only works if the data was exported from Google while the main account language is set to English, and Google again did some changes on the naming etc. I have adjusted the Python Code: #!/usr/bin/env python3
import argparse
import csv
import os
import pathlib
import shutil
parser = argparse.ArgumentParser()
parser.add_argument("input_dir")
parser.add_argument("output_dir")
args = parser.parse_args()
input_dir = pathlib.Path(args.input_dir).resolve()
output_dir = pathlib.Path(args.output_dir).resolve()
os.chdir(input_dir)
playlists = {}
with open("playlists.csv") as f:
for record in csv.DictReader(f):
playlist_name = record["Playlist title (original)"] # title with lower case t
playlists[playlist_name] = record
videos_by_playlist = {}
mangled_names = {
"Winter _23": "Winter '23",
"New Year_s Day": "New Year's Day",
}
for fname in os.listdir():
if not fname.endswith(".csv"):
continue
if fname == "playlists.csv":
continue
assert fname.endswith(" videos.csv"), fname # - removed
playlist_name = fname.removesuffix(" videos.csv") # - removed
playlist_name = mangled_names.get(playlist_name, playlist_name)
videos = []
with open(fname) as f:
for record in csv.DictReader(f):
videos.append(record)
playlists[playlist_name]["videos"] = videos
try:
shutil.rmtree(output_dir)
except FileNotFoundError:
pass
output_dir.mkdir()
for playlist_name, playlist in playlists.items():
visibility = playlist["Playlist visibility"] # visibility with lower case v
header = "Video Id,Unused 1,Unused 2,Unused 3,Title,Description,Visibility"
lines = []
lines.append(header)
lines.append(f",,,,{playlist_name},,{visibility}")
lines.append("")
lines.append(header)
for video in playlist["videos"]:
video_id = video["Video ID"].strip()
lines.append(f"{video_id},,,,,,")
with open(output_dir / f"{playlist_name}.csv", "w") as f:
f.writelines(line + "\n" for line in lines) |
Just a quick workaround so people don't need to find this bug report in order to import their playlists again, since it's been 2+ months now. edit: Recreated all the sheebang to add a pull request with a feature as per documentation :-) |
Modified the script for the German version and also fixed newline character in windows. If you only need the newline fix:
with
Whole script:
|
With a few modifications, I was able to get the script provided above to work with my playlists. Well, mostly. I found that importing a long list causes the web server to time out with "504 Gateway Time-out". The timeout (on the inv.n8pjl.ca instance) occurs 60 seconds after the import begins. The playlist does get partially imported, creating an Invidious playlist with 256 entries out of the 267 in the source list. Update: I tried a few more, and it's definitely a hard time limit of 60 seconds, and nothing to do with the number of entries in the playlist. Here's my version (used with Python 3 on Windows):
|
Serious warning: Do not set the output_dir as |
My workaround from the beginning of this year works for the most part. |
How abourt someone of the devs adjusts the playlist import csv function after over 8 months of this? The python script is great and all, but I think, it is getting past the time, where we still should use a hack to import playlist, isn't it? Please, update the invidious' playlist import from csv function to the latest csv scheme of googles export automation |
Since the workaround I submitted at the beginning of January 2024 was marked as "uncompleted", now as "stale" while commenting on the pull request is unavailable, I wonder.. Just what is the problem with the patch? I just recompiled and patched it from scratch. Used the current google takeout format (in english!) and it still works as beautifully as before. Can any of the more knowing people please point me to why the patch is "uncompleted" as the tag says.. and more importantly, what I can do about it? Thanks! :-) |
A solution that worked for me is given below. Developers, please note first that the second script, given above by @Ajimaru , worked for me for some playlists apparently whenever the script was able to change the filename ending from
An alternative solution that worked for me: one might have success making Invidious to import the playlists as they are currently being exported from Youtube with the following modifications performed on each playlist, one at a time:
To:
In the second line, change PlaylistNameGoesHere to your playlist's name; there is no need to mention .csv in that line because this field refers to the playlist name you gave it, not the playlist's filename. If your playlist name you gave it had spaces, use them without using quotation marks. For example:
Note that these playlists also displayed correctly in Playlet when tried, which is an app/'channel' to watch Youtube without ads in Roku that uses Invidious as a backend, including playlists and subscription lists exported to Invidious: |
Modified version for italian takeout https://github.com/Nardo86/VarieEdEventuali/blob/main/InvidiousYTPlaylistConverter, I just finished converting all my youtube music playlist |
Describe the bug
I am unable to import any playlists, and this is the given error. I have tried on multiple instances and received the same error.
Steps to Reproduce
Logs
Title:
Index out of bounds (IndexError)
Date:
2023-08-24T20:18:37Z
Route:
/data_control?referer=%2Fsubscription_manager
Version:
2023.08.07-3450896 @ master
Backtrace
Screenshots
Additional context
The text was updated successfully, but these errors were encountered: