SocketTimeout Error

[ElonMusked]

Hi, sometimes when I'm scraping comments, I get this error after a while. And the worst part is, it hits 5/5 max retry after some time and program stops working. And I have to start the whole process from first again...

Is there anyway to fix this? Or if you don't know what's going on, is there anyway I can increase that 5 max tries to a bigger number like 1,000 or something?

[mattwright324]

Hi @ElonMusked

There currently isn't a way to increase the max retry count but that could be a good idea for more settings/customization.

What is the quality of your internet? download speed, over ethernet, over wifi? are there any other things downloading on your device or elsewhere on your network (windows updates, game updates, netflix...) that could be degrading your local internet?

[ElonMusked]

Thanks for replying back.
I got that error on my laptop, thought there is something wrong with my 3 MB internet so bought a VPS today and got the exact same errors after a while. Right now, I'm testing it on a windows server 2016 with 760 Mbps internet speed (~90 MB/s). No updates, no downloads, nothing. It's just the app itself running (double confirmed it via resource monitor of task manager).
Tho I might have an idea what's going on but I'm not sure.
So here is the situation, the video I'm trying to get all comments of has around 5 million comments (😂).
I tried it 3 different times and all of them failed in middle of the way, first one at around 3 million, second one around 1.2 million and the last one was around 4 million. I tried different refresh styles (everything, moderate, light) but nope. The only thing that is stopping the process is that [5/5] error, everything else works just fine.

[mattwright324]

Are you using a VPN at all? There was an issue #78 that I didn't get to verify but sounds like may cause connection issues in a Java application.

[ElonMusked]

Nope, it's the internet connection itself, no vpn, no proxy. Btw Sorry I checked again and the comments were not 5 million, I thought it's a different video.
The process is going with [3/5] error and over 2 million comments fetched so far, would you like to have a look at my vps to see the case in real time? Of course I won't post the credentials here, I can send them to your gmail. (Ain't got anything personal on it)

[mattwright324]

The application is currently working fine on my internet and the daily quota for today looks fine too.

What specifically is happening then or the order than things occured. Do you mind sharing the specific video to verify its total comment count?

If a single video really has more than 2 million comments...

1. Was it grabbing fine until it hit 2 million and then times out?
2. Grabbed fine until hit it 2 million, now always times out on any other attempt?
3. Always times out and never grabbed any/many comments?
4. Is it just this video or times out with any action in the app, other videos, youtube search?

[mattwright324]

Alternatively you could just share the statistics for the video if you don't want to share which one specifically. My metadata tool could help with that.

https://mattw.io/youtube-metadata/

[ElonMusked]

Yeah sure, here is the video:
https://www.youtube.com/watch?v=jNQXAC9IVRw

So to answer your questions, I calculated everything:

Started at 04:20
First error at 04:38 ~225,000 comments
Second error at 04:46 ~325,000 comments

I created this topic at 05:26
Third error at 05:27 ~837,500 comments

But no error since then, so basically from 837k till 2.5m no error. Now another thing to note is that, this time for [1/5 ]and [2/5] error, program was minimized and I had my vps closed (not shutdown). But [3/5] was when vps was open BUT program still minimized.
The other two times that program failed at 3m and 1.2m, I was asleep and your program was doing it it again in minimized mode. Idk it could be totally irrelevant to the issue but had to tell you every detail.

I didn't really have any problem with other videos because they didn't have as much comments. And also for this video I didn't take a look at comments when program failed cuz it was pointless anyway.

[mattwright324]

A single video with so many comments on it is certainly an edge case. I will try it out on my own and see what happens. It isn't the same thing but potential for something undocumented to be the case such as this issue where the YouTube API will stop returning videos from a playlist when you get to 20,000 videos. So you can't query for every video from a channel that has uploaded more than that.

https://github.com/mattwright324/youtube-metadata/wiki/YouTube-API-Notes#limit-for-playlistitemslist

[ElonMusked]

Thanks.
I'll let you know if I could grab all comments because so far without any minimizes, it's going smoothly with no issues.

[mattwright324]

I'm assuming its just network variability and bad luck. I haven't reached any limitations yet thankfully, over 2 million comments and not one timeout, will keep it going. Still would be good for me to add retry configuration.

[ElonMusked]

Mine is also over 5 millions now and no errors yet. But yeah, a retry configuration (both retries count and timeout maybe?) would be greatly appreciated cuz it shows its use when you have a weak internet connection that drops all the time or in my case having bad luck 😁
Thanks again for all your support

[ElonMusked]

Mission success!
Your program successfully dumped all those 11m comments. Nice job