Add support for WebDataset

[jbilcke]

Feature request / 功能建议

I propose for Finetrainers to support WebDataset as a dataset format.

Motivation / 动机

While working on VMS (UI wrapper around Finetrainers) I realized that I ended up using a similar format to WebDataset, except I upload multiple .zip files containing .mp4/.txt pairs, instead of .tar shards.

Unrelated to my project, I also notice some interest about using WebDataset in Finetrainers

Your contribution / 您的贡献

I've started refactoring my project to support WebDataset

[a-r-r-o-w]

@jbilcke Thanks for the recommendation! I recently rewrote a majority of the codebase to allow for this. This file lists all the supported dataset formats:

finetrainers/finetrainers/data/dataset.py

Line 467 in ee83d55

class VideoWebDataset(torch.utils.data.IterableDataset, torch.distributed.checkpoint.stateful.Stateful):

I haven't tested on a large scale run with a big webdataset yet, but I did verify it to be working with in a smaller setting so please let me know if it works when you give it a try.

I have a simple test dataset which I use for verifying loading in the fast-tests. You could try with it for a quick look:

finetrainers/tests/data/test_dataset.py

Lines 174 to 178 in ee83d55

	class VideoWebDatasetFastTests(unittest.TestCase):
	def setUp(self):
	self.num_data_files = 15
	self.dataset = VideoWebDataset("finetrainers/dummy-squish-wds", infinite=False)