Welcome to BS-RoFormer Discussions!

[lucidrains]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[lucidrains]

is speech separation and music separation related fields, or different?

[lucidrains]

@jarredou wow, didn't expect such a lengthy response! this was informative, thank you!

[faroit]

@lucidrains @jarredou i could probably answer even more lengthy but I just wanted to drop here that we are organizing a source separation workshop next week (registration is free) and the authors of the RoFormer will be there to present the model.

[lucidrains]

@faroit awesome! won't be able to make it, but definitely want to return to Italy at some point (was at Lake Como for a wedding earlier this year)

[jarredou]

I wish I could free myself for it. I see that ZFTurbo is also participating to the workshop ! Nice ! :)

[turian]

@lucidrains I like @jareddou's analysis, in particular pointing to the cocktail party problem.

Historically, speech and music ML have been very different, but I think they are converging because of increasingly sophisticated ML algorithms. (With that said, speech or music people will typically choose hyperparameters and do model design based upon their domain, and then sometimes miss certain error classes of other domains that still require some expert knowledge. I think this is particularly acute in audio ML versus NLP or vision ML because our evaluation measures like SDR etc aren't that close to human perception, which is poorly understand.)

But to return to the key question about MSS versus speech separation, the difference is one of problem formulation. Separating speech from non-speech is relatively simply formulated, similar to vocal and instrumental separation. Separating individual speakers is a much more challenging formulation, because you're essentially predicting an ordered set.

MSS thought of as 4 (or 6 well specified instruments) makes it easy to formulate, but a bit boring. I think a lot of music producers would be very interested in separating music into all the individual instruments. This wouldn't really be done by having 100 different instrument models (and that wouldn't span all identifiable instruments anyway), rather by thinking of music source separation as predicting an unordered set of stems.

[jarredou]

A new potential SOTA in the domain was published few days ago, less ressources hungry than BS/MelRoformer it seems.
It would be great to see it open-sourced if authors don't publish their code !
https://arxiv.org/abs/2401.13276