Music-STAR

Music style translation aims to generate variations of existing pieces of music by altering the style-related characteristics of the original piece while content, such as the melody, remains unchanged. These alterations could involve timbre translation, re-harmonization, or music rearrangement. Previous studies have achieved promising results utilizing time-frequency and symbolic music representations. Music style translation on raw audio has also been investigated and applied to single-instrument pieces. Although processing raw audio is more challenging, it provides richer information about timbres, dynamics, and articulations. We introduce Music-STAR, the first audio-based translation system that translates the existing instruments in a piece into a set of target instruments without using source separation. To conduct our experiments, we use the StarNet dataset, which includes strings-piano and vibraphone-clarinet mixtures alongside their stems. We also compare the performance of our model to baseline approaches performed by single-instrument translation and separation-translation pipelines.

Music-STAR StarNet

1. Dataset Examples
2. Single-Instrument Pipeline
3. Seperation-Translation Pipeline
4. Embedding-supervised Method
5. Stem-supervised Music-STAR
6. Mixture-supervised Music-STAR
7. Comparison
8. Evaluation
1. Statistical Analysis on Subjective Evaluation
2. Demucs Performance Analysis
9. More Audio Samples

The following tables include some random samples both from the StarNet dataset and some famous songs, demosntrating ten music pieces performed by strings-piano and clarinet-vibraphone combinations.

Name	Clarinet	Vibraphone	Clarinet-Vibraphone	String	Piano	Strings-Piano
Pirates of Caribbean Theme
My Heart Will Go On
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Bianche

The following tables include the results obtained by applying single-instrument translation and then mixing the outputs.

Clarinet-Vibraphone to Strings-Piano:

Name	Input Clarinet	Input Vibraphone	Target String	Target Piano	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

Strings-Piano to Clarinet-Vibraphone:

Name	Input String	Input Piano	Target Clarinet	Target Vibraphone	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

The following tables include the results obtained by applying the separation-translation pipeline and then mixing the outputs.

Clarinet-Vibraphone to Strings-Piano:

Name	Input Mixture	Isolated Clarinet	Isolated Vibraphone	Target String	Target Piano	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

Strings-Piano to Clarinet-Vibraphone:

Name	Input Mixture	Isolated String	Isolated Piano	Target Clarinet	Target Vibraphone	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

The following tables include the results obtained by applying the embedding-supervised method and then mixing the outputs.

Clarinet-Vibraphone to Strings-Piano:

Name	Input Mixture	Target String	Target Piano	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

Strings-Piano to Clarinet-Vibraphone:

Name	Input Mixture	Target Clarinet	Target Vibraphone	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

The following tables include the results obtained by applying stem-supervised Music-STAR.

Clarinet-Vibraphone to Strings-Piano:

Name	Input Mixture	Target String	Target Piano	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

Strings-Piano to Clarinet-Vibraphone:

Name	Input Mixture	Target Clarinet	Target Vibraphone	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

The following tables include the results obtained by applying mixture-supervised Music-STAR.

Clarinet-Vibraphone to Strings-Piano:

Name	Input Mixture	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

Strings-Piano to Clarinet-Vibraphone:

Name	Input Mixture	Target Mixture	Gold Standard
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Bianche

The following tables include the final results obtained by applying all five methods.

Clarinet-Vibraphone to Strings-Piano:

Name	Single-Instrument	Separation-Translation	Embedding-supervised	Stem-supervised Music-STAR	Mixture-supervised Music-STAR
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

Strings-Piano to Clarinet-Vibraphone:

Name	Single-Instrument	Separation-Translation	Embeddin Music-STAR	Stem-supervised Music-STAR	Mixture-supervised Music-STAR
Pirates of Caribbean Theme
My Heart Will Go on
Beethoven's String
Moonlight Sonata
Fur Elise
Brahms's Clarinet
Beethoven's Piano
Dvorak's String
Romeo and Juliet
Nuvole Blanche

A. Statistical Analysis on Subjective Evaluation

Below you can see the results of subjective evaluation as bar charts. Each method was ranked by the participants from 1 to 5 (1 being the best) in terms of content preservation, style fit, and audio quality.

Content preservation

Style fit

Audio quality

B. Demucs Performance Analysis