sync_audio_score_full.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Demo: Audio-Score synchronization with high-resolution features and MrMsDTW\n",
    "\n",
    "In this notebook, we'll show a full music synchronization pipeline using the SyncToolbox, including feature extraction and high-resolution synchronization.\n",
    "\n",
    "We will take a recording of a musical piece and its .csv pitch annotation, created from a MIDI file, compute their feature representations, align them using multi-resolution multi-scale DTW (MrMsDTW), and show how to sonify the alignment and use it for automated transfer of annotations.\n",
    "\n",
    "The pipeline in this notebook exactly reproduces the techniques described in [1], which in turn is based on [2]. On the finest synchronization, we use the high-resolution features described in [3]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Loading some modules and defining some constants used later\n",
    "import IPython.display as ipd\n",
    "from libfmp.b import list_to_pitch_activations, plot_chromagram, plot_signal, plot_matrix, \\\n",
    "                     sonify_pitch_activations_with_signal\n",
    "import librosa.display\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.interpolate\n",
    "\n",
    "from synctoolbox.dtw.mrmsdtw import sync_via_mrmsdtw\n",
    "from synctoolbox.dtw.utils import compute_optimal_chroma_shift, shift_chroma_vectors, make_path_strictly_monotonic\n",
    "from synctoolbox.feature.csv_tools import read_csv_to_df, df_to_pitch_features, df_to_pitch_onset_features\n",
    "from synctoolbox.feature.chroma import pitch_to_chroma, quantize_chroma, quantized_chroma_to_CENS\n",
    "from synctoolbox.feature.dlnco import pitch_onset_features_to_DLNCO\n",
    "from synctoolbox.feature.pitch import audio_to_pitch_features\n",
    "from synctoolbox.feature.pitch_onset import audio_to_pitch_onset_features\n",
    "from synctoolbox.feature.utils import estimate_tuning\n",
    "%matplotlib inline\n",
    "\n",
    "Fs = 22050\n",
    "feature_rate = 50\n",
    "step_weights = np.array([1.5, 1.5, 2.0])\n",
    "threshold_rec = 10 ** 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the recording\n",
    "\n",
    "Here, we take an interpretation of the first 8 measures of the Etude Op.10 No.3 in E major by Frederic Chopin, played by Valentina Igoshina ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "audio, _ = librosa.load('data_music/Chopin_Op010-03-Measures1-8_Igoshina.wav', sr=Fs)\n",
    "\n",
    "plot_signal(x=audio, Fs=Fs, figsize=(9,3))\n",
    "plt.title('Etude Op.10 No.3 in E Major by Frederic Chopin\\n Performer: Valentina Igoshina')\n",
    "ipd.display(ipd.Audio(audio, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the .csv annotation file, created from a MIDI file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "df_annotation = read_csv_to_df('data_csv/Chopin_Op010-03-Measures1-8_MIDI.csv', csv_delimiter=';')\n",
    "html = df_annotation.to_html(index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating tuning\n",
    "\n",
    "We use a simple comb-based algorithm to detect the tuning deviation in the audio recording. This will be used to adjust the filterbanks for feature computation. If we do not adjust for tuning, our chroma representation may look \"smeared\", leading to bad synchronization results. We refer to <a href=\"https://www.audiolabs-erlangen.de/resources/MIR/FMP/C3/C3S1_TranspositionTuning.html\">the FMP notebook on Transposition and Tuning</a> for more information on tuning issues and the algorithm used for tuning estimation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import libfmp.c2\n",
    "# Alternative: librosa.estimate_tuning\n",
    "tuning_offset = estimate_tuning(audio, Fs)\n",
    "print('Estimated tuning deviation for recording: %d cents' % (tuning_offset))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing quantized chroma and DLNCO features\n",
    "\n",
    "We now compute the feature representations used in the alignment procedure. Note that we include the 'tuning_offset' calculated in the previous step. In our pipeline, we use CENS features, which are similar to standard chroma but first quantized, then smoothed, downsampled and normalized. The MrMsDTW procedure just requires the quantized chromas, since smoothing, downsampling and normalization happens internally.\n",
    "\n",
    "In addition to these chroma-like features, we also use special onset-related features called DLNCO (described in [3]). These are helpful to increase synchronization accuracy, especially for music with clear onsets.\n",
    "\n",
    "Both features are computed from the audio using a multi-rate IIR filterbank. See [4] for details.\n",
    "\n",
    "In the next cell, we also display the computation steps leading to both features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def get_features_from_audio(audio, tuning_offset, Fs, feature_rate, visualize=True):\n",
    "    f_pitch = audio_to_pitch_features(f_audio=audio, Fs=Fs, tuning_offset=tuning_offset, feature_rate=feature_rate, verbose=visualize)\n",
    "    f_chroma = pitch_to_chroma(f_pitch=f_pitch)\n",
    "    f_chroma_quantized = quantize_chroma(f_chroma=f_chroma)\n",
    "    if visualize:\n",
    "        plot_chromagram(f_chroma_quantized, title='Quantized chroma features - Audio', Fs=feature_rate, figsize=(9,3))\n",
    "\n",
    "    f_pitch_onset = audio_to_pitch_onset_features(f_audio=audio, Fs=Fs, tuning_offset=tuning_offset, verbose=visualize)\n",
    "    f_DLNCO = pitch_onset_features_to_DLNCO(f_peaks=f_pitch_onset, feature_rate=feature_rate, feature_sequence_length=f_chroma_quantized.shape[1], visualize=visualize)\n",
    "    return f_chroma_quantized, f_DLNCO\n",
    "\n",
    "\n",
    "f_chroma_quantized_audio, f_DLNCO_audio = get_features_from_audio(audio, tuning_offset, Fs, feature_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_features_from_annotation(df_annotation, feature_rate, visualize=True):\n",
    "    f_pitch = df_to_pitch_features(df_annotation, feature_rate=feature_rate)\n",
    "    f_chroma = pitch_to_chroma(f_pitch=f_pitch)\n",
    "    f_chroma_quantized = quantize_chroma(f_chroma=f_chroma)\n",
    "    if visualize:\n",
    "        plot_chromagram(f_chroma_quantized, title='Quantized chroma features - Annotation', Fs=feature_rate, figsize=(9, 3))\n",
    "    f_pitch_onset = df_to_pitch_onset_features(df_annotation)\n",
    "    f_DLNCO = pitch_onset_features_to_DLNCO(f_peaks=f_pitch_onset,\n",
    "                                            feature_rate=feature_rate,\n",
    "                                            feature_sequence_length=f_chroma_quantized.shape[1],\n",
    "                                            visualize=visualize)\n",
    "    \n",
    "    return f_chroma_quantized, f_DLNCO\n",
    "\n",
    "\n",
    "f_chroma_quantized_annotation, f_DLNCO_annotation = get_features_from_annotation(df_annotation, feature_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding optimal shift of chroma vectors\n",
    "\n",
    "The interpretation might be played in a different key than the one in the original score. This can also be seen in the chroma representations above and will lead to complete degradation of the alignment if this effect is not accounted for. The SyncToolbox provides a built-in function for finding the shift between two recordings. This is done in the following cell and the feature sequences are subsequently adjusted to account for this shift. The plots show the chroma sequences after shifting.\n",
    "\n",
    "Internally, the function just performs DTW using all possible shifts and returns the shift yielding the lowest total cost. To save computation time, we here first downsample the sequences.\n",
    "\n",
    "NOTE: The chroma shift doesn't apply in this running example, since both the score and recording are in E major."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "f_cens_1hz_audio = quantized_chroma_to_CENS(f_chroma_quantized_audio, 201, 50, feature_rate)[0]\n",
    "f_cens_1hz_annotation = quantized_chroma_to_CENS(f_chroma_quantized_annotation, 201, 50, feature_rate)[0]\n",
    "opt_chroma_shift = compute_optimal_chroma_shift(f_cens_1hz_audio, f_cens_1hz_annotation)\n",
    "print('Pitch shift between the audio recording and score, determined by DTW:', opt_chroma_shift, 'bins')\n",
    "\n",
    "f_chroma_quantized_annotation = shift_chroma_vectors(f_chroma_quantized_annotation, opt_chroma_shift)\n",
    "f_DLNCO_annotation = shift_chroma_vectors(f_DLNCO_annotation, opt_chroma_shift)\n",
    "\n",
    "_,_,_= plot_chromagram(f_chroma_quantized_audio[:, :30 * feature_rate], Fs=feature_rate, title='Chroma representation for the audio', figsize=(9, 3))\n",
    "_,_,_= plot_chromagram(f_chroma_quantized_annotation[:, :30 * feature_rate], Fs=feature_rate, title='Chroma representation for the score', figsize=(9, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performing MrMsDTW\n",
    "\n",
    "We now perform alignment using MrMsDTW. The extracted chroma sequences are used on the coarser levels of the procedure, while the DLNCO features are additionally used on the finest level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "wp = sync_via_mrmsdtw(f_chroma1=f_chroma_quantized_audio, \n",
    "                      f_onset1=f_DLNCO_audio, \n",
    "                      f_chroma2=f_chroma_quantized_annotation, \n",
    "                      f_onset2=f_DLNCO_annotation, \n",
    "                      input_feature_rate=feature_rate, \n",
    "                      step_weights=step_weights, \n",
    "                      threshold_rec=threshold_rec, \n",
    "                      verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For applications: Make warping path strictly monotonic\n",
    "The standard step sizes used in DTW allow for horizontal and vertical steps, which leads to warping paths that are not guaranteed to be strictly monotonous. This is usually not a problem. However, for applications such as transferring annotations, it may be better to use a strictly monotonous path and employ linear interpolation inside non-monotonous segments. See also <a href=\"https://www.audiolabs-erlangen.de/resources/MIR/FMP/C3/C3S3_MusicAppTempoCurve.html\">the FMP notebook on Tempo Curves</a> for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Length of warping path obtained from MrMsDTW:', wp.shape[1])\n",
    "wp = make_path_strictly_monotonic(wp)\n",
    "print('Length of warping path made strictly monotonic:', wp.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sonifying warping path\n",
    "\n",
    "In order to listen to the synchronization result, the synthesized score will now be time-scaled (according to the computed warping path) to run synchronous to the audio recording. The result is sonified by putting the audio recording into the left channel and the warped, synthesized score into the right channel of a stereo audio file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_annotation_warped = df_annotation.copy(deep=True)\n",
    "df_annotation_warped[\"end\"] = df_annotation_warped[\"start\"] + df_annotation_warped[\"duration\"]\n",
    "df_annotation_warped[['start', 'end']] = scipy.interpolate.interp1d(wp[1] / feature_rate, \n",
    "                           wp[0] / feature_rate, kind='linear', fill_value=\"extrapolate\")(df_annotation[['start', 'end']])\n",
    "df_annotation_warped[\"duration\"] = df_annotation_warped[\"end\"] - df_annotation_warped[\"start\"]\n",
    "note_list = df_annotation_warped[['start', 'duration', 'pitch', 'velocity']].values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H = 512\n",
    "num_frames = int(len(audio) / H)\n",
    "Fs_frame = Fs / H\n",
    "X_ann, F_coef_MIDI = list_to_pitch_activations(note_list, num_frames, Fs_frame)\n",
    "title = 'Piano-roll representation (Fs_frame = %.3f) of the synchronized annotation' % Fs_frame\n",
    "plot_matrix(X_ann, Fs=Fs_frame, F_coef=F_coef_MIDI,  ylabel='MIDI pitch number', title=title, figsize=(9, 4))\n",
    "plt.ylim([36, 78])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sonification\n",
    "harmonics = [1, 1/2, 1/3, 1/4, 1/5]\n",
    "fading_msec = 0.5\n",
    "x_pitch_ann, x_pitch_ann_stereo = sonify_pitch_activations_with_signal(X_ann, audio, Fs_frame, Fs,\n",
    "                                                                       fading_msec=fading_msec, \n",
    "                                                                       harmonics_weights=harmonics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO This sonification procedure is very slow for long recordings and will be improved in a future version of synctoolbox.\n",
    "x_peaks = np.zeros(len(audio))\n",
    "for row in note_list:\n",
    "    second, duration, pitch, velocity = row\n",
    "    freq = 2 ** ((pitch - 69) / 12) * 440\n",
    "    for harmonic_num, harmonic_weight in enumerate(harmonics):\n",
    "        x_peaks += velocity / 128 * harmonic_weight * librosa.clicks(times=second, \n",
    "                                                                     sr=Fs,\n",
    "                                                                     click_freq=(harmonic_num+1)*freq,\n",
    "                                                                     length=len(audio),\n",
    "                                                                     click_duration=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Sonification with colored clicks (mono):')\n",
    "ipd.display(ipd.Audio(x_peaks, rate=Fs))\n",
    "\n",
    "print('Sonification with colored clicks and sinusoids (mono):')\n",
    "ipd.display(ipd.Audio(x_peaks + x_pitch_ann, rate=Fs))\n",
    "\n",
    "print('Sonification of colored clicks and original audio (stereo):')\n",
    "ipd.display(ipd.Audio(np.array([audio, x_peaks]), rate=Fs))\n",
    "\n",
    "print('Sonification of colored clicks with sinusoids and original audio (stereo):')\n",
    "ipd.display(ipd.Audio(np.array([audio, x_peaks + x_pitch_ann]), rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "[1] Thomas Prätzlich, Jonathan Driedger, and Meinard Müller: Memory-Restricted Multiscale Dynamic Time Warping,\n",
    "In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 569–573, 2016.\n",
    "\n",
    "[2] Meinard Müller, Henning Mattes, and Frank Kurth:\n",
    "An Efficient Multiscale Approach to Audio Synchronization,\n",
    "In Proceedings of the International Conference on Music Information Retrieval (ISMIR): 192–197, 2006.\n",
    "\n",
    "[3] Sebastian Ewert, Meinard Müller, and Peter Grosche:\n",
    "High Resolution Audio Synchronization Using Chroma Onset Features,\n",
    "In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 1869–1872, 2009.\n",
    "\n",
    "[4] Meinard Müller: Information Retrieval for Music and Motion, ISBN: 978-3-540-74047-6, Springer, 2007."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}