There are many options for augmentation we can apply to this competition some on the spectral level while others on the signal itself. In this post I try to go over the most promising ones:
\nUsed the following augmentations (Implementations - My own. No code released for this solution)
\nTime Masking
\n\n\nTL;DR: Masking the time of the input signal by a certain amount. This can be done by multiplying the time domain representation of the audio signal by a masking function.
\n
(): \n nn.Sequential(torchaudio.transforms.TimeMasking(time_mask_param = time_mask_param),)(torch_img)\nFrequency Masking:
\n\n\nTL;DR: Masking the frequency of the input audio signal by a certain amount. This can be done by multiplying the Fourier transform of the audio signal by a masking function.
\n
():\n nn.Sequential(torchaudio.transforms.FrequencyMasking(freq_mask_param = freq_mask_param),)(torch_img)\n\n\nThe first place writeup also used minor time shifts between channels
\n
Combined Time And Freq: SpecAug (Not used by the 1st place solution)
\n\n\n**Paper:SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. Park, et al.
\n
():\n nn.Sequential(torchaudio.transforms.FrequencyMasking(freq_mask_param = freq_mask_param),\n torchaudio.transforms.TimeMasking(time_mask_param = time_mask_param))(torch_img)\n\n\nSource: code release
\n
Used 2 augmentations.
Gaussian noise for 2d-CNN networks
():\n ():\n ().__init__(always_apply, p)\n self.min_snr = min_snr\n self.max_snr = max_snr\n\n ():\n snr = np.random.uniform(self.min_snr, self.max_snr)\n white_noise = np.random.randn((y))\n add_noise_snr(y, white_noise, snr)\n ():\n (): ().__init__(always_apply, p)\n (): y * - \nVertical Shifting
\n\n\nTL;DR: Shifting the image vertically by a certain amount.
\n
():\n img[] = np.roll(img[], vshift, axis = )\n img[] = np.roll(img[], vshift, axis = )\n img\nColored Noise Augmentation
\nColored noise augmentation was done channel-wise (No further information was provided).
Random Phase Augmentation (Implementation: My own. Might be inaccurate)
\n\n\nTL;DR: Randomly adding and subtracting phase to the spectral representation of the audio signal (Fourier basis and ifft).
\n
():\n rand_phase = np.exp( * np.random.uniform(, * np.pi, Z.shape))\n np.(Z) * rand_phase\nZebra Mixup
\nThey report that this augmentation works really well.
\nwave_new[:, ::] = wave1[:, ::]\nwave_new[:, ::] = wave2[:, ::]\nNegative Flip
\nThey report that this worked for only negative sample
wave = wave[:, ::-].copy()\nSwap with Other Negative
\nNegative samples swapped with other negative samples
\n np.random.uniform() > : wave1[] = wave2[]\n np.random.uniform() > : wave1[] = wave2[]\n np.random.uniform() > : wave1[] = wave2[]\nLIGO Swap
\nSince both logos are completely the same - we can swap them.
\nw0 = wave1[].copy()\nwave1[] = wave1[]\nwave1[] = w0\nRandom shift each channel
\n\n\nTL;DR: Randomly shifting each channel in the time domain representation of the audio signal by a certain amount.
\n
():\n\n shift = np.random.randint(-max_shift, max_shift, )\n\n img[][:, ] = np.roll(img[][:, ], shift[], axis = )\n img[][:, ] = np.roll(img[][:, ], shift[], axis = )\n img[][:, ] = np.roll(img[][:, ], shift[], axis = )\n img[][:, ] = np.roll(img[][:, ], shift[], axis = )\n img\nRandom turning off of a single channel
\n\n\nTL;DR: Randomly turning off a channel of the audio signal.
\n
():\n channel = np.random.randint()\n img[][:, channel] = np.zeros(img[].shape[])\n img[][:, channel] = np.zeros(img[].shape[])\n img\nRandom swapping channel of the two waves (target=0 only)
\n\n\nTL;DR: Randomly swapping channels of the audio signal.
\n
():\n channel = np.random.randint()\n img_swap = img[][:, channel].copy()\n img[][:, channel] = img[][:, channel]\n img[][:, channel] = img_swap\n img\nMixUp
\n\n\nPaper: Mixup: Beyond Empirical Risk Minimization
\nTL;DR: Mixing the signals together.
\n
():\n lam = np.random.beta(alpha, alpha)\n img = lam * img1 + ( - lam) * img2\n img\nRandomly rolling time shifts
\n\n\nTL;DR: Shifting the signal by a random amount.
\n
():\n\n time_shift = np.random.randint(-time_shift, time_shift, )[]\n\n img[] = np.roll(img[], time_shift, axis = )\n img[] = np.roll(img[], time_shift, axis = )\n img\nThe followings are more augmentation functions that based on the above, also might work well on this competition.
\nHorizontal Flipping
\n\n\nTL;DR: Horizontal flipping of the image.
\n
():\n img[] = np.fliplr(img[])\n img[] = np.fliplr(img[])\n img\nVertical Flipping
\n\n\nTL;DR: Vertical flipping of the image.
\n
():\n img[] = np.flipud(img[])\n img[] = np.flipud(img[])\n img\nTime stretching
\n\n\nTL;DR: Time stretching of the input audio signal by a certain amount. This can be done by interpolating the audio signal.
\n
():\n nn.Sequential(torchaudio.transforms.TimeStretch(rate=),)(torch_img)\nPitch shifting
\n\n\nTL;DR: Shifting the pitch of the input audio signal by a certain amount. This can be done by interpolating the audio signal.
\n
():\n nn.Sequential(torchaudio.transforms.PitchShift(n_steps=),)(torch_img)\nNoise injection
\n\n\nTL;DR: Add noise to the signal by a certain amount in relation to SNR.
\n
$$SNR = \\frac{P_{signal}}{P_{noise}}$$
\nNoise can come in different flavors depending on what we are modeling, a good start is Additive White Gaussian Noise (AWGN). To model AWGN we need to add a zero-mean gaussian random variable to our original signal. The variance of that random variable will affect the average noise power.
\nFor a Gaussian random variable X, the average power E[X^2], also known as the second moment:
\n$$E[X^2] = \\sigma_x^2 + \\mu_x^2$$
\n\n ():\n signal_power = (torch_img**).mean()\n noise_power = signal_power/(**(snr/))\n noise = (torch.randn(torch_img.size())*(noise_power**)).(torch.cuda.FloatTensor)\n (torch_img + noise)\nThere are many options for augmentation we can apply to this competition some on the spectral level while others on the signal itself. In this post I try to go over the most promising ones:
\nUsed the following augmentations (Implementations - My own. No code released for this solution)
\nTime Masking
\n\n\nTL;DR: Masking the time of the input signal by a certain amount. This can be done by multiplying the time domain representation of the audio signal by a masking function.
\n
(): \n nn.Sequential(torchaudio.transforms.TimeMasking(time_mask_param = time_mask_param),)(torch_img)\nFrequency Masking:
\n\n\nTL;DR: Masking the frequency of the input audio signal by a certain amount. This can be done by multiplying the Fourier transform of the audio signal by a masking function.
\n
():\n nn.Sequential(torchaudio.transforms.FrequencyMasking(freq_mask_param = freq_mask_param),)(torch_img)\n\n\nThe first place writeup also used minor time shifts between channels
\n
Combined Time And Freq: SpecAug (Not used by the 1st place solution)
\n\n\n**Paper:SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. Park, et al.
\n
():\n nn.Sequential(torchaudio.transforms.FrequencyMasking(freq_mask_param = freq_mask_param),\n torchaudio.transforms.TimeMasking(time_mask_param = time_mask_param))(torch_img)\n\n\nSource: code release
\n
Used 2 augmentations.
Gaussian noise for 2d-CNN networks
():\n ():\n ().__init__(always_apply, p)\n self.min_snr = min_snr\n self.max_snr = max_snr\n\n ():\n snr = np.random.uniform(self.min_snr, self.max_snr)\n white_noise = np.random.randn((y))\n add_noise_snr(y, white_noise, snr)\n ():\n (): ().__init__(always_apply, p)\n (): y * - \nVertical Shifting
\n\n\nTL;DR: Shifting the image vertically by a certain amount.
\n
():\n img[] = np.roll(img[], vshift, axis = )\n img[] = np.roll(img[], vshift, axis = )\n img\nColored Noise Augmentation
\nColored noise augmentation was done channel-wise (No further information was provided).
Random Phase Augmentation (Implementation: My own. Might be inaccurate)
\n\n\nTL;DR: Randomly adding and subtracting phase to the spectral representation of the audio signal (Fourier basis and ifft).
\n
():\n rand_phase = np.exp( * np.random.uniform(, * np.pi, Z.shape))\n np.(Z) * rand_phase\nZebra Mixup
\nThey report that this augmentation works really well.
\nwave_new[:, ::] = wave1[:, ::]\nwave_new[:, ::] = wave2[:, ::]\nNegative Flip
\nThey report that this worked for only negative sample
wave = wave[:, ::-].copy()\nSwap with Other Negative
\nNegative samples swapped with other negative samples
\n np.random.uniform() > : wave1[] = wave2[]\n np.random.uniform() > : wave1[] = wave2[]\n np.random.uniform() > : wave1[] = wave2[]\nLIGO Swap
\nSince both logos are completely the same - we can swap them.
\nw0 = wave1[].copy()\nwave1[] = wave1[]\nwave1[] = w0\nRandom shift each channel
\n\n\nTL;DR: Randomly shifting each channel in the time domain representation of the audio signal by a certain amount.
\n
():\n\n shift = np.random.randint(-max_shift, max_shift, )\n\n img[][:, ] = np.roll(img[][:, ], shift[], axis = )\n img[][:, ] = np.roll(img[][:, ], shift[], axis = )\n img[][:, ] = np.roll(img[][:, ], shift[], axis = )\n img[][:, ] = np.roll(img[][:, ], shift[], axis = )\n img\nRandom turning off of a single channel
\n\n\nTL;DR: Randomly turning off a channel of the audio signal.
\n
():\n channel = np.random.randint()\n img[][:, channel] = np.zeros(img[].shape[])\n img[][:, channel] = np.zeros(img[].shape[])\n img\nRandom swapping channel of the two waves (target=0 only)
\n\n\nTL;DR: Randomly swapping channels of the audio signal.
\n
():\n channel = np.random.randint()\n img_swap = img[][:, channel].copy()\n img[][:, channel] = img[][:, channel]\n img[][:, channel] = img_swap\n img\nMixUp
\n\n\nPaper: Mixup: Beyond Empirical Risk Minimization
\nTL;DR: Mixing the signals together.
\n
():\n lam = np.random.beta(alpha, alpha)\n img = lam * img1 + ( - lam) * img2\n img\nRandomly rolling time shifts
\n\n\nTL;DR: Shifting the signal by a random amount.
\n
():\n\n time_shift = np.random.randint(-time_shift, time_shift, )[]\n\n img[] = np.roll(img[], time_shift, axis = )\n img[] = np.roll(img[], time_shift, axis = )\n img\nThe followings are more augmentation functions that based on the above, also might work well on this competition.
\nHorizontal Flipping
\n\n\nTL;DR: Horizontal flipping of the image.
\n
():\n img[] = np.fliplr(img[])\n img[] = np.fliplr(img[])\n img\nVertical Flipping
\n\n\nTL;DR: Vertical flipping of the image.
\n
():\n img[] = np.flipud(img[])\n img[] = np.flipud(img[])\n img\nTime stretching
\n\n\nTL;DR: Time stretching of the input audio signal by a certain amount. This can be done by interpolating the audio signal.
\n
():\n nn.Sequential(torchaudio.transforms.TimeStretch(rate=),)(torch_img)\nPitch shifting
\n\n\nTL;DR: Shifting the pitch of the input audio signal by a certain amount. This can be done by interpolating the audio signal.
\n
():\n nn.Sequential(torchaudio.transforms.PitchShift(n_steps=),)(torch_img)\nNoise injection
\n\n\nTL;DR: Add noise to the signal by a certain amount in relation to SNR.
\n
$$SNR = \\frac{P_{signal}}{P_{noise}}$$
\nNoise can come in different flavors depending on what we are modeling, a good start is Additive White Gaussian Noise (AWGN). To model AWGN we need to add a zero-mean gaussian random variable to our original signal. The variance of that random variable will affect the average noise power.
\nFor a Gaussian random variable X, the average power E[X^2], also known as the second moment:
\n$$E[X^2] = \\sigma_x^2 + \\mu_x^2$$
\n\n ():\n signal_power = (torch_img**).mean()\n noise_power = signal_power/(**(snr/))\n noise = (torch.randn(torch_img.size())*(noise_power**)).(torch.cuda.FloatTensor)\n (torch_img + noise)\n