We demonstrate that a single 3×3 convolutional kernel can produce emergent audio effects when trained on 200 samples from a personalized corpus. We achieve this through two key techniques: (1) Conditioning Aware Kernels (CAK), where output = input + (learned_pattern × control), with a soft-gate mechanism supporting identity preservation at zero control; and (2) AuGAN (Audit GAN), which reframes adversarial training from "is this real?" to "did you apply the requested value?" Rather than learning to generate or detect forgeries, our networks cooperate to verify control application, discovering unique transformations. The learned kernel exhibits a diagonal structure creating frequency-dependent temporal shifts that are capable of producing musical effects based on input characteristics. Our results show the potential of adversarial training to discover audio transformations from minimal data, enabling new approaches to effect design.

Generative AI has captured the world’s imagination by transforming deep learning from an analytical tool into a creative medium. While GANs and diffusion models enable visual art and music generation, the manipulation of existing audio remains primarily rooted in traditional signal processing. Mathematical DSP has given us powerful audio effects, from convolution reverbs to analog circuit models, but these rely on human insight to translate acoustic phenomena into equations. What if we could learn audio effects directly from sound itself? We bridge this gap with Conditioning Aware Kernels (CAK), a modulation technique that discovers transformations directly from data through adversarial training.

Our approach explores an alternate perspective: neural networks as simplification approximators. By constraining our model to find minimal viable solutions, we investigate whether sophisticated audio transformations can arise from 200 training samples and 11 learnable parameters (a single 3×3 kernel with bias and scale). This framework allows us to study what emerges when model capacity is deliberately limited to match realistic data constraints.

Human perception mirrors this efficiency. We learn to recognize complex patterns from a handful of experiences. Similarly, experienced audio engineers develop intuition for effects through limited but focused interaction. CAK captures this principle computationally: given a small corpus of audio with varying features, our system learns not just to reproduce but to discover and apply learned qualities across unseen inputs.

Traditional GANs (Goodfellow et al., 2014) pit generator against discriminator in a forgery detection contest. We propose an ’audit game’ (AuGAN) where the discriminator must verify that the generator applied the user’s control value to its learned features, whatever those features turn out to be. This structural shift, from deception to verification, enables the discriminator to guide the discovery of audio transformations.

Through CAK, we investigate whether neural transformations require architectural complexity, or whether they can emerge from the interaction between minimal learned operations and the inherent richness of audio signals.

Neural Audio Synthesis and Effects: WaveGAN (Donahue et al., 2018) and GANSynth (Engel et al., 2019) first showed that adversarial training can generate raw waveform audio using large datasets and models. DDSP (Engel et al., 2020) and RAVE (Caillon & Esling, 2021) achieve high quality synthesis through compact architectures and strong inductive biases, enabling efficient training even with limited data. Unlike those works, CAK is not a standard generator; it learns effects from a small, personalized corpus.

Conditioning Mechanisms: Feature-wise Linear Modulation (FiLM; Perez et al., 2018) conditions deep networks via channel wise affine transforms. In a 2019 retrospective, the authors note that FiLM often needs additional task-specific inductive biases to remain data efficient (Perez et al., 2019 retrospective), a limitation we also observed when applying FiLM to complex conditioning vectors. Dynamic kernel methods such as CondConv (Yang et al., 2019) synthesize weights by mixing basis filters. CondConv effectively asks which kernel a layer should use; CAK instead detects a salient pattern and modulates its residual contribution by the user’s control value.

Few-Shot Learning: Audio few-shot work typically relies on meta-learning (MAML, Finn et al., 2017; Prototypical Networks, Snell et al., 2017), which require access to large and diverse meta-training sets composed of many tasks. CAK operates in a more extreme regime: it learns directly from a single, 50-minute corpus without episodic sampling or meta-tasks.

Emergent Complexity from Simple Rules: Growing Isotropic Neural Cellular Automata (Mordvintsev, Randazzo, and Fouts, 2022) demonstrates similar principles in the visual domain, where simple local update rules produce complex emergent patterns. Like CAK, this work shows that behavioral diversity can arise from the interaction between minimal fixed rules and varying initial conditions, rather than from architectural complexity.

Biological Inspiration: Our emphasis on minimal, data-efficient representations echoes the efficient coding hypothesis (Barlow, 1961) and sparse coding results in V1 (Olshausen & Field, 1996) which suggest biological systems seek minimal representations. Feature Integration Theory (Treisman & Gelade, 1980) likewise suggests that selective modulation of simple detectors can explain complex percepts, paralleling CAK’s single-kernel modulation experiment.

Our initial approach followed established conditioning methods, using FiLM (Perez et al., 2018) with 24-dimensional control vectors encoding categorical tags, continuous DSP parameters, and perceptual attributes. This exhibited training instability. FiLM’s affine transformations rely on deep networks to achieve complex modulation, making them potentially unsuitable for rich musical descriptors and small datasets. We do not claim that FiLM is fundamentally limited in this regard, only that we could not find the optimal solution for this problem using conventional modulation methods.

This failure revealed an insight: complex control may not always require complex modulation. Through ablation, we discovered that our initially complex network consistently relied on a small subset of detected patterns, regardless of the control vector’s dimensionality. This suggested a different approach: instead of learning how to modulate based on complex inputs, learn what patterns to detect, then simply scale them.

The audit game naturally enforces this sparsity. The discriminator must verify control values, incentivizing the generator to find distinct patterns. This led to CAK: a single learned detector whose output is scaled by a scalar control value. The shift from 24-dimensional modulation to 1-dimensional scaling did not reduce expressiveness; it revealed that one well-learned pattern could create effects through context-aware application.

Figure 1: Training dynamics comparison between FiLM and CAK architectures. Both networks were trained under identical conditions with no divergence mitigation strategies. FiLM-based conditioning with 24-dimensional control vectors exhibits instability, with discriminator and generator loss exceeding 10⁴ and training failure. In contrast, CAK achieves stable training on the same dataset, enabling the discovery of patterns despite its ultimate convergence to a single dominant feature.

Instead of fighting against the results of the network, we theorized that dimensional collapse might contain a fascinating study of how neural networks process complex domains. We consider music to be infinitely analyzable. A single sample contains many possible interpretations. Collapse is usually treated as a failure to be mitigated through architectural tricks or regularization, and in some cases may be the correct motive, but perhaps we could design architectures that embrace, rather than resist, this tendency toward simplification.

The core CAK operation implements a simple principle:

output = input + (learned_pattern × control)

Formally, this becomes:

y = x + (D(x) × c × σ(c) × s) (1)

where:

• x ∈ ℝ^F×T is the input magnitude spectrogram
• D : ℝ^F×T → ℝ^F×T is a learned 3×3 convolutional detector (same padding, with bias)
• c ∈ ℝ is a per-example control scalar that broadcasts over the F×T dimensions
• σ(c) = sigmoid((c − τ) × temp) is a soft-gate function with:
  • τ ∈ ℝ: threshold (0.3 in our experiments, ensuring control values below 0.3 produce minimal activation)
  • temp ∈ ℝ: temperature parameter (2 → 20 linear ramp during training); lower values create gradual transitions, higher values approach a hard cutoff at τ
• s ∈ ℝ is a learned scale parameter that correlates with effect intensity

During training, each spectrogram is paired with a randomly sampled control value c. This scalar multiplies the detected patterns D(x) element-wise across the entire spectrogram, creating a simple yet effective modulation mechanism. The audit game ensures the network learns to detect patterns whose intensity scales meaningfully with c. We think of c as a continuous modulation knob where higher values produce proportionally stronger effects. While we use random sampling in this work, the framework supports experiments using c to encode possible semantic attributes.

The soft-gate σ(c) provides smooth onset based on the control value, which also directly scales the effect through multiplication. We choose multiplication as it preserves sonic character while scaling intensity, aligning with human auditory perception, which responds to amplitude ratios rather than absolute differences. This is arguably the simplest scaling solution possible. The dual use of c ensures both proportional intensity and gated activation, while the residual path assists with transparency at zero control.

Traditional GANs optimize min_G max_D V(D, G) where the generator G tries to fool the discriminator D. We reformulate this as AuGAN (Audit GAN), where both networks cooperate to verify control application.

Generator Objective: Apply transformations proportional to the control value.

Discriminator Objective: Verify whether the correct control amount was applied.

Crucially, both networks share the same detector D. This prevents the generator from learning arbitrary transformations; any pattern it uses must also help the discriminator verify control values. AuGAN’s cooperative dynamics promote:

1. Distinct Features: Random patterns will not help verification.
2. Proportional Application: The transformation strength must scale consistently with the control value.
3. Smooth Control: The discriminator needs to distinguish nearby control values.

We implement AuGAN using WGAN-GP with additional terms to enforce control compliance. Following WGAN convention, we refer to the discriminator as the Critic C.

Discriminator Loss:

L_C = −𝔼[C(x_real, c)] + 𝔼[C(x_fake, c)] + λ_gp · GP + λ_comp · 𝔼[V(x_fake, c)](2)

Generator Loss:

L_G = −𝔼[C(x_fake, c)] + λ_comp · 𝔼[V(x_fake, c)] + λ_recon · ‖x_fake − x_real‖₁− λ_reg · 𝔼[log(ε + mean_F,T|D(x_in)|)](3)

Pairing strategy: We train on tuples (x_in, x_real, c) of two types: (i) Identity pairs: x_real = x_in, c = 0 anchor the operator to the identity. (ii) Transformation pairs: x_in = x_low, x_real = x_high, c = g(high) − g(low) supply a target change and a control value. The L1 term pulls G(x_in, c) toward x_real while the compliance term enforces m(G(x_in, c)) ≈ c; together with shared D, this rules out the trivial copy solution yet preserves identity at c = 0.

where:

• x_fake = G(x_in, c) is the generator output
• x_real is the paired target from the dataset:
  • For identity pairs: x_real = x_in with c = 0
  • For transformation pairs: x_real is a different sample with c encoding their relationship
• C(·) outputs a realness score and violation V(·, c) for control verification
• V(x, c) = |measured_texture(x) − c|, where measured_texture(x) = mean(D(x)) using the shared detector; we measure texture as a signed mean of D(x), and regularization uses |D(x)| to avoid collapse
• GP is the gradient penalty for the Lipschitz constraint, computed on interpolations between x_real and x_fake with the same control value c
• D(x_in) in the regularization term refers to the shared detector patterns on input
• ε = 10⁻⁸ before the logarithm in the regularizer for numerical stability
• λ_gp = 10.0, λ_comp = 2.0, λ_recon = 5.0, λ_reg = 0.01

We designed our experiments to reflect realistic artistic workflows. Musicians and sound designers typically work with curated personal collections rather than massive datasets. Our setup mirrors this reality:

Dataset: 200 fifteen-second audio segments derived from the author’s musical corpus, representing the scale of material an artist might realistically collect and curate for a specific project. This includes varied timbral content mainly centered around electronic and electroacoustic music composition: synthesized textures, field recordings, and acoustic instrumentation.

Preprocessing: STFT with 2048-point FFT, 512 sample hop, 44.1 kHz sample rate, standard parameters for musical applications allowing the learned kernel to discover patterns directly from minimally processed spectrograms.

Training: 100 epochs on Apple M4 (48GB unified memory), completing in approximately 2 hours. Figure 2 shows stable training without divergence.

Figure 2: Training dynamics of CAK over 100 epochs using 200 15-second samples. Generator and discriminator losses show stable convergence. Increasing Wasserstein distance indicates healthy adversarial learning. Decreasing audit violations demonstrate successful effect control learning. Temperature annealing (orange) sharpens the soft-gate while the scale parameter (brown) adapts to optimal effect strength.

The identity constraint at control value zero is a calibration mechanism, ensuring that our learned transformation maintains ideal magnitude reconstruction when no modulation is desired. This helps prevent neural spectral coloration in bypass mode and forces the network to learn truly residual transformations. Identity preservation was tested on held out, diverse audio sources, with gate activation of 0.0025 on average at zero control and negligible magnitude difference (difference < 10^-9), confirming the soft-gate mechanism contributes to transparent pass-through. Naturally, various inputs will have different responses and we do not claim perfect unity gain at bypass. These results can be tested audibly in our GUI by simply processing a sample at a control value of 0.

The learned 3 × 3 detector kernel reveals how CAK achieves feature learning through a single pattern. Figure 3 shows the learned weights and their interpretable structure.

Figure 3: The learned detector kernel shows the asymmetric pattern that underlies CAK’s behavior. During convolution with STFT magnitude, each weight indicates how strongly that time-frequency relationship contributes to the output. The rightward-biased pattern (high weights at positions [0,2], [0,1], and [2,2]) creates asymmetric time-frequency smoothing, emphasizing future time steps and producing spectral-temporal diffusion across the magnitude representation. The frequency band response (right) shows emergent selectivity, with stronger low-frequency weighting (0.115) despite no explicit frequency conditioning during training. This demonstrates how the CAK framework discovers both spectral and temporal patterns directly from data.

The learned transformation resists simple categorization, which we view as a fundamental characteristic of neural audio processing. Unlike traditional effects with clear design goals, CAK discovers patterns that produce varied perceptual results across different input types. This difficulty in defining the transformation using conventional audio terminology highlights both a limitation and a strength: while we cannot provide a traditional taxonomy, the effect represents a fascinating transformation learned from the data itself. This suggests future possibilities for audio processing that move beyond recreating known effects.

Several directions merit exploration:

Alternative Training Frameworks: Although we trained CAK using adversarial dynamics, the architecture itself may be training agnostic. Investigating CAK within VAE frameworks or through direct supervised learning could reveal different emergent behaviors and potentially simpler training procedures.

Semantic Control: Our current approach learns effects from data without semantic labels. Incorporating dilated convolutions or attention mechanisms may enable targeting specific perceptual qualities (e.g., “brightness,” “warmth”) while maintaining our minimal parameter philosophy.

Architectural Extensions: Stacking multiple CAK layers with varying kernel sizes may capture multi-scale patterns. Additionally, frequency-band-specific CAK modules may enable surgical audio manipulation by applying different learned transformations to isolated spectral regions and recombining them for complex, structured effects. We are currently focusing further research efforts in this direction.

Cross-Domain Applications: The principle of learning minimal patterns that interact with input characteristics may extend beyond audio. Investigating CAK on image or video data could validate whether this emergence phenomenon generalizes across modalities.