Research · Song-Ze (Jimmy) Yu

Public work and ongoing research, newest first.

2026 · Preprint BAIR

PitchBench hierarchy from atomic to contextual and melodic pitch perception — Click to enlarge

PitchBench: Measuring Pitch Hearing in Audio-Language Models

Milan Liessens Dujardin^*, Song-Ze Yu^*, Craver Corbyn Thomas-Smith, David M. Chan, Karina Nguyen ^* Equal contribution

ALMs are moving quickly toward high-level music understanding and captioning, yet lacks a systematic, hierarchical benchmark for the perceptual abilities those applications assume. PitchBench addresses this gap with 28 experiments that disentangle what a model perceives from when it perceives it, progressing from I. atomic pitch to II. contextual, time-grounded perception and III. melodic pitch perceptions. Together, these levels test the absolute and relative pitch hearing needed to reason about high-level intervals, melodies, and harmony.

Paper Dataset Code

2026 · DAFx CNMAT

InstructFX2FX architecture and multi-turn effect-chain refinement — Click to enlarge

InstructFX2FX: A Multi-Turn Text-to-Effect System for Sequential Audio Effect Refinement

Song-Ze Yu, Milan Liessens Dujardin, Yuxuan Cai, Wantong Zhang, Brian Cruz, Jeremy Wagner, Carmine-Emanuele Cella

Introduces sequential FX refinement: given the current effect state and a new instruction, update the sound while preserving what earlier instructions have already achieved. InstructFX2FX combines LLM planning with CLAP-guided optimization for iterative, multi-turn audio effect design.

Paper Demo Code

2026 · Under review CNMAT

MuNo-LM compared with score image, ABC, MIDI, and MusicXML representations — Click to enlarge

MuNo-LM: Unifying Score and Performance for Audio-Language Modeling

Milan Liessens Dujardin, Song-Ze Yu

Fine-grained music understanding requires models to represent both what is played and how it is performed, grounded in time. MuNo-LM (Music Notation for Language Models) replaces coarse, weakly time-grounded synthesized captions with an LLM-readable notation that unifies score structure and time-aligned perceptual features, then generates musically informed, time-grounded captions and question-answer pairs.

Demo

2026 · Under review BAIR

Anamnesis pipeline connecting LLM-generated personas, virtual opinions, and human studies — Click to enlarge

Anamnesis: An Open-Source Platform for Large-Scale Backstory-Conditioned Survey Simulation

Song-Ze Yu, Joseph Suh, Serina Chang, David M. Chan

Anamnesis is an interactive system for demographically controllable survey simulation using large language models. Researchers can sample virtual respondents from a validated pool of 30K+ richly detailed narrative backstories pool to conduct multiple choice questions, open-ended or multimodal surveys, yielding opinion distributions that more closely match real-world data than standard persona prompting or LLM-as-a-judge baseline.

Paper Demo

2025 · Preprint AHG Music Lab

VTR system design from audio feature extraction to EQ parameter prediction and JUCE processing — Click to enlarge

From Sound to Setting: AI-Based EQ Parameter Prediction for Piano Tone Replication

Song-Ze Yu, Yi-Wen Liu

A supervised model predicts multi-band EQ settings from reference-audio DSP features. The work became VTR-SmartEQ, a JUCE plugin for real-time tone replication.

Paper Code Plugin

Research

My research taste.

Music AI has never looked more promising nowadays: companies like Suno, Udio, and Ace Studio show how generative models may reshape music itself. However, I'm deeply concerned that much of this field is moving toward replacing the humanity, intention, and creativity from the soul of a musician.

I tried to reflect what was missing and how to leverage the advantage of this exploding generative AI era. My research starts from three missing pieces:

Controllability.

Many generative models can produce impressive zero-shot results, but offer musicians little control over how an output should evolve. Music making is rarely a one-shot process: musicians compose, listen, revise, and negotiate intent through a sequence of small decisions. I am interested in systems that support this process, from controllable music generation to iterative audio effect design, while preserving what the musician has already built.

Music understanding.

Audio-language models are moving quickly toward high-level tasks such as captioning and dialogue, often before we know whether they can reliably hear basic musical structure. I am interested in building and evaluating music understanding from the ground up: from pitch and rhythm, through temporal and harmonic structure, to performance expression and musical intention. Before an ALM can reason or communicate about music, it should first demonstrate what it hears and when it hears it.

Humanity.

Music is the most abstract language we have. Its meaning is not fully specified by the notes or audio signal, and the same musical phrase can be understood differently by composers, performers, and listeners. I am interested in whether ALMs can model these differences and align with human preferences without collapsing them into a single ground truth.

Ultimately, this is also an HCI question: can a model become a medium through which musicians and audiences discuss, shape, and experience music together?

Long-term direction

Together, these questions shape my long-term research direction: building a unified audio-language model that can truly hear, understand, and communicate about music, enabling new forms of HCI between musicians, computers, and audiences.

Research by Song-Ze (Jimmy) Yu