JamSketch: Improvisation Support System with GA-based Melody Creation from User’s Drawing

Feel free to take & distribute photos! JamSketch: Improvisation Support System with GA-based Melody Creation from User’s Drawing Tetsuro Kitahara*, Sergio Giraldo**, and Rafael Ramirez** *Nihon University, Japan **Universitat Pompeu Fabra, Spain Twitter: @tetsurokitahara

Our goal Enable non-musicians to enjoy improvisation Why it’s difficult Players have to create melodies and play an instrument at the same time Technical issues Human Interface Non-musician-friendly, easy-to-input, intuitive & quickly inputable Melody generation Automatic melody generation based on the user’s abstract, incomplete input in real time

Our idea: JamSketch Human Interface User draws a melodic outline on the pianoroll Music generation Once the melodic outline is drawn, a melody is immediately generated with a genetic algorithm

System overview During the playback of the accompaniment 0th bar 1st bar 2nd bar time User Draws a melodic outline at least one bar in advance If the mouse cursor moves to the next bar or the mouse button is released, Determine the rhythm of the melody Determine the pitch of each note Determine expression parameters Play the melody

Determine the rhythm Basic idea High gradient → short notes ● Low gradient → long notes ● 3 Details of algorithm 0 Candidates 1 Tentative rhythm 3 3 3 3 3 3 3 Threshold the gradient 2 Best-fit rhythm Search the closest one from 0 3

Determine the pitches Basic idea Search a sequence of notes satisfying both closeness to and musical the outline likeliness C using a genetic algorithm (GA) 3 A F Details of formulation Chromosome N = (n0, n1, …, nL-1) (ni: note number) Fitness function Similarity to outline Bigram P(ni | ni-1) Delta bigram P(ni-ni-1 | ni-1-ni-2) F(N) = w0 sim(N) + w1 seq1(N) + w2 seq2(N) + w3 harm(N) + w4 ent(N) P(notes | chord) Entropy

Determine expression parameters Use a model presented at [Giraldo & Ramirez 2016] C Feature extraction for each note (Duration, onset, prev. duration, next duration, prev. interval, next interval, namour, chord, etc.) k-NN, multilayer perceptron, ... onset deviation duration ratio energy ratio

Implementation Main program JamSketch CrestMuse Toolkit of Weka Estimation expression parameters Apache Commons Math GA Feature extraction Groovy Java MIDI Toolbox Java Octave Octave Training data 53 melodies with the Tonality-type of Blues from Weimar Jazz Database

Experiment Purpose Confirm the quality of generated melodies Method 1. Participants play improvisation with our system and a normal MIDI keyboard 2. The performances with our system and the keyboard are listed without distinction 3. Other participants evaluate them If there are no clear differences between evaluations for our system and the keyboard, our system can be considered to have human-like composition capability

Participants & performances Participants Performances Experience Play Impr. Use KB Syst. P1 Yes Yes Yes No P2 Yes No Yes P3 Yes No P4 No No P5 P6 Yes Yes No Yes ? System s2 ? System s1 ? KB (P2) u2 Yes ? KB (P1) u1 No Yes ? System s4 No Yes ? KB (P5) u3 ? System s3 ? KB (P6) u4 Yes Yes Yes No STOP

Listening test ● ● We asked 12 researchers / students in musical fields to evaluate them on a scale of 0 to 10 Criteria – Overall quality – Blues-likeness – Pitch appropriateness – Skillfulness – Rhythmic appropriateness – Human-likeness – Melodic variation ● http://www.kthrlab.jp/proj/jamsketch/listen/

• http://www.kthrlab.jp/proj/jamsketch/listen/

Result: Overall quality {u4} > {u1, u2} > {s1, s2, s3, s4} ≈ {u3}

Result: Pitch appropriateness {u1, u2, u4} > {s1, s2, s3, s4} > {u3}

Result: Rhythmic appropriateness {u2, u4} > {u1, u3} > {s1, s2, s3, s4}

Result: Melodic variation {u4} >> {s1, s2, s4} ≈ {u1, u2, u3} > {s3} ≈ {u3}

Result: Blues-likeness {u4} > {u2} > {s1, s2, s3, s4} ≈ {u1} >> {u3}

Result: Skillfulness {u4} >> {u2} > {s1, s2, s3, s4} ≈ {u1, u3}

Result: Human-likeness {u4} > {u1, u2} > {u3} >> {s1, s2, s3, s4}

Discussions ● Our system is inferior to experienced players ● But it is superior (or close) to inexperienced players ● Possible reasons: – Out-of-scale pitches appear at the ends of phrases ● The system does not consider bounradies of phrases – Out-of-scale pitches appear at long notes ● Rhythm and pitches are independently determined – Rhythm is too monotonous ● ● Rhythm needs more variations We obtained positive opinions from participants

Conclusion ● JamSketch: – enables users to improvise by drawing melodic outlines – generates melodies from the outlines using a GA – is inferior to experienced players – but is superior (or close) to inexperienced players ● Future issue: – Improve rhythm determination – Consider dependencies between pitch and rhythm Let’s try it! (on Windows or other Java-compatible platforms) https://github.com/kthrlab/JamSketch/tree/jamsketch-0.1.0-lite

• https://github.com/kthrlab/JamSketch/tree/jamsketch-0.1.0-lite