北原研究室の研究事例紹介：ベーシストの旋律分析とイコライザーの印象分析（Music×Analytics Meetup vol.5 ロングトーク）

Music×Analytics Meetup #5 北原研究室の研究事例紹介： ベーシストの旋律分析と イコライザーの印象分析 日本大学 文理学部 情報科学科 北原 鉄朗 Twitter: @tetsurokitahara

北原研究室のご紹介 ● 日本大学 文理学部 情報科学科 ● 2010年に設立 ● 合言葉： Technology Makes Music More Fun ● 音楽情報処理、音声情報処理、 エンターテインメントコンピューティングなど ● 例年、3・4年6～10名ずつ、院生はいたりいなかったり ● Web: http://www.kthrlab.jp/ Twitter: @kthrlab YouTube: 「日大文理 北原研」で検索

• http://www.kthrlab.jp/

本研究室で行ってきた研究テーマ（主なもの） 音楽を聴く人を助けたい 演奏する人を助けたい 複数人が同じBGMを ピアノ初見支援 MIDIギター精度改善 聴く場面の楽曲推薦 演奏時の筋活動分析 即興演奏支援 歌う人を助けたい 作曲する人を助けたい カラオケを盛り上げる スマートタンバリン 旋律概形による作曲 四声体和声生成 ハモリ練習支援 音痴な人の分析 ループシーケンサ ベースライン生成 マッシュアップ支援 イコライザー支援 ベーシストの 特徴分析 土台となる技術 HCI 音響信号処理 機械学習 etc

本研究室で行ってきた研究テーマ（主なもの） 音楽を聴く人を助けたい 演奏する人を助けたい 複数人が同じBGMを ピアノ初見支援 MIDIギター精度改善 聴く場面の楽曲推薦 演奏時の筋活動分析 即興演奏支援 歌う人を助けたい 作曲する人を助けたい カラオケを盛り上げる スマートタンバリン 旋律概形による作曲 四声体和声生成 ハモリ練習支援 マッシュアップ支援 1 ベーシストの 特徴分析 2 イコライザー支援 etc 音痴な人の分析 分析 ループシーケンサ ベースライン生成 （基礎寄り） 要素技術開発 システム開発 （応用寄り）

1. ベーシストの旋律分析 A Pattern Recognition Approach to Analyze Temporal Evolution of a Bassist's Musical Styles Y. Matsuura, T. Tanahashi, and T. Kitahara Proc. CSMC 2017

Background ● ● ● Bass is an important part from both rhythmic & harmonic aspects of music Bassists have their own styles in bass phrases But their styles evolve due to: – changes in their musical preferences, – changes in musical trends, – changes in band members, – etc. Red Hot Chili Peppers ■ Year 1989 Year 1999 Ba. Ba. 『Higher Ground』 『Parallel Universe』

Background ● ● Such evolution is qualitatively mentioned in commercial music magazines But few attempts to investigate them quantitively “By realizing John Frusciante’s swelled ideas, the band has shifted its musical style to being focused on the vocal.” “Flea began to play bass lines that support the vocals with his vintage jazz bass.” (originally in Japanese; translated by us)

Our aim To investigate the temporal evolution of a bassist’s musical styles in a quantitative approach Target bassist Our approach Flea (Red Hot Chilli Peppers) ● ● ● Extract features from MIDI data of his 1st- and 2nd-half bass melodies Classify them with pattern recognition methods Investigate features that contributes to accurate classification

What’s commonly known about Flea Active bass lines Year 1989 Ba. 『Higher Ground』 - Low pitch range - Pitch moves up and down - Often use slapping 1999 John Frusciante (guitarist) came back Simple bass lines - High pitch range - Pitch doesn’t move largely - Enhance vocals Year 1999 Ba. 『Parallel Universe』

Data Year Album name #pieces 1989 1991 1999 2002 2006 2011 Mother's Milk Blood Sugar Sex Magik Californication By the Way Stadium Acadium I'm with You 13 17 15 16 28 14 ● We made MIDI data from scores ● Timbral or expressive features are not available

Feature extraction ● Pitch-based features (P-1 to P-37) – P-1: Mean pitch through the whole piece – P-2: Num of note seqs having the same note number – ... ● Duration-based features (D-1 to D-9) – D-1: Mean duration in all notes – D-3: Ratio of num of successive note pairs with different durations – ... ● Count-based features (C-1 to C-11) – C-1: Num of notes in the whole piece – ... (1 vector from a whole piece)

Experiment 1: Division of period ● Classify of 1st- and 2nd-period bass melodies ● Flea’s style is considered to have changed in 1999 ● Classification should be the most accurate when the boundary is set to 1999 Year Album name #pieces 1989 1991 1999 2002 2006 2011 Mother's Milk Blood Sugar Sex Magik Californication By the Way Stadium Acadium I'm with You 1st period ３０ 2nd period ７３

Results (10-fold cross validation) J48 IBk BayesNet Multilayer Perceptron 1999 76% 78% 73% 84% 2002 61% 54% 61% 63% 2006 65% 55% 62% 50% The boundary of 1999 made the highest accuracy Bass styles actually changed in 1999

Experiment 2: Feature selection ● Search a small feature set maximizing accuracy ● Selected features illustrate changes in bass lines ● Analyze important features with decision trees

Selected features ● P-1: Mean pitch ● P-6: Num of successive note pairs where the absolute diff. in the note nums is 3 ● P-16: Ratio of successive note pairs where Pitch Local pitch motions the absolute diff. in the note nums is 0 ● C-10: Ratio of notes with top 5 note nums Only with 4 features, we obtain 82% accuracy Pitch variation

Obtained decision tree P-1: Mean pitch C-10: Ratio of notes with top 5 note nums P-1: Mean pitch P-6: Num of succ note pairs where abs diff in note nums is 3

Discussion Magazines and web sites say: ● ● ● ● Since the album “Californication”, Anthony Kiedis’s vocal has been melodious, and accordingly Flea’s bass line has become snuggled up to the melodies “By The Way” has been surprisingly a John-centric work. Following the melody-enhanced line shown in the previous work, in “Don’t Forget Me” for example, the guitar sounding tremolos madly plays the leading role; Flea dryly plays the chord progression behind it. In “Californization”, released after John Frusciante came back, melodious songs increased. The sound on the basis of Anthony’s “singing” vocal confirm that this band has proceeded to a new step. By realizing John Frusciante’s swelled ideas, the band has shifted its musical style to being focused on the vocal. At the same time, Flea began to play bass lines that support the vocals with his vintage jazz bass.

Discussion To summarize Selected features Active bass lines - Low pitch range - Pitch moves up and down - Often use slapping 1999 ● Overall mean pitch ● Local pitch motions ● Pitch variation John Frusciante (guitarist) came back Simple bass lines Selected features match well-known tendencies - High pitch range - Pitch doesn’t move largely - Enhance vocals

2. イコライザーの印象分析 An Investigation towards Verbally Controllable Graphic Equalizer for Singing Voices S. Masuda, E. Aiba, and T. Kitahara Proc. WIMP 2019

Motivation ● Recently, many amateur singers publish their singing performances on YouTube etc. 「歌ってみた」 (tried to sing)

Motivation ● Recently, many amateur singers publish their singing performances on YouTube etc. Why Because the cost is decreasing (PC, audio I/F, and DAW software) Problem Processing voices is not easy Record voice 12dB boost at 2-16 kHz 12dB cut at 125-1000 Hz Equalize (EQ) Use other effects Mix with backing track - EQ is important to obtain their desired timbre - but not easy for people without know-how

Goal ● To enable amateur singers to equalize their voices in an intuitive way Key idea We often verbally express sounds (e.g. bright sounds, warm sounds) Control EQ with verbal expression Verbally Controllable Equalizer ...... Bright- Warmness ness Specified by the user 500 Hz 2 kHz 8 kHz Automatically obtained

Investigation of relationship between EQ settings and verbal evaluation Participants ● 11 university students – 4 have experience in musical performances – 1 has experience in DAW Sound stimuli (recording) ● Song: "Yoake to Hotaru" (Nabuna) ● Sung by the 1st author ● Recorded only singing voice ● Mic: ATAT4040 (Audio Technica) ● Audio I/F: UM2 (Behringer) >

Sound stimuli (EQ) ● ● Used Cubase 7.5 Treated the 10-band EQ as a 4-band EQ 500-1000Hz 125-250Hz 8-16kHz 2-4kHz – Lowest F0: 112 Hz ● Prepared 27 stimuli – not EQ – 1 band boosted by 12 dB – 1 band cut by 12 dB – 2 bands boosted by 12dB excluded – 1 band boosted by 12 dB and 1 band cut by 12 dB ● RMS is normalized

Examples of Sound stimuli Low Mid-lo Mid-hi High Low 0 0 0 0 0 0 0 + > 0 0 + 0 + + Mid-lo Mid-hi High 0 + + 0 + + > > 0 0 0 > + 0 0 + > 0 0 > 0 + + 0 > 0 0 0 > 0 0 - + > 0 0 0 - > 0 - 0 + > 0 0 - 0 > - 0 0 + > 0 - 0 0 0 0 0 > 0 0 0 + + 0 - > > > >

Selection of Sound Expression Words ● ● Extracted 10 words expressing the effects of EQ from EQ know-how books for hobby musicians Words: 暖かさ (warmness), 存在感 (presence), 派手さ (showiness), 濁り (muddiness), まろやかさ (mellowness), 柔らかさ (softness), 明るさ (brightness), 軽さ (lightness), 太さ (thickness), クリアさ (clearness)

Procedure 1. Listen to both pre-EQ & EQed stimuli 2. Evaluate how stronger the characteristics expressed by each word became through EQ Timeline 4.3 cm 1.3 cm 13.9 cm 1.3 cm Pre-EQ Rest (3s) EQed stimulus (evaluate it during this) Rest (10s) Example of evaluations Warmness > Showiness > Clearness >

Result of linear regression ● Approximated the average yj of the participants’ evaluations w.r.t. word j as yj ~ bj0 + Σ bjkxk (where xk is boost/cut level at each freq. range) The results match descriptions in know-how books

Application to verbal EQ controller We implemented a prototype on GNU Octave Direct controller Verbal controller Mutually transformable

おわりに ベーシストの旋律分析 ● Red Hot Chilli Peppers の Flea に着目 ● 1999年前後でベースラインの特徴が変化 ● パターン認識的アプローチで確認 イコライザーの印象分析 印象語で操作できるイコライザーを目標 ● 印象語と高音・低音のブースト・カットの関係を調査 ● そして… ● さらなる情報： YouTubeで「日大文理 北原研」 ● 大学院生募集中