Final Report of GSoC 2018

Music Blocks widget improvement

So finally I am writing this. It has been a long journey and I had a very good learning experience. I almost can’t believe how much I have learned from the project. I would like to thank my mentors Walter Bender and Devin Ulibarri. They helped a lot to complete the project, and are very good people. I would like to thank Google Summer of code to give me this opportunity.

Aim

To add Keyboard and Pitch tracker widgets to Music Blocks and fix bugs.

Work Done

1)Music Keyboard – This is similar to a piano keyboard. Clicking on keys gives sound output corresponding to that key. User can save created music in terms of blocks. Its main feature is that it is customizable. Only those keys are shown which are inside keyboard widget clamp. If user removes all the keys from clamp then a standard keyboard with keys from C4 to G5 is shown.
Screenshot from 2018-08-03 00-24-59
Link to keyboard branch is here
Pull request is here

2)Pitch Tracker – Taking input from microphone it uses auto-correlation in real time to find fundamental frequency of input waveform. It then finds musical notes which are closest to the detected frequencies. On clicking save buttons it gives output of blocks corresponding to detected frequencies. It is working nicely with musical instruments. For human voice, it sometimes does not give correct output.
pitch tracker
Link to pitch tracker branch is here
Pull request is here

3)Fixed several bugs – I had no earlier experience of working with project of this size. Even a bug which looked small, it took a large amount of time to trace the code which is causing the problem, and more time to fix the code. It took almost a month to fix the bugs.
Merged Bug fixes are
here
here
here

Challenges faced

1)In initial days I could not understand overall workflow of Music Blocks code. I then studied logo.js code. This gave me an overview how things are getting done in Music Blocks.
2)For human voice it is much harder to find fundamental frequency, because speech waveform is composed of a lot of frequencies. We need to use a more sophisticated algorithm. I am taking speech processing course this semester and would like to contribute more as I learn better techniques to do it.

Future Work

1)Adding timing information for Music Keyboard.
2)Getting accurate results from Pitch tracker in case of human voice.

Advertisements

Week 12 ( July 31- August 06)

This week I worked more on pitch tracker widget. The things that I have done are
1)Detected notes are now shown below the widget buttons. These values are updated depending on values of bps and beats.
2)Added user input boxes for BPM and Beats.
3)Added tempo block inside pitch tracker clamp.
4)Used better icons for buttons and added beep sound before start of recording.
5)Wrote documentation for keyboard and pitch tracker

Now pitch tracker looks like this
pitch tracker

Commits are
here
here
here
here

Week 11 (July 23 – July 30)

So basic functionality of Software Keyboard was completed in previous week. I am now working on pitch tracker. It takes audio input from microphone and finds the most dominant frequency. It then uses this frequency to find musical note which is closest to it. Eg if frequency is 560Hz then it chooses C#5. Because C5(523Hz), C#5(554), D5(587) and 560 is closest to 554. So main things done are

1)Added code which is common to all widgets.

2)Used autocorrelation as feature vector. It gives good results in case of musical instruments. It gives frequency as output. We can use whatever bps we want.

3)Defined a function which returns musical note corresponding to frequency.

4)Stored all the notes detected and used them to have save functionality so that we can get output in terms of blocks.

5)Clear, close buttons added.
6)I think we can add waveform recorded pictorially.

 Commits are
here
here
here
here
here

Week 10 (July 16 – July 23)

I finished basic functionality of software keyboard this week. In parts I did following things

1)Till now keyboard and widgets buttons were far away from each other. I combined them inside one html element and now keyboard is just below buttons.

2)Added dragging fuctionality to keyboard.

3)Clicking on play now works both in custom and standard keyboard.

4)Minor changes

So now keyboard looks like thisScreenshot from 2018-08-03 00-24-59

Commits are
here
here
here
here

Week 9 (July 09 – July 16)

This week I did following things

1) Saving functionality added in standard keyboard. By default C4-G5 shown.

So standard keyboard was pretty much done now. So I am now focussing on custom keyboard. In custom keyboard user decides which keys of keyboard she/he wants to use (and see). So a key inside clamp only is shown as part of keyboard.

2)Made a blank space for keeping custom keyboard.

3)Worked to make sure pitches inside clamp are store in an array and are used to show keys.

4)Sound on clicking keys of custom keyboard. I was slightly stuck here because tone js was not accepting string in arguments, but we found out another way.

Commits are
here
here
here
here
here

Week 8 ( July 02 – July 09)

Although the keyboard looks like it is the same element whether or not you put pitch blocks in keyboard widget clamp, It is not so. There is a standard keyboard which is shown when there is no pitch block in the clamp, and custom keyboard which is shown when there are one or more pitches in the clamp. We are using two different html elements for standard and custom keyboard. Currently I am focussing on standard keyboard. So I made following changes this week

1) Standard keyboard is shown by default i.e. no pitches in the clamp by default

2) Since we want to store keys which were pressed by user hence created two arrays which store pitches and their octave.

3) Added widget buttons which are common to every widget i.e. close, clear, play etc.

4) Fixed connection problem by making changes in macros.js. Removed unnecessary thing which were not required. Commits are

here
here
here
here

Week 7 (June 25 – July 02)

I resumed working on meter in rhythm ruler this week. The changes that I made this week include

1)Added a row for showing markings of the corresponding meter below every ruler.

2)Changed the width of the row such that it is lesser that ruler’s width and changed¬† background color.

3)Used our under the hood calculations to show how many times each ruler has to repeat so that there is alignment. Now the rhythm ruler looks like thisScreenshot from 2018-08-02 23-18-41

The working remaining is marking with length of meter. Commits are
here
here
here
here