Final Report of GSoC 2018

Music Blocks widget improvement

So finally I am writing this. It has been a long journey and I had a very good learning experience. I almost can’t believe how much I have learned from the project. I would like to thank my mentors Walter Bender and Devin Ulibarri. They helped a lot to complete the project, and are very good people. I would like to thank Google Summer of code to give me this opportunity.

Aim

To add Keyboard and Pitch tracker widgets to Music Blocks and fix bugs.

Work Done

1)Music Keyboard – This is similar to a piano keyboard. Clicking on keys gives sound output corresponding to that key. User can save created music in terms of blocks. Its main feature is that it is customizable. Only those keys are shown which are inside keyboard widget clamp. If user removes all the keys from clamp then a standard keyboard with keys from C4 to G5 is shown.

Link to keyboard branch is here
Pull request is here

2)Pitch Tracker – Taking input from microphone it uses auto-correlation in real time to find fundamental frequency of input waveform. It then finds musical notes which are closest to the detected frequencies. On clicking save buttons it gives output of blocks corresponding to detected frequencies. It is working nicely with musical instruments. For human voice, it sometimes does not give correct output.

Link to pitch tracker branch is here
Pull request is here

3)Fixed several bugs – I had no earlier experience of working with project of this size. Even a bug which looked small, it took a large amount of time to trace the code which is causing the problem, and more time to fix the code. It took almost a month to fix the bugs.
Merged Bug fixes are
here
here
here

Challenges faced

1)In initial days I could not understand overall workflow of Music Blocks code. I then studied logo.js code. This gave me an overview how things are getting done in Music Blocks.
2)For human voice it is much harder to find fundamental frequency, because speech waveform is composed of a lot of frequencies. We need to use a more sophisticated algorithm. I am taking speech processing course this semester and would like to contribute more as I learn better techniques to do it.

Future Work

1)Adding timing information for Music Keyboard.
2)Getting accurate results from Pitch tracker in case of human voice.

Week 12 ( July 31- August 06)

This week I worked more on pitch tracker widget. The things that I have done are
1)Detected notes are now shown below the widget buttons. These values are updated depending on values of bps and beats.
2)Added user input boxes for BPM and Beats.
3)Added tempo block inside pitch tracker clamp.
4)Used better icons for buttons and added beep sound before start of recording.
5)Wrote documentation for keyboard and pitch tracker

Now pitch tracker looks like this

Commits are
here
here
here
here

Week 11 (July 23 – July 30)

So basic functionality of Software Keyboard was completed in previous week. I am now working on pitch tracker. It takes audio input from microphone and finds the most dominant frequency. It then uses this frequency to find musical note which is closest to it. Eg if frequency is 560Hz then it chooses C#5. Because C5(523Hz), C#5(554), D5(587) and 560 is closest to 554. So main things done are

1)Added code which is common to all widgets.

2)Used autocorrelation as feature vector. It gives good results in case of musical instruments. It gives frequency as output. We can use whatever bps we want.

3)Defined a function which returns musical note corresponding to frequency.

4)Stored all the notes detected and used them to have save functionality so that we can get output in terms of blocks.

5)Clear, close buttons added.
6)I think we can add waveform recorded pictorially.

 Commits are
here
here
here
here
here

Week 10 (July 16 – July 23)

I finished basic functionality of software keyboard this week. In parts I did following things

1)Till now keyboard and widgets buttons were far away from each other. I combined them inside one html element and now keyboard is just below buttons.

2)Added dragging fuctionality to keyboard.

3)Clicking on play now works both in custom and standard keyboard.

4)Minor changes

So now keyboard looks like this

Commits are
here
here
here
here

Week 9 (July 09 – July 16)

This week I did following things

1) Saving functionality added in standard keyboard. By default C4-G5 shown.

So standard keyboard was pretty much done now. So I am now focussing on custom keyboard. In custom keyboard user decides which keys of keyboard she/he wants to use (and see). So a key inside clamp only is shown as part of keyboard.

2)Made a blank space for keeping custom keyboard.

3)Worked to make sure pitches inside clamp are store in an array and are used to show keys.

4)Sound on clicking keys of custom keyboard. I was slightly stuck here because tone js was not accepting string in arguments, but we found out another way.

Commits are
here
here
here
here
here

Week 8 ( July 02 – July 09)

Although the keyboard looks like it is the same element whether or not you put pitch blocks in keyboard widget clamp, It is not so. There is a standard keyboard which is shown when there is no pitch block in the clamp, and custom keyboard which is shown when there are one or more pitches in the clamp. We are using two different html elements for standard and custom keyboard. Currently I am focussing on standard keyboard. So I made following changes this week

1) Standard keyboard is shown by default i.e. no pitches in the clamp by default

2) Since we want to store keys which were pressed by user hence created two arrays which store pitches and their octave.

3) Added widget buttons which are common to every widget i.e. close, clear, play etc.

4) Fixed connection problem by making changes in macros.js. Removed unnecessary thing which were not required. Commits are

here
here
here
here

Week 7 (June 25 – July 02)

I resumed working on meter in rhythm ruler this week. The changes that I made this week include

1)Added a row for showing markings of the corresponding meter below every ruler.

2)Changed the width of the row such that it is lesser that ruler’s width and changed  background color.

3)Used our under the hood calculations to show how many times each ruler has to repeat so that there is alignment. Now the rhythm ruler looks like this

The working remaining is marking with length of meter. Commits are
here
here
here
here

Week 6 (June 18 – June 25)

In initial 4-5 days of this week I worked on software keyboard and added a few commits. These commits do mainly these things

1. Length and width of keys were changed to make them look similar to an actual keyboard
2. I looked into Tone js metthods and made changes according to make sure the sound is coming on keypress instead of on releasing the key.
3. Used onkeypress ‘mouseup’ to make sure that color of keys are changed only till they are pressed and the color revert backs to normal color when the keys are not pressed.

 

In the last 2 days of this week I worked on showing the output of meter in rhythm ruler. I discussed with my mentor about the design. Devin sent a very good design which I will be implementing. The design is shown below

I am working to implement it and till now I have added a horizontal white line with each ruler where we will be marking meter size and repeat till it alignes with ruler. Currently widget table looks like this

Week 5 (June 11 – June 18)

I worked with software keyboard this week. We want to integrate a software keyboard in Music Blocks as a widget. By clicking on the widget what we should see by default is a keyboard with 2 octaves at a time. There is a horizontal scroll bar using which we can change the octaves being shown currently.

When user puts pitches inside of the keyboard widget clamp, then those notes, and only those notes) appear on the keyboard. So we might have a single key keyboard, so we may  have the keys in some random order, or skip the black keys, or skip the white keys.And just like other widgets It will have

-play (to play back your composition)
-save (to save it as blocks)
-clear (to restart)
-sort (to sort the keyboard from low to high pitch)
-close (to exit the widget)
-drag (to reposition the widget)

At the end of this week the keyboard looks like this

It is giving output of sound as expected in a keyboard, and color of keys are changing on keypress. Till now it is not giving output based on time of keypress.

Week 4 ( June 4 – June 11)

I worked with issue 523 this week. Currently the widget simplifies the calculations and outputs in terms of divide only. We want to generate the output of pitch time matrix in terms of divide and multiply when tuplets are used. Instead of 1/12 we want 1/(3*4). We want right side action block instead of current left side action block on clicking save.

To solve I looked at the JSON output of different action blocks. It gave me idea in which format we have to store values and connect the blocks with each other. Multiplication block is needed to be inside divide block hence the sequence of connection is

The vspace is idx+1
The divide block is idx+2
The numerator is idx+3
The multiply is idx +4
Then idx +5 and idx+6 attached to the multiply block.

Walter helped me by adding a commit to code.
Github discussion is here
Commit is here