GSoC 2017 comes to a close

This year's GSoC is ending now, so this will be my last blog for the journey I went through with my teammates and mentors. Working with cmusphinx has been an amazing experience and a constructive way ahead in my basic understanding of speech research. 

James Salsman conceived the idea of creating an authentic pronunciation intelligibility remediation framework using pocketsphinx.js. We started working on it since the start of this summer, enhanced pocketsphinx.js to extract pronunciation specific features, used DNNs to predict intelligibility score and finally came up with the framework which can be integrated with Wiktionary's native interface to open this functionality to all the users.

Details of the final product and a live demo is up at this link: DEMO

Here's how the new interface would look on Wiktionary.

We integrated this repository with pocketsphinx.js which will enable it to accept recorded audio in proper format and extract pronunciation specific features from the audio. These features have shown tremendous performance boost as compared to other state-of-the-art algorithms. These features are then sent to the REST service hosted on wikimedia tools page, here. The service houses Keras DNN models trained on these features in a supervised manner. The ground truth is obtained using the data collected using Amazon's mechanical turk followed by manual intelligibility score assignment. Once the service responds the user is shown his/her score. The combination of these features and score can be used for further feedback and remediation using game-theoretical methods.

We are still working with Wiktionary admins to provide us the permission to publish our product as a Wiktionary gadget so that everyone can use it. We have also identified certain directions for further improvement. So this is not the end!

This was my GSoC 2017 work and I really hope English learners can benefit from this learning assistance tool. I will send an update once the Wiktionary gadget is public.

Adios :)