Click on text below to jump to the corresponding location in the video (except iOS)



We have Radim Rehurek.
Google's word2vec
There is no manual input
Unsupervised deep learning

It needs a lot of data to do this

This talk is about implementation

The code is written in C, very optimized, but hard to read and optimize

I ported it to Python to understand the code better.
streaming (generator) for inputI put a nicer interface. Connected to that, you can use the whole ecosystem of Python.
NumPy
My port used NumPy which is the standard in the Python world. I was pretty happy with the result. But it was still 20 times slower than the C implementation.
Cython
I had to translate some Python back to C. But it is a HPC application, tiny core which takes a lot of time and the rest doesn't consume much. After some profiling and debugging, I managed to match the speed of the C implementation.
BLAS
Basic Linear Algebra Subroutines
I translated the core routine into using these BLAS.

Benchmark
Optimized NumPy can still be optimized by about two orders of magnitude.
Diagram explaining implementation
The sentences come from a stream - there is no seeking to middle of sentences.
Worker threads
The port has slightly better accuracy too.
not so good
Cython memoryviews - did not perform as well as plain pointers.
Cython's dynamic compliation doesn't work as expected

BLAS idiosyncracies

Python 3 compatibility

Conclusion slide
QuestionsMemory usage - is this linear growth?It takes about double the amount of memory.
Have you tried numba?
Engineering performant code for deployment scenarios? Tradeoff between speed and maintainability?The tradeoff I make is - depends on how much performance it gets.
You mentioned gensim. Can you tell us more about it?
Gensim is a tool for large scale semantic modeling - I am the author.
Correctness of the code - can you tell us about your test strategy?
It was a bit easier than usual academic papers. We already had a reference implementation. Testing can be done by comparing how it works with word2vec. Mostly standard unit testing stuff.
Video outline created using VideoJots. Click and drag lower right corner to resize video. On iOS devices you cannot jump to video location by clicking on the outline.