- Data Science PyData
- Radim Řehůřek - Faster than Google? Optimization lessons in Python.
Click on text below to jump to the corresponding location in the video (except iOS)
We have Radim Rehurek.
Google's word2vecThere is no manual input
Unsupervised deep learning
It needs a lot of data to do this
This talk is about implementation
The code is written in C, very optimized, but hard to read and optimize
I ported it to Python to understand the code better. streaming (generator) for inputI put a nicer interface. Connected to that, you can use the whole ecosystem of Python.
NumPyMy port used NumPy which is the standard in the Python world. I was pretty happy with the result. But it was still 20 times slower than the C implementation.
CythonI had to translate some Python back to C. But it is a HPC application, tiny core which takes a lot of time and the rest doesn't consume much. After some profiling and debugging, I managed to match the speed of the C implementation.
BLASBasic Linear Algebra Subroutines
I translated the core routine into using these BLAS.
BenchmarkOptimized NumPy can still be optimized by about two orders of magnitude.
Diagram explaining implementationThe sentences come from a stream - there is no seeking to middle of sentences.
Worker threadsThe port has slightly better accuracy too.
not so goodCython memoryviews - did not perform as well as plain pointers.
Cython's dynamic compliation doesn't work as expected
Python 3 compatibility
Conclusion slideQuestionsMemory usage - is this linear growth?It takes about double the amount of memory.
Have you tried numba?Engineering performant code for deployment scenarios? Tradeoff between speed and maintainability?The tradeoff I make is - depends on how much performance it gets.
You mentioned gensim. Can you tell us more about it?Gensim is a tool for large scale semantic modeling - I am the author.
Correctness of the code - can you tell us about your test strategy?It was a bit easier than usual academic papers. We already had a reference implementation. Testing can be done by comparing how it works with word2vec. Mostly standard unit testing stuff. Video outline created using VideoJots. Click and drag lower right corner to resize video. On iOS devices you cannot jump to video location by clicking on the outline.