Accelerate Pandas 20x using FireDucks
…by changing just one line of code.
Two of the biggest problems with Pandas is that:
- It always adheres to a single-core computation on a CPU.
- It creates bulky DataFrames.
Moreover, since Pandas follows an eager execution mode (every operation triggers immediate computation), it cannot prepare a smart execution plan that optimizes the entire sequence of operations.
FireDucks is a heavily optimized alternative to Pandas with exactly the same API as Pandas’ that addresses these limitations.
Let’s learn more about it today!
How to use it?
First, install the library:
Next, there are three ways to use it:
- If you are using IPython or Jupyter Notebook, load the extension as follows:
2. Additionally, FireDucks also provides a pandas-like module (fireducks.pandas), which can be imported instead of using Pandas. Thus, to use FireDucks in an existing Pandas pipeline, replace the standard import statement with the one from FireDucks:
3. Lastly, if you have a Python script, executing it as shown below will automatically replace the Pandas import statement with FireDucks:
Done!
It’s that simple to use FireDucks.
The speedup is evident from the gif below from my personal experimentation:
I haven’t shown the full animation due to size limitations, but Pandas code ran in 12.3 seconds while the FireDucks code ran in 3.5 seconds, ~4x speedup:
Speedups typically vary from system to system since FireDucks is driven with multiple cores. The same code above, on a system with more CPU cores, will most likely result in more speedup. In my experimentation, I used the standard Google Colab runtime and FireDucks 1.0.3.
As per FireDucks’ official benchmarks, it can be ~20x faster than Pandas and ~2x faster than Polars, as shown below on several queries:
