PatANN Benchmark Environment #

Overview #

PatANN’s performance has been evaluated using ann-benchmarks, a well-respected benchmarking suite for ANN search algorithms. While ann-benchmarks provides valuable comparison metrics, we’ve identified two key limitations:

• The current ann-benchmarks implementation runs in single-core, single-threaded mode, which doesn’t reflect real-world deployment environments where multi-core CPUs are standard. This limitation prevents algorithms like PatANN and other popular libraries such as HNSWlib from demonstrating their full multi-core capabilities, potentially leading users to make suboptimal implementation choices based on single-CPU benchmarks.

• The single query mode feeds queries one by one to algorithms and expects synchronous results. This approach is problematic for parallel processing algorithms like PatANN, blocking the massive parallelism it offers. While a batch mode exists, results are presented separately from single query mode, preventing unified comparison views.

Benchmark Improvements #

We have filed two detailed issues on GitHub (#568 and #571) that are currently under consideration by the ann-benchmarks maintainers. In the meantime, we’ve implemented two key modifications to address these limitations:

• Removed CPU restriction flags in the Docker container configuration to enable multi-core, multi-threaded execution
• Created a unified view that combines results from both single query and batch modes for consistent comparison

Our modified implementation is available on GitHub which you can download and run on your setup. We hope these improvements will eventually be merged into the original ann-benchmarks repository to benefit the broader ANN community.

Testing Environment #