fast_vector_similarity
High-performance vector similarity library in Rust with Python bindings: Spearman, Kendall, distance correlation, Jensen-Shannon, Hoeffding's D, and bootstrappe
What is fast_vector_similarity?
High-performance vector similarity library in Rust with Python bindings: Spearman, Kendall, distance correlation, Jensen-Shannon, Hoeffding's D, and bootstrapped confidence intervals
Overview of fast_vector_similarity
What it is
The Fast Vector Similarity Library is a high-performance Rust-based tool for computing similarity measures between vectors. It provides Python bindings for seamless integration with Python workflows, making it suitable for data analysis, machine learning, and statistical tasks.
Key features
- Implements six similarity measures: Spearman's rank-order correlation, Kendall's tau rank correlation, approximate distance correlation, Jensen-Shannon dependency measure, Hoeffding's D measure, and normalized mutual information - Includes bootstrapping functionality to estimate the distribution of similarity measures and improve confidence in results - Performance optimizations: parallel processing using the
rayoncrate, efficient algorithms (e.g., merge sort for inversion counting), and vectorized operations via thendarraycrate - Benchmarking suite that verifies numerical correctness while measuring performance gains - Python bindings exposing two key functions:py_compute_vector_similarity_statsandpy_compute_bootstrapped_similarity_stats, both returning JSON results
Use cases
- Data analysis and statistical tasks requiring comparison of vectors - Machine learning workflows, such as analyzing text embeddings from large language models (e.g., Llama2)
- Computing similarity measures between high-dimensional embeddings (e.g., 4096-dimensional vectors) and generating ranked lists of similar vectors
Getting started
bash pip install fast_vector_similarity For Rust projects, add the library to your Cargo.toml file. See the repository README for details.
License and pricing
Open source (license not explicitly stated).
Based on the project's public README and repository metadata.