connector-x
Fastest library to load data from DB to DataFrames in Rust and Python
What is connector-x?
Fastest library to load data from DB to DataFrames in Rust and Python
Overview of connector-x
What it is
ConnectorX is a library written in Rust that enables loading data from databases into Python DataFrames. It is designed to be fast and memory efficient, following a zero-copy principle. Users can load data with a single line of code using the cx.read_sql function. Optionally, data loading can be accelerated by specifying a partition column for parallel execution. ConnectorX also supports federated queries, allowing joins across tables from multiple databases in a single query. The project is hosted on GitHub under the sfu-db organization and has a documentation site at sfu-db.github.io/connector-x.
Key features
- Load data from databases to DataFrames in Python with one line of code.
- Supports partitioning on numerical columns (cannot contain NULL) for SPJA queries to enable parallel data loading.
- Federated query support: join tables from two or more databases in a single query.
- Written in Rust, following a zero-copy principle to minimize data copying.
- Supports multiple data sources: Postgres, MySQL, Mariadb, Sqlite, Redshift, Clickhouse, SQL Server, Azure SQL Database, Oracle, Big Query, Trino.
- Supports multiple destinations: Pandas, PyArrow, Modin, Dask, Polars.
- Parallel data loading using multiple threads, one per partition.
Use cases
- Loading large datasets from databases into Python DataFrames efficiently, with reduced memory and time compared to other solutions. According to the README, ConnectorX uses up to 3x less memory and 21x less time overall, and 3x less memory and 13x less time compared with Pandas.
- Accelerating data loading by partitioning a query on a numerical column and downloading partitions in parallel.
- Performing federated queries that join tables from different databases (e.g., Postgres and MySQL) in a single SQL statement.
Getting started
Install ConnectorX via pip: bash pip install connectorx Basic usage to load data from a Postgres database: python import connectorx as cx cx.read_sql('postgresql://username:password@server:port/database', 'SELECT * FROM lineitem') For parallel loading with partitioning: python cx.read_sql('postgresql://...', 'SELECT * FROM lineitem', partition_on='l_orderkey', partition_num=10) For building from source, see the repository README.
License and pricing
The project is open source under the MIT license. Pricing is not stated; it is listed as open source.
Based on the project's public README and repository metadata.