connector-x

Fastest library to load data from DB to DataFrames in Rust and Python

What is connector-x?

Fastest library to load data from DB to DataFrames in Rust and Python

Overview of connector-x

What it is

ConnectorX is a library written in Rust that enables loading data from databases into Python DataFrames. It is designed to be fast and memory efficient, following a zero-copy principle. Users can load data with a single line of code using the cx.read_sql function. Optionally, data loading can be accelerated by specifying a partition column for parallel execution. ConnectorX also supports federated queries, allowing joins across tables from multiple databases in a single query. The project is hosted on GitHub under the sfu-db organization and has a documentation site at sfu-db.github.io/connector-x.

Key features

  • Load data from databases to DataFrames in Python with one line of code.
  • Supports partitioning on numerical columns (cannot contain NULL) for SPJA queries to enable parallel data loading.
  • Federated query support: join tables from two or more databases in a single query.
  • Written in Rust, following a zero-copy principle to minimize data copying.
  • Supports multiple data sources: Postgres, MySQL, Mariadb, Sqlite, Redshift, Clickhouse, SQL Server, Azure SQL Database, Oracle, Big Query, Trino.
  • Supports multiple destinations: Pandas, PyArrow, Modin, Dask, Polars.
  • Parallel data loading using multiple threads, one per partition.

Use cases

  • Loading large datasets from databases into Python DataFrames efficiently, with reduced memory and time compared to other solutions. According to the README, ConnectorX uses up to 3x less memory and 21x less time overall, and 3x less memory and 13x less time compared with Pandas.
  • Accelerating data loading by partitioning a query on a numerical column and downloading partitions in parallel.
  • Performing federated queries that join tables from different databases (e.g., Postgres and MySQL) in a single SQL statement.

Getting started

Install ConnectorX via pip: bash pip install connectorx Basic usage to load data from a Postgres database: python import connectorx as cx cx.read_sql('postgresql://username:password@server:port/database', 'SELECT * FROM lineitem') For parallel loading with partitioning: python cx.read_sql('postgresql://...', 'SELECT * FROM lineitem', partition_on='l_orderkey', partition_num=10) For building from source, see the repository README.

License and pricing

The project is open source under the MIT license. Pricing is not stated; it is listed as open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/sfu-dbconnector-x.svg)](https://yunodb.com/tools/sfu-dbconnector-x?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/sfu-dbconnector-x?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/sfu-dbconnector-x.svg" alt="Featured on YunoDB" height="20"></a>