pinot

Apache Pinot: a real-time analytics database for low-latency SQL and vector search, serving user-facing and agent-facing applications

What is pinot?

Apache Pinot: a real-time analytics database for low-latency SQL and vector search, serving user-facing and agent-facing applications

Overview of pinot

What it is

Apache Pinot is a real-time distributed OLAP datastore, built to deliver scalable real-time analytics with low latency. It can ingest from batch data sources (such as Hadoop HDFS, Amazon S3, Azure ADLS, Google Cloud Storage) as well as stream data sources (such as Apache Kafka). Pinot also supports vector similarity search for vector database use cases such as semantic search, recommendations, and retrieval-augmented generation (RAG). Its low-latency SQL queries over fresh data make it well suited to agent-facing analytics, including AI assistants and automated decision systems. Pinot was built by engineers at LinkedIn and Uber and is designed to scale up and out with no upper bound. Performance always remains constant based on the size of your cluster and an expected query per second (QPS) threshold.

Key features

  • Fast Queries: Filter and aggregate petabyte data sets with P90 latencies in the tens of milliseconds.
  • High Concurrency: Can serve hundreds of thousands of concurrent queries per second.
  • SQL Query Interface: Accessible through a built-in query editor and a REST API.
  • Versatile Joins: Perform arbitrary fact/dimension and fact/fact joins on petabyte data sets.
  • Column-oriented: A column-oriented database with various compression schemes such as Run Length, Fixed Bit Length.
  • Pluggable indexing: Includes timestamp, inverted, StarTree, Bloom filter, range, text, JSON, and geospatial options.
  • Stream and batch ingest: Ingest from Apache Kafka, Apache Pulsar, and AWS Kinesis in real time. Batch ingest from Hadoop, Spark, AWS S3, and more.
  • Upsert during real-time ingestion: Update the data at-scale with consistency.
  • Built-in Multitenancy: Manage and secure data in isolated logical namespaces.
  • Built for Scale: Horizontally scalable and fault-tolerant.
  • Cloud-native on Kubernetes: Helm chart provides a horizontally scalable and fault-tolerant clustered deployment.

Use cases

Pinot is designed to execute real-time OLAP queries with low latency on massive amounts of data and events. It is suited in contexts where fast analytics, such as aggregations, are needed on immutable data, possibly with real-time data ingestion. Pinot works very well for querying time series data with lots of dimensions and metrics. Example applications include user-facing analytics (e.g., Who Viewed Profile, Company Analytics, Talent Insights, UberEats Restaurant Manager) and agent-facing analytics (e.g., AI assistants, automated decision systems).

Getting started

With Docker running and at least 6 GB of memory allocated to it, start Pinot with a bundled sample dataset:bashdocker run --rm --name pinot-quickstart -p 127.0.0.1:9000:9000 apachepinot/pinot:1.6.0 QuickStart -type batchWait for ***** Quick start setup complete ***** in the logs, then open the query console and run a query. To stop the demo, run docker stop pinot-quickstart in another terminal. For streaming examples and other ways to run Pinot, see the quickstart guide.

License and pricing

Apache Pinot is licensed under the Apache-2.0 license. It is open source and free to use.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/apachepinot.svg)](https://yunodb.com/tools/apachepinot?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/apachepinot?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/apachepinot.svg" alt="Featured on YunoDB" height="20"></a>