weld
High-performance runtime for data analytics applications
What is weld?
High-performance runtime for data analytics applications
Overview of weld
What it is
Weld is a language and runtime designed to improve the performance of data-intensive applications. It optimizes across libraries and functions by expressing core computations using a common intermediate representation, then optimizing across each framework. Modern analytics applications often combine multiple functions from different libraries, leading to extensive data movement that limits performance. Weld addresses this by lazily building a computation for the entire workflow, then optimizing and evaluating it only when a result is needed.
Key features
- Expresses core computations using a common intermediate representation.
- Optimizes across different libraries and functions to reduce data movement.
- Lazy evaluation: builds a computation for the entire workflow before optimizing and evaluating.
- Provides a C API for interfacing with Weld.
- Includes Python bindings.
- Offers an interactive REPL for inspecting and debugging programs.
- Integrates with Grizzly, a subset of Pandas.
Use cases
Weld is suitable for data analytics applications that combine multiple functions from different libraries and frameworks. It can be used to optimize complex workflows where individual functions are already performant but the combined workflow suffers from data movement overhead. Example use cases include building vector libraries, integrating with Pandas via Grizzly, and any scenario where a lazy, optimized evaluation of a data pipeline is beneficial.
Getting started
To build Weld, you need the latest stable version of Rust and LLVM/Clang++ 6.0. After installing those dependencies, run:bashgit clone https://www.github.com/weld-project/weldcd weld/export WELD_HOME=`pwd`cargo build --releaseThen run the unit and integration tests:bashcargo test
License and pricing
Weld is licensed under the BSD-3-Clause license. It is open source and free to use.
Based on the project's public README and repository metadata.