weld

High-performance runtime for data analytics applications

What is weld?

High-performance runtime for data analytics applications

Overview of weld

What it is

Weld is a language and runtime designed to improve the performance of data-intensive applications. It optimizes across libraries and functions by expressing core computations using a common intermediate representation, then optimizing across each framework. Modern analytics applications often combine multiple functions from different libraries, leading to extensive data movement that limits performance. Weld addresses this by lazily building a computation for the entire workflow, then optimizing and evaluating it only when a result is needed.

Key features

  • Expresses core computations using a common intermediate representation.
  • Optimizes across different libraries and functions to reduce data movement.
  • Lazy evaluation: builds a computation for the entire workflow before optimizing and evaluating.
  • Provides a C API for interfacing with Weld.
  • Includes Python bindings.
  • Offers an interactive REPL for inspecting and debugging programs.
  • Integrates with Grizzly, a subset of Pandas.

Use cases

Weld is suitable for data analytics applications that combine multiple functions from different libraries and frameworks. It can be used to optimize complex workflows where individual functions are already performant but the combined workflow suffers from data movement overhead. Example use cases include building vector libraries, integrating with Pandas via Grizzly, and any scenario where a lazy, optimized evaluation of a data pipeline is beneficial.

Getting started

To build Weld, you need the latest stable version of Rust and LLVM/Clang++ 6.0. After installing those dependencies, run:bashgit clone https://www.github.com/weld-project/weldcd weld/export WELD_HOME=`pwd`cargo build --releaseThen run the unit and integration tests:bashcargo test

License and pricing

Weld is licensed under the BSD-3-Clause license. It is open source and free to use.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/weld-projectweld.svg)](https://yunodb.com/tools/weld-projectweld?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/weld-projectweld?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/weld-projectweld.svg" alt="Featured on YunoDB" height="20"></a>