SynapseML

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

What is SynapseML?

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

Overview of SynapseML

What it is

SynapseML (formerly MMLSpark) is an open-source library that simplifies the creation of massively scalable machine learning pipelines. It is built on the Apache Spark distributed computing framework and shares the same API as SparkML/MLLib, allowing seamless embedding into existing Apache Spark workflows. SynapseML provides simple, composable, and distributed APIs for tasks such as text analytics, vision, anomaly detection, and others. It can train and evaluate models on single-node, multi-node, and elastically resizable clusters, and is usable across Python, R, Scala, Java, and .NET. The library abstracts over various databases, file systems, and cloud data stores. SynapseML publishes runtime-specific JVM artifacts: Spark 3.5 uses Scala 2.12, while Spark 4.0 and 4.1 use Scala 2.13.

Key features

  • Vowpal Wabbit on Spark: Fast, sparse, and effective text analytics.
  • Cognitive Services for Big Data: Leverage Microsoft Cognitive Services at scale in SparkML pipelines.
  • LightGBM on Spark: Train gradient boosted machines with LightGBM.
  • Spark Serving: Serve any Spark computation as a web service with sub-millisecond latency.
  • HTTP on Spark: Integration between Spark and the HTTP protocol for distributed microservice orchestration.
  • ONNX on Spark: Distributed and hardware accelerated model inference on Spark.
  • Responsible AI: Understand opaque-box models and measure dataset biases.

Use cases

SynapseML can be used to build scalable and intelligent systems for anomaly detection, computer vision, deep learning, text analytics, and other domains. It is suitable for scenarios requiring distributed machine learning on large datasets, such as training gradient boosted machines, performing outlier detection with Isolation Forest, or serving Spark computations as web services. The library also supports cyber security machine learning tools and scalable KNN models with conditional queries.

Getting started

See the repository README for installation instructions. The README provides setup guides for Microsoft Fabric, Synapse Analytics, Databricks, Python Standalone, Spark Submit, SBT, Apache Livy and HDInsight, Docker, R, and building from source.

License and pricing

The license is MIT. The pricing type is open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/microsoftsynapseml.svg)](https://yunodb.com/tools/microsoftsynapseml?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/microsoftsynapseml?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/microsoftsynapseml.svg" alt="Featured on YunoDB" height="20"></a>