SynapseML
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
What is SynapseML?
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
Overview of SynapseML
What it is
SynapseML (formerly MMLSpark) is an open-source library that simplifies the creation of massively scalable machine learning pipelines. It is built on the Apache Spark distributed computing framework and shares the same API as SparkML/MLLib, allowing seamless embedding into existing Apache Spark workflows. SynapseML provides simple, composable, and distributed APIs for tasks such as text analytics, vision, anomaly detection, and others. It can train and evaluate models on single-node, multi-node, and elastically resizable clusters, and is usable across Python, R, Scala, Java, and .NET. The library abstracts over various databases, file systems, and cloud data stores. SynapseML publishes runtime-specific JVM artifacts: Spark 3.5 uses Scala 2.12, while Spark 4.0 and 4.1 use Scala 2.13.
Key features
- Vowpal Wabbit on Spark: Fast, sparse, and effective text analytics.
- Cognitive Services for Big Data: Leverage Microsoft Cognitive Services at scale in SparkML pipelines.
- LightGBM on Spark: Train gradient boosted machines with LightGBM.
- Spark Serving: Serve any Spark computation as a web service with sub-millisecond latency.
- HTTP on Spark: Integration between Spark and the HTTP protocol for distributed microservice orchestration.
- ONNX on Spark: Distributed and hardware accelerated model inference on Spark.
- Responsible AI: Understand opaque-box models and measure dataset biases.
Use cases
SynapseML can be used to build scalable and intelligent systems for anomaly detection, computer vision, deep learning, text analytics, and other domains. It is suitable for scenarios requiring distributed machine learning on large datasets, such as training gradient boosted machines, performing outlier detection with Isolation Forest, or serving Spark computations as web services. The library also supports cyber security machine learning tools and scalable KNN models with conditional queries.
Getting started
See the repository README for installation instructions. The README provides setup guides for Microsoft Fabric, Synapse Analytics, Databricks, Python Standalone, Spark Submit, SBT, Apache Livy and HDInsight, Docker, R, and building from source.
License and pricing
The license is MIT. The pricing type is open source.
Based on the project's public README and repository metadata.