spark-nlp

State of the Art Natural Language Processing

What is spark-nlp?

State of the Art Natural Language Processing

Overview of spark-nlp

What it is

Spark NLP is a Natural Language Processing library built on top of Apache Spark. It provides NLP annotations for machine learning pipelines that scale in a distributed environment. The library includes over 100,000 pretrained pipelines and models in more than 200 languages. It supports tasks such as tokenization, word segmentation, part-of-speech tagging, word and sentence embeddings, named entity recognition, dependency parsing, spell checking, text classification, sentiment analysis, token classification, machine translation (over 180 languages), summarization, question answering, table question answering, text generation, image classification, image to text (captioning), automatic speech recognition, zero-shot learning, and many other NLP tasks. Spark NLP is the only open-source NLP library in production that offers state-of-the-art transformers such as BERT, CamemBERT, ALBERT, ELECTRA, XLNet, DistilBERT, RoBERTa, DeBERTa, XLM-RoBERTa, Longformer, ELMO, Universal Sentence Encoder, Llama-2, M2M100, BART, Instructor, E5, Google T5, MarianMT, OpenAI GPT2, Vision Transformers (ViT), OpenAI Whisper, Llama, Mistral, Phi, Qwen2, and more. It extends Apache Spark natively to Python, R, and the JVM ecosystem (Java, Scala, Kotlin).

Key features

  • Text preprocessing, parsing and analysis, sentiment and classification, embeddings, classification and question answering, machine translation and generation, image and speech tasks.
  • Integration and interoperability with ONNX, OpenVINO, TensorFlow, and Llama.cpp (GGUF) for model importing.
  • Over 36,000 pretrained models in more than 200 languages.
  • Multi-lingual support.
  • Supports Apache Spark 3.x (Scala 2.12) and validated Apache Spark 4 versions (Scala 2.13).

Use cases

  • Text classification and sentiment analysis.
  • Named entity recognition and dependency parsing.
  • Machine translation and text generation.
  • Question answering and summarization.
  • Image classification and captioning.
  • Automatic speech recognition.
  • Zero-shot learning.

Getting started

Install Spark NLP with pip and PySpark: sh $ conda create -n sparknlp python=3.9 -y $ conda activate sparknlp # Apache Spark 3.x (Scala 2.12) -> spark-nlp_2.12 $ pip install spark-nlp==7.0.1 pyspark==3.5.1 # Apache Spark 4.0.0 (Scala 2.13) -> spark-nlp-spark400_2.13 $ pip install spark-nlp==7.0.1 pyspark==4.0.0 # Apache Spark 4.0.1 and later (Scala 2.13) -> spark-nlp_2.13 $ pip install spark-nlp==7.0.1 pyspark==4.0.1 Then in Python: python from sparknlp.base import * from sparknlp.annotator import * from sparknlp.pretrained import PretrainedPipeline import sparknlp spark = sparknlp.start() pipeline = PretrainedPipeline('explain_document_dl', lang='en') text = 'The Mona Lisa is a 16th century oil painting created by Leonardo. It is held at the Louvre in Paris.' result = pipeline.annotate(text) print(result['entities']) For more examples, see the repository README.

License and pricing

The license is Apache-2.0. The project is open source; no pricing information is stated.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/johnsnowlabsspark-nlp.svg)](https://yunodb.com/tools/johnsnowlabsspark-nlp?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/johnsnowlabsspark-nlp?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/johnsnowlabsspark-nlp.svg" alt="Featured on YunoDB" height="20"></a>