MMLSpark provides a number of deep learning and data science tools for Apache Spark, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK) and OpenCV, enabling you to quickly create powerful, highly-scalable predictive and analytical models for large image and text datasets.
Table of Contents
- Easily ingest images from HDFS into Spark
- Pre-process image data using transforms from OpenCV (example:302)
- Featurize images using pre-trained deep neural nets using CNTK (example:301)
- Use pre-trained bidirectional LSTMs from Keras for medical entity extraction (example:304)
- Train DNN-based image classification models on N-Series GPU VMs on Azure (example:401)
- Featurize free-form text data using convenient APIs on top of primitives in SparkML via a single transformer (example:201)
- Fit a lightGBM classification or regression model (example:106)
- Perform parallel distributed hyperparameter tuning with randomized grid search on any spark estimators with a convenient API (example:203)
- Train classification and regression models easily via implicit featurization of data (example:101)
- Compute a rich set of evaluation metrics including per-instance metrics (example:102)
See our notebooks for all examples.
A short example
Below is an excerpt from a simple example of using a pre-trained CNN to classify images in the CIFAR-10 dataset. View the whole source code as an example notebook.
... import mmlspark # Initialize CNTKModel and define input and output columns cntkModel = mmlspark.CNTKModel() \ .setInputCol("images").setOutputCol("output") \ .setModelLocation(modelFile) # Train on dataset with internal spark pipeline scoredImages = cntkModel.transform(imagesWithLabels) ...
Setup and installation
The easiest way to evaluate MMLSpark is via our pre-built Docker container. To do so, run the following command:
docker run -it -p 8888:8888 -e ACCEPT_EULA=yes microsoft/mmlspark
To read the EULA for using the docker image, run
docker run -it -p 8888:8888 microsoft/mmlspark eula
GPU VM Setup
MMLSpark can be used to train deep learning models on GPU nodes from a Spark application. See the instructions for setting up an Azure GPU VM.
MMLSpark can be conveniently installed on existing Spark clusters via the
--packages option, examples:
spark-shell --packages Azure:mmlspark:0.11 pyspark --packages Azure:mmlspark:0.11 spark-submit --packages Azure:mmlspark:0.11 MyApp.jar
To try out MMLSpark on a Python (or Conda) installation you can get
Spark installed via pip with
pip install pyspark. You can then use
pyspark as in the above example, or from python:
import pyspark sp = pyspark.sql.SparkSession.builder.appName("MyApp") \ .config("spark.jars.packages", "Azure:mmlspark:0.11") \ .getOrCreate() import mmlspark
The script action url is: https://mmlspark.azureedge.net/buildartifacts/0.11/install-mmlspark.sh.
If you're using the Azure Portal to run the script action, go to
Script actions →
Submit new in the
Overview section of your cluster blade. In
Bash script URI field, input the script action URL provided above. Mark
the rest of the options as shown on the screenshot to the right.
Submit, and the cluster should finish configuring within 10 minutes or so.
For the coordinates use:
Azure:mmlspark:0.11. Ensure this library is
attached to all clusters you create.
Finally, ensure that your Spark cluster has at least Spark 2.1 and Scala 2.11.
You can use MMLSpark in both your Scala and PySpark notebooks.
If you are building a Spark application in Scala, add the following lines to
resolvers += "MMLSpark Repo" at "https://mmlspark.azureedge.net/maven" libraryDependencies += "com.microsoft.ml.spark" %% "mmlspark" % "0.11"
Building from source
You can also easily create your own build by cloning this repo and use the main
./runme. Run it once to install the needed dependencies, and
again to do a build. See this guide for more
To try out MMLSpark using the R autogenerated wrappers see our instructions. Note: This feature is still under development and some necessary custom wrappers may be missing.
Contributing & feedback
See CONTRIBUTING.md for contribution guidelines.
To give feedback and/or report an issue, open a GitHub Issue.
Other relevant projects
Apache®, Apache Spark, and Spark® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.