mitdbg/modeldb: A system to manage machine learning models

• Read original article here

See the ModelDB frontend in action:

ModelDB is an end-to-end system to manage machine learning models. It ingests models and associated metadata as models are being trained, stores model data in a structured format, and surfaces it through a web-frontend for rich querying. ModelDB can be used with any ML environment via the ModelDB Light API. ModelDB native clients can be used for advanced support in and .

The ModelDB frontend provides rich summaries and graphs showing model data. The frontend provides functionality to slice and dice this data along various attributes (e.g. operations like filter by hyperparameter, group by datasets) and to build custom charts showing model performance.

ModelDB's Light API can be used with any ML environment to sync model metrics and metadata or even entire config files by calling a few functions (e.g. see here and here respectively).

Alternatively, ModelDB native clients for and can be used to perform automatic, fine-grained logging. Unlike the Light API, native clients do not require the user to explicitly provide model data to ModelDB. The native clients can automatically extract relevant pieces of model data as the model is being built and sync them with ModelDB. Incorporating ModelDB into a scikit-learn / spark.ml workflow is as simple as appending or to relevant methods in the respective libraries. See samples for spark.ml here and those for scikit-learn here.

2017.02.08: ModelDB publicly available! Try it out and contribute.

If you have Docker Compose installed, you can bring up a ModelDB server with just a couple commands.

To run ModelDB with Docker, but without Docker Compose, see detailed instructions.

Note by default ModelDB will listen on localhost:3000.

Watch a video of the setup and installation process here.

ModelDB requires Linux or MacOS. The code below shows how you can install the dependencies on each of them. A detailed list of all the dependencies with the recommended and required versions can be found here. Depending on the client you're using, we assume you have the following already installed: (**) = Python 3.5.1+ is only compatible with thrift 0.10.0+ On OSX, we also assume that you have homebrew installed. Use homebrew to install the dependencies
brew install sqlite
brew install maven
brew install node
brew install sbt
brew install mongodb

ModelDB works only with Thrift 0.9.3 and 0.10.0. Python 3 is only compatible with thrift 0.10.0 If you do not have thrift installed, install via brew.


pip install -r [path_to_modeldb]/client/python/requirements.txt apt-get update
sudo apt-get install sqlite
sudo apt-get install maven
sudo apt-get install sbt
sudo apt-get install nodejs may need to symlink node to nodejs. "cd /usr/bin; ln nodejs node"
sudo apt-get install -y mongodb-org


ModelDB works with thrift 0.9.3 and 0.10.0. The following instructions are for 0.9.3
[path_to_thrift]
wget http://mirror.cc.columbia.edu/pub/software/apache/thrift/0.9.3/thrift-0.9.3.tar.gz
tar -xvzf thrift-0.9.3.tar.gz
thrift-0.9.3
./configure
make
PATH=[path_to_thrift]/:

pip install -r [path_to_modeldb]/client/python/requirements.txt For Linux, you can also refer to this script. ModelDB is composed of three components: the ModelDB server, the ModelDB client libraries, and the ModelDB frontend. In the following, [path_to_modeldb] refers to the directory into which you have cloned the modeldb repo and [thrift_version] is 0.9.3 or 0.10.0 depending on your thrift version (check by running ). run the script to set up the sqlite and the mongodb databases that modeldb will use

***IMPORTANT NOTE: This clears any previous modeldb databases. This should only be done once.***
[path_to_modeldb]/server/codegen
./gen_sqlite.sh


..
./start_server.sh [thrift_version]
NOTE: if you are building the project in eclipse, you may need to uncomment the pluginManagement tags in pom.xml located in the server directory


pip install modeldb

-- or --

[path_to_modeldb]/client/python
./build_client.sh


[path_to_modeldb]/client/scala/libs/spark.ml
./build_client.sh


[path_to_modeldb]/frontend
./start_frontend.sh the frontend will now be available in http://localhost:3000/




mongo --eval

After incorporating ModelDB into your models, follow these steps to run and view them in ModelDB.

More comprehensive documentation on ModelDB, including answers to FAQ, will be available soon in the wiki. Information about the server documentation can be found in the docs folder. For other questions, don't hesitate to contact us.

Questions? Bugs? We're happy to talk about all things ModelDB! Reach out to modeldb at lists.csail.mit.edu or post in the ModelDB Google Group.

Contributions are welcome! Please read this to get started.