Create an Evaluation Recorder

You can run a model evaluation against a registered model object. There are two ways to create a model object on Markov.

1. Using Experiment Recorder

2. Creating Placeholder Model in Project

Note: The model object is a harness for the model artifact generated from a model-training process.

Using Experiment Recorder

When an experiment is run, the model object is automatically created and registered with MarkovML. You can get the model_id of the associated model object by the model_id property of the ExperimentRecorder instance.

Creating a Model when using auto_record

When an experiment is run using auto_record, the model object is automatically created and registered with MarkovML.

For example, if t created an experiment using a keras model by calling markov.keras.auto_record. After you have called model.fit() method, you can get the model_id using model.markov_model_id on the model on which fit() was called.

Creating New Model

You can also create the Model placeholder explicitly using SDK.

from markov import Project, ModelClass

# get existing project by name
my_project = Project.get_by_name(name="MarkovMLTestProject")
# or get existing project by id
my_project = Project.get_by_id(project_id="project_id")
# If the model object does not exist, create a placeholder using this code
my_model = my_project.create_model(
    model_name="MyModel",
    model_description="Model created from project object",
    model_class=ModelClass.CLASSIFICATION
)

# Call this method to register the Model with MarkovML backend. 
my_model.register()

To run an evaluation against an existing model, you can get the model object as follows.

The evaluation dataset should be registered with markovML. This is required to create a lineage between the dataset and evaluations. This also helps in root cause analysis to plan future iterations.

Getting Existing Model for Evaluation

from markov import Model

my_project = Project.get_by_name(name="MarkovMLTestProject")
# or get existing project by id
my_project = Project.get_by_id(project_id="project_id")
# get an existing Model of this project by name
my_model = my_project.get_model_by_name(name="MyFavoriteModel")
# or get Model by model_id
my_model = my_project.get_model_by_id(model_id="model_id")

Evaluating a Model

from markov import EvaluationRecorder

# You can use the model_id from my_model.model_id to create
# and register EvaluationRecorder
model_id = my_model.model_id
dataset_id = markov.dataset.get_by_name("my_dataset").ds_id
evaluation_recorder = EvaluationRecorder(
    name=f"YOUR_MODEL_NAME",
    notes=f"Testing evaluation with MarkovML",
    model_id= model_id,
    # dataset_id these evaluation records belong to. 
    # You should register the evaluation dataset with MarkovML 
    # for lineage and advanced analytics. 
    dataset_id=dataset_id,
    # If you are evaluating a binary classifier, set the positive label
    # to compute the AUC/PR curve. POS_LABEL provided.
    pos_label = "POS_LABEL" 
)
# This method should be called to register a recorder with MarkovML to accept
# the incoming evaluation records.
evaluation_recorder.register()

After creating an evaluation recorder, you can proceed to input evaluation records. Further instructions can be found on the following page.

You can evaluate a model multiple times by supplying the same model_id to your EvaluationRecorder.