Clipper Manager API Reference

class clipper_admin.Clipper(host, user=None, key_path=None, sudo=False, ssh_port=22, check_for_docker=True, redis_ip='redis', redis_port=6379, redis_persistence_path=None, restart_containers=False)[source]

Connection to a Clipper instance for administrative purposes.

Sets up the machine for running Clipper. This includes verifying SSH credentials and initializing Docker.

Docker and docker-compose must already by installed on the machine before connecting to a machine.

Parameters:
  • host (str) – The hostname of the machine to start Clipper on. The machine should allow passwordless SSH access.
  • user (str, optional) – The SSH username. This field must be specified if host is not local.
  • key_path (str, optional.) – The path to the SSH private key. This field must be specified if host is not local.
  • sudo (bool, optional.) – Specifies level of execution for docker commands (sudo if true, standard if false).
  • ssh_port (int, optional) – The SSH port to use. Default is port 22.
  • check_for_docker (bool, optional) – If True, checks that Docker is running on the host machine. Default is True.
  • redis_port (int, optional) – The port to use for connecting to redis. Default is port 6379.
  • redis_ip (string, optional) – The ip address of the redis instance that Clipper should use. If unspecified, a docker container running redis will be started on host at the port specified by redis_port.
  • redis_persistence_path (string, optional) – The directory path to which redis data should be persisted. The directory should not already exist. If unspecified, redis will not persist data to disk.
  • restart_containers (bool, optional) – If true, containers will restart on failure. If false, containers will not restart automatically.
start()[source]

Start a Clipper instance.

register_application(name, model, input_type, default_output, slo_micros)[source]

Register a new Clipper application and returns the response object.

Parameters:
  • name (str) – The name of the application.
  • model (str) – The name of the model this application will query.
  • input_type (str) – One of “integers”, “floats”, “doubles”, “bytes”, or “strings”.
  • default_output (string) – The default prediction to use if the model does not return a prediction by the end of the latency objective.
  • slo_micros (int) –

    The query latency objective for the application in microseconds. This is the processing latency between Clipper receiving a request and sending a response. It does not account for network latencies before a request is received or after a response is sent.

    If Clipper cannot process a query within the latency objective, the default output is returned. Therefore, it is recommended that the objective not be set aggressively low unless absolutely necessary. 40000 (40ms) is a good starting value, but the optimal latency objective will vary depending on the application.

Returns:

Returns true iff the app registration request was successful

Return type:

bool

get_all_apps(verbose=False)[source]

Gets information about all applications registered with Clipper.

Parameters:verbose (bool) – If set to False, the returned list contains the apps’ names. If set to True, the list contains application info dictionaries. These dictionaries have the same attribute name-value pairs that were provided to register_application.
Returns:Returns a list of information about all apps registered to Clipper. If no apps are registered with Clipper, an empty list is returned.
Return type:list
get_app_info(name)[source]

Gets detailed information about a registered application.

Parameters:name (str) – The name of the application to look up
Returns:Returns a dictionary with the specified application’s info. This will contain the attribute name-value pairs that were provided to register_application. If no application with name name is registered with Clipper, None is returned.
Return type:dict
deploy_model(name, version, model_data, container_name, input_type, labels=['DEFAULT'], num_containers=1)[source]

Registers a model with Clipper and deploys instances of it in containers.

Parameters:
  • name (str) – The name to assign this model.
  • version (Any object with a string representation (with __str__ implementation)) – The version to assign this model.
  • model_data (str or BaseEstimator) – The trained model to add to Clipper. This can either be a Scikit-Learn trained model object (an instance of BaseEstimator), or a path to a serialized model. Note that many model serialization formats split the model across multiple files (e.g. definition file and weights file or files). If this is the case, model_data must be a path to the root of a directory tree containing ALL the needed files. Depending on the model serialization library you use, this may or may not be the path you provided to the serialize method call.
  • container_name (str) – The Docker container image to use to run this model container.
  • input_type (str) – One of “integers”, “floats”, “doubles”, “bytes”, or “strings”.
  • labels (list of str, optional) – A list of strings annotating the model
  • num_containers (int, optional) – The number of replicas of the model to create. More replicas can be created later as well. Defaults to 1.
register_external_model(name, version, input_type, labels=['DEFAULT'])[source]

Registers a model with Clipper without deploying it in any containers.

Parameters:
  • name (str) – The name to assign this model.
  • version (Any object with a string representation (with __str__ implementation)) – The version to assign this model.
  • input_type (str) – One of “integers”, “floats”, “doubles”, “bytes”, or “strings”.
  • labels (list of str, optional) – A list of strings annotating the model.
deploy_pyspark_model(name, version, predict_function, pyspark_model, sc, input_type, labels=['DEFAULT'], num_containers=1)[source]

Deploy a Spark MLLib model to Clipper.

Parameters:
  • name (str) – The name to assign this model.
  • version (Any object with a string representation (with __str__ implementation)) – The version to assign this model.
  • predict_function (function) – A function that takes three arguments, a SparkContext, the model parameter and a list of inputs of the type specified by the input_type argument. Any state associated with the function other than the Spark model should be captured via closure capture. Note that the function must not capture the SparkContext or the model implicitly, as these objects are not pickleable and therefore will prevent the predict_function from being serialized.
  • pyspark_model (pyspark.mllib.util.Saveable) – An object that mixes in the pyspark Saveable mixin. Generally this is either an mllib model or transformer. This model will be loaded into the Clipper model container and provided as an argument to the predict function each time it is called.
  • sc (SparkContext) – The SparkContext associated with the model. This is needed to save the model for pyspark.mllib models.
  • input_type (str) – One of “integers”, “floats”, “doubles”, “bytes”, or “strings”.
  • labels (list of str, optional) – A set of strings annotating the model
  • num_containers (int, optional) – The number of replicas of the model to create. More replicas can be created later as well. Defaults to 1.
Returns:

True if the model was successfully deployed. False otherwise.

Return type:

bool

deploy_predict_function(name, version, predict_function, input_type, labels=['DEFAULT'], num_containers=1)[source]

Deploy an arbitrary Python function to Clipper.

The function should take a list of inputs of the type specified by input_type and return a Python or numpy array of predictions as strings. All dependencies for the function must be installed with Anaconda or Pip and this function must be called from within an Anaconda environment.

Parameters:
  • name (str) – The name to assign this model.
  • version (Any object with a string representation (with __str__ implementation)) – The version to assign this model.
  • predict_function (function) – The prediction function. Any state associated with the function should be captured via closure capture.
  • input_type (str) – One of “integers”, “floats”, “doubles”, “bytes”, or “strings”.
  • labels (list of str, optional) – A list of strings annotating the model.
  • num_containers (int, optional) – The number of replicas of the model to create. More replicas can be created later as well. Defaults to 1.
Returns:

True if the model was successfully deployed. False otherwise.

Return type:

bool

Example

Define a feature function center() and train a model on the featurized input:

def center(xs):
    means = np.mean(xs, axis=0)
    return xs - means

centered_xs = center(xs)
model = sklearn.linear_model.LogisticRegression()
model.fit(centered_xs, ys)

def centered_predict(inputs):
    centered_inputs = center(inputs)
    return model.predict(centered_inputs)

clipper.deploy_predict_function(
    "example_model",
    1,
    centered_predict,
    "doubles",
    num_containers=1)
register_app_and_deploy_predict_function(name, predict_function, input_type, default_output='None', model_version=1, slo_micros=100000, labels=['DEFAULT'], num_containers=1)[source]

Registers an app and deploys provided predict function as a model.

Parameters:
  • name (str) – The to be assigned to the registered app and deployed model.
  • predict_function (function) – The prediction function. Any state associated with the function should be captured via closure capture.
  • input_type (str) – The input_type to be associated with the registered app and deployed model. One of “integers”, “floats”, “doubles”, “bytes”, or “strings”.
  • default_output (string, optional) – The default prediction to use if the model does not return a prediction by the end of the latency objective.
  • model_version (Any object with a string representation (with __str__ implementation), optional) – The version to assign the deployed model.
  • slo_micros (int) – The query latency objective for the application in microseconds. This is the processing latency between Clipper receiving a request and sending a response. It does not account for network latencies before a request is received or after a response is sent.
  • labels (list of str, optional) – A list of strings annotating the model.
  • num_containers (int, optional) – The number of replicas of the model to create. More replicas can be created later as well.
register_app_and_deploy_pyspark_model(name, predict_function, pyspark_model, sc, input_type, default_output='None', model_version=1, slo_micros=100000, labels=['DEFAULT'], num_containers=1)[source]

Registers an app and deploys provided spark model.

Parameters:
  • name (str) – The to be assigned to the registered app and deployed model.
  • predict_function (function) – A function that takes three arguments, a SparkContext, the model parameter and a list of inputs of the type specified by the input_type argument. Any state associated with the function other than the Spark model should be captured via closure capture. Note that the function must not capture the SparkContext or the model implicitly, as these objects are not pickleable and therefore will prevent the predict_function from being serialized.
  • pyspark_model (pyspark.mllib.util.Saveable) – An object that mixes in the pyspark Saveable mixin. Generally this is either an mllib model or transformer. This model will be loaded into the Clipper model container and provided as an argument to the predict function each time it is called.
  • sc (SparkContext) – The SparkContext associated with the model. This is needed to save the model for pyspark.mllib models.
  • input_type (str) – The input_type to be associated with the registered app and deployed model. One of “integers”, “floats”, “doubles”, “bytes”, or “strings”.
  • default_output (string, optional) – The default prediction to use if the model does not return a prediction by the end of the latency objective.
  • model_version (Any object with a string representation (with __str__ implementation), optional) – The version to assign the deployed model.
  • slo_micros (int, optional) – The query latency objective for the application in microseconds. This is the processing latency between Clipper receiving a request and sending a response. It does not account for network latencies before a request is received or after a response is sent.
  • labels (list of str, optional) – A list of strings annotating the model.
  • num_containers (int, optional) – The number of replicas of the model to create. More replicas can be created later as well.
get_all_models(verbose=False)[source]

Gets information about all models registered with Clipper.

Parameters:verbose (bool) – If set to False, the returned list contains the models’ names. If set to True, the list contains model info dictionaries.
Returns:Returns a list of information about all apps registered to Clipper. If no models are registered with Clipper, an empty list is returned.
Return type:list
get_model_info(model_name, model_version)[source]

Gets detailed information about a registered model.

Parameters:
  • model_name (str) – The name of the model to look up
  • model_version (Any object with a string representation (with __str__ implementation)) – The version of the model to look up
Returns:

Returns a dictionary with the specified model’s info. If no model with name model_name@model_version is registered with Clipper, None is returned.

Return type:

dict

get_all_containers(verbose=False)[source]

Gets information about all containers registered with Clipper.

Parameters:verbose (bool) – If set to False, the returned list contains the apps’ names. If set to True, the list contains container info dictionaries.
Returns:Returns a list of information about all apps registered to Clipper. If no containerss are registered with Clipper, an empty list is returned.
Return type:list
get_container_info(model_name, model_version, replica_id)[source]

Gets detailed information about a registered container.

Parameters:
  • model_name (str) – The name of the container to look up
  • model_version (Any object with a string representation (with __str__ implementation)) – The version of the container to look up
  • replica_id (int) – The container replica to look up
Returns:

A dictionary with the specified container’s info. If no corresponding container is registered with Clipper, None is returned.

Return type:

dict

add_container(model_name, model_version)[source]

Create a new container for an existing model.

Starts a new container for a model that has already been added to Clipper. Note that models are uniquely identified by both name and version, so this method will fail if you have not already called Clipper.deploy_model() for the specified name and version.

Parameters:
  • model_name (str) – The name of the model
  • model_version (Any object with a string representation (with __str__ implementation)) – The version of the model
Returns:

True if the container was added successfully and False if the container could not be added.

Return type:

bool

get_clipper_logs()[source]

Copies the logs from all Docker containers running on the host machine that have been tagged with the Clipper label (ai.clipper.container.label) into the local filesystem.

Returns:Returns a list of local filenames containing the Docker container log snapshots.
Return type:list(str)
inspect_instance()[source]

Fetches metrics from the running Clipper instance.

Returns:The JSON string containing the current set of metrics for this instance. On error, the string will be an error message (not JSON formatted).
Return type:str
set_model_version(model_name, model_version, num_containers=0)[source]

Changes the current model version to model_version.

This method can be used to do model rollback and rollforward to any previously deployed version of the model. Note that model versions automatically get updated when deploy_model() is called, so there is no need to manually update the version as well.

Parameters:
  • model_name (str) – The name of the model
  • model_version (Any object with a string representation (with __str__ implementation)) – The version of the model. Note that model_version must be a model version that has already been deployed.
  • num_containers (int) – The number of new containers to start with the newly selected model version.
remove_inactive_containers(model_name)[source]

Removes all containers serving stale versions of the specified model.

Parameters:model_name (str) – The name of the model whose old containers you want to clean.
stop_all()[source]

Stops and removes all Clipper Docker containers on the host.

deploy_R_model(name, version, model_data, labels=['DEFAULT'], num_containers=1)[source]

Registers a model with Clipper and deploys instances of it in containers. :param name: The name to assign this model. :type name: str :param version: The version to assign this model. :type version: int :param model_data: The trained model to add to Clipper.The type has to be rpy2.robjects.vectors.ListVector,

this is how python’s rpy2 encapsulates any given R model.This model will be loaded into the Clipper model container and provided as an argument to the predict function each time it is called.
Parameters:
  • labels (list of str, optional) – A set of strings annotating the model
  • num_containers (int, optional) – The number of replicas of the model to create. More replicas can be created later as well. Defaults to 1.