Tutorial - Running ML Inference with OctaiOxide (OctaiPipe in WASM)#

In this tutorial we will take you through deployment of two machine learning models using our WASM (WebAssembly) Inference image named OctaiOxide. The yml configurations are identical between our Python OctaiPipe and Wasm OctaiOxide images. You can choose which to deploy by specifying the image name in the deployment configuration.

Currently, WASM inference supports dataloading via csv on your local device however more options will become available in future releases. If you need to use other datastores, please use the standard OctaiPipe image.

We will demonstrate inference deployment using two examples:

Benefits of using our WASM based image#

  • Web assembley compiled bytecode runs at near native speed without the need for a garbage collector giving faster performance to our equivalent python image.

  • Reduced CPU and memory footprint allow ML Inference to run on lower spec devices.

  • Smaller image size reduces startup time and storage space.

  • WASM provides a secure sandbox environment since all file and network access is deny by default unless explicitly granted.

Setup environment and devices#

  • Make sure that your Portal url is set as the “OCTAICLIENT_ENDPOINT” environment variable.

  • We recommend using the latest octaioxide image as specified below, however you can choose a specific version.

  • You will need to define a device to use for this tutorial as well as it’s credentials as we will be using these to send across and manage datasets via ssh connection. These details are not required by OctaiPipe but are being used in this instance to simplify the dataset setup and teardown process.

The devices dictionary below should use the following format:

devices = {
    'device_id': <device_id as referenced in your Portal>,
    'ip': <device's external IP>,
    'user': <os username for logging into your device>
    'password': <password for logging into your device>
}

  • The device_id should correspond to the device ID on the devices page of your Portal front end - it is the device Id that was given when registering the device. If you have not registered your device yet, you will need to do so. Check the registration documentation for instructions.

  • The device’s ip can be retrieved by running curl ifconfig.me from on the device.

  • The user and password are the credentials used to log into the device.

[ ]:

import os

import yaml

# Set your device credentials:
device = {
    'device_id': 'device_id',
    'ip': 'XXX.XXX.XXX.XXX',
    'user': 'octaipipe',
    'password': 'password'
}
os.environ["DEVICE_ID"] = device['device_id']

# Set SSH commands to facilitate dataset transfer and teardown
device['ssh_command'] = f"sshpass -p '{device['password']}' ssh -o StrictHostKeyChecking=no {device['user']}@{device['ip']}"
device['scp_command'] = f"sshpass -p {device['password']} scp -o StrictHostKeyChecking=no"

# Set OctaiClient Endpoint, this is the url of your Portal
os.environ["OCTAICLIENT_ENDPOINT"] = "https://app.demo22.octaipipe.ai/"

# Set Wasm Inference image
os.environ["IMAGE_NAME"] = "octaipipe.azurecr.io/octaioxide:latest"

# Substitute device and image into your deployment config
! envsubst < configs/templates/deployment_wasm.yml > configs/dynamic_configs/deployment_wasm.yml
! envsubst < configs/templates/deployment_wasm_bearing.yml > configs/dynamic_configs/deployment_wasm_bearing.yml

Glance at registered devices#

By finding all devices we can take a look at the devices that have been registered in your Portal. Ensure that the device you have configured above is registered and has device status as ‘Available’

[ ]:

import pandas as pd

from octaipipe import devices, models, experiments
from octaipipe.deployment.edge.octaideploy import octaideploy
import logging
import yaml

logging.basicConfig(level=logging.INFO, format='%(message)s')

all_devices = devices.find_all_devices()
all_devices = pd.DataFrame(all_devices)
all_devices.head(10)

Load trained models to portal#

Since this tutorial focuses on inference, we will be using existing trained models. You will need to run setup_model() to load the experiment, model meta data and trained models to your Portal.

[ ]:

from portal_record.setup_model import setup_model
setup_model()

CMAPSS Dataset#

[ ]:

# Display dataset
df = pd.read_csv("dataset/testdata_tutorial_cmapss_latest.csv")
df.head(5)

Explore experiment record#

Lets check that the experiment was correctly loaded to your Portal:

[ ]:

exp = experiments.get_experiment_by_id('inference_test_experiment_0')
pd.Series(exp)

Explore model record#

Lets also check that the model was loaded:

[ ]:

model = models.find_models_by_experiment_id('inference_test_experiment_0')
pd.DataFrame(model)

We can search for the model by the experiment_id or model_id:

[ ]:

model = models.find_models_by_name('inference_test_regressor')
pd.DataFrame(model)

Upload CMAPSS dataset#

Here we upload a sample csv of the dataset to your device * Send_file_to_device() is a helper function to facilitate sending the dataset to your device specifically for this tutorial. * To run Inference on a local csv, it must be saved in the /home/octaipipe/ folder of your device

[22]:

# Display inference config
with open("configs/inference_config_wasm.yml", 'r') as file:
    inference_config = yaml.safe_load(file)
print(yaml.dump(inference_config, sort_keys=False))

name: model_inference
input_data_specs:
  datastore_type: csv
  settings:
    delimiter: ','
    file_path: /home/octaipipe/testdata_tutorial_cmapss_latest.csv
    headers: true
    index:
    - _time
    - Machine number
model_specs:
  fl_model: true
  model_id: inference_test_regressor_0
  name: inference_test_regressor
  type: base_SGDRegressor
  version: '1'
output_data_specs:
- datastore_type: csv
  settings:
    append: true
    delimiter: ','
    file_path: /home/octaipipe/testdata_tutorial_cmapss_out.csv
run_specs:
  only_last_batch: true
  prediction_period: 5s

[23]:

# Display deployment config
# Note that you need to have generated this file from earlier in the notebook
with open("configs/dynamic_configs/deployment_wasm.yml", 'r') as file:
    inference_config = yaml.safe_load(file)
print(yaml.dump(inference_config, sort_keys=False))

name: deployment_config
device_ids:
- device_id
image_name: octaipipe.azurecr.io/octaioxide:latest
env: {}
datasources:
  environment:
  - INFLUX_ORG
  - INFLUX_URL
  - INFLUX_TOKEN
  - INFLUX_DEFAULT_BUCKET
  env_file:
  - ./credentials.env
grafana_deployment: null
grafana_cloud_config_path: null
pipelines:
- model_inference:
    config_path: configs/inference_config_wasm.yml

[ ]:

def send_file_to_device(local_path: str, destination_file_path: str, device: dict):
    """Helper function to send datasets to your device

    Args:
        local_path (str): Path to the local file
        destination_file_path (str): Path to the file on the device
        device (dict): Device dictionary with the following keys:
            - device_id (str): device_id as referenced in Portal
            - ip (str): IP address of the device
            - user (str): Username
            - password (str): Password
    """

    full_dest_path = f"{device['user']}@{device['ip']}:{destination_file_path}"
    result = ! {device['scp_command']} {local_path} {full_dest_path}
    if len(result) != 0:
        print(f'Potential issue sending file: {result}')
    else:
        print(f'Successfully sent file to {device}:{destination_file_path}.')
    return result

[ ]:

send_file_to_device(local_path="dataset/testdata_tutorial_cmapss_latest.csv",
                    destination_file_path="/home/octaipipe/testdata_tutorial_cmapss_latest.csv",
                    device=device)

CMAPSS Model Deploy to your Device using Wasm#

Finally we can call OctaiDeploy using our deployment configuration. This will deploy our WASM inference container to our device and run inference continually against the data we sent across as specified in the configuration: wasm-inference/configs/inference_config_wasm.yml. To learn more about inference configurations check our inference step documentation.

[ ]:

from octaipipe.deployment.edge.octaideploy import octaideploy

deployment_description: str = """
Model inference deployment for tutorial purposes.
Model deployed is an SGD Regressor model trained
on the CMAPSS dataset, predicting remaining useful life
of turbofan engines.
"""
cmapss_deployment_id = octaideploy(config_path='./configs/dynamic_configs/deployment_wasm.yml',
                                   compose_action='start',
                                   name='Tutorial test deployment',
                                   description=deployment_description)

[ ]:

# To confirm that predictions are being made, check the csv output file:
! {device['ssh_command']} 'cat /home/octaipipe/testdata_tutorial_cmapss_out.csv'

To shutdown the container we can run OctaiDeploy with compose_ation=down

[ ]:

# Teardown deployment
octaideploy(deployment_id=cmapss_deployment_id, compose_action='down')

[ ]:

# Remove CMAPSS dataset and prediction files from your device
! {device['ssh_command']} 'rm -f /home/octaipipe/testdata_tutorial_cmapss*'

NASA Bearing Dataset#

This time we will perform inference using the NASA Bearing Dataset using a neural network model.

[ ]:

# Display dataset
df = pd.read_csv("dataset/testdata_tutorial_bearing_latest.csv")
df.head(5)

[ ]:

# Explore the Experiment Record:
exp = experiments.get_experiment_by_id('inference_test_experiment_1')
pd.Series(exp)

[ ]:

# Explore the Model:
model = models.find_models_by_experiment_id('inference_test_experiment_1')
pd.DataFrame(model)

[ ]:

# Send Nasa Bearing dataset to your device:
send_file_to_device(local_path="dataset/testdata_tutorial_bearing_latest.csv",
                    destination_file_path="/home/octaipipe/testdata_tutorial_bearing_latest.csv",
                    device=device)

NASA Bearing neural network model deployment to device#

[ ]:

from octaipipe.deployment.edge.octaideploy import octaideploy

deployment_description: str = '''
Model inference deployment for demo purposes.
Model deployed is an FL neural network model trained
on the Nasa Bearing dataset.
'''
bearing_deployment_id = octaideploy(config_path='./configs/dynamic_configs/deployment_wasm_bearing.yml',
                                    compose_action='start',
                                    name='Tutorial test deployment',
                                    description=deployment_description)

[ ]:

# Check predictions:
! {device['ssh_command']} 'cat /home/octaipipe/testdata_tutorial_bearing_out.csv'

[ ]:

# Teardown deployment
octaideploy(deployment_id=bearing_deployment_id, compose_action='down')

[ ]:

# Remove bearing dataset and prediction files from your device
! {device['ssh_command']} 'rm -f /home/octaipipe/testdata_tutorial_bearing*'