Release Notes#

2.5.0#

Release Date: December 20, 2024

What’s changed#

Python 3.11#

In OctaiPipe 2.5.0, we have upgraded all components using Python from 3.9 to 3.11. We have also upgraded our Linux versions from Debian 11 (Bullseye) to Debian 12 (Bookworm). We have also upgraded dependencies to our Python packages.

Infrastructure cost reduction and management#

In 2.5, we have spent a significant amount of effort tuning and optimizing infrastructure costs based on usage metrics from current customers. This work has allowed us to reduce cloud costs of the platform by 75%, while maintaining the same performance.

In addition, Azure users are now able to manage some of their own infrastructure costs by controlling Kubernetes nodepools and accessing Cost Management Analytics.

See https://docs.octaipipe.ai/Cloud_Infrastructure_Management.html#azure

Validation of config files#

We have added the functionality to validate your config files before running jobs in OctaiPipe. You can now send a config file to our backend, and if it has been misconfigured, we will return a list containing each misconfiguration so that you don’t have to find out by your process failing during runtime.

Check out the new functionality in the develop section of the user interface: https://docs.octaipipe.ai/User_Interface.html

Stop device containers from Portal UI#

We now have a button in the Portal UI for stopping device containers running in a deployment. This is a way to quickly interrupt processes if they go wrong or if a user without access to the development interface needs to stop a deployment.

Email alert on malicious clients#

We have added an option to send email alerts to select email addresses if malicious clients are detected during Federated Learning. This means that not only is OctaiPipe resilient to malicious clients in the model aggregation process, but you will also be notified which of your assets have potentially been compromised.

See https://docs.octaipipe.ai/usage/federated_learning/Adversarial_Fortification.html#enabling-notifications

Jupyter Development environment improvements#

The Jupyter Development interface has been improved by adding extensions for code formatting, syntax highlighting and code documentation integration. This is to make the development environment as comfortable and easy as possible to work with.

Improved dashboard management#

We have added the grafana_dashboard_paths argument to the deployment config to help users deploy and manage custom dashboards at scale. With this, you can provide a list of your own Grafana dashboards which you wish to deploy to every asset in your fleet.

Additional Features#

  • Continued security updates and updates of dependencies based on vulnerability scans and code analysis

  • Verification of the ability to run federated EDA and federated learning on single assets

  • Added a silhouette plot to the Portal UI as the default visualization of k-FED models

  • Updated all tutorials to not rely on public IP addresses

  • Continued updates and amendments to user documentation and tutorials

Bug fixes#

  • Fixed excel reader requiring engine

  • Fixed being able to delete device even if it’s in a group (removes device from group and then deletes)

  • Fixed warning appearing saying “no step name given” when running local custom step and providing step name

  • Fixed circular import warning when running custom step locally

  • Fixed model versioning for FL models (now uses semantic model versioning in the same way as non-FL models)

  • Fixed a bug when spinning up Grafana where additional device commands are blocked for 10 minutes

  • Fixed issue in 2D and 3D plots of cluster centroids for k-FED

2.4.2#

Release Date: November 29, 2024

What’s changed#

  • Fixed minor cloud deployment bugs when passing empty strings

  • Updated documentation on intermittent devices

  • Updated cost optimization documentation

  • Updated package dependencies in Portal UI

2.4.1#

Release Date: November 1, 2024

What’s changed#

  • Updates to documentation

  • Fixed issue in Silhouette Plot when no data for a global cluster

  • Fixed excel reader

  • Made k-FED tutorial not use public IP

  • Fixed BasePytorch model eval results

2.4.0#

Release Date: October 1, 2024

The OctaiPipe version 2.4 release brings a number of new exciting features to the user including FL on Android phones, differential for NN, evaluation of unsupervised k-FED, improved platform UI, support for device intermittency, and improved platform security.

New features#

  • Federated learning on Android phones

  • Differential privacy for neural network models using Opacus for torch

  • Evaluation of the k-FED algorithm to aide clustering, EDA and labeling

  • Added more options to handle device intermittency

Additional updates#

  • Improved Portal UI with more nuanced deployments tracking individual containers

  • Made time to setting device status to offline configurable

  • General updates to documentation and Tutorials

  • Smaller bug fixes and security improvements

Known issues#

The update to bring a more secure octaipipe user image works for all Linux based machines, except for Windows machines running Linux in WSL.

2.3.16#

Release Date: September 30, 2024

  • Upgraded jupyterlab due to package vulnerability raised on previous version

2.3.15#

Release Date: September 5, 2024

  • Fix title of k-FED tutuorial in user documentation

2.3.14#

Release Date: August 29, 2024

  • Revert ODBC driver back to 18.3.1.1 due to connection issues in 18.4.1.1

2.3.13#

Release Date: August 22, 2024

  • Update vulnerable package versions.

2.3.12#

Release Date: August 21, 2024

  • Update testing workflows.

2.3.11#

Release Date: August 21, 2024

  • Update environment for building docs.

2.3.10#

Release Date: August 20, 2024

  • Update docs to include instructions on running multiple devices locally.

2.3.9#

Release Date: August 14, 2024

  • Add missing aggregate deployment SQL view

2.3.8#

Release Date: August 13, 2024

  • Added an update_custom_step function in the development interface

  • Fixed a logging bug in FEDA

2.3.7#

Release Date: August 13, 2024

  • Fixed weighted mean formula in FedEDA

  • Fixed a small bug in FedEDA where the user was unable to only specify either ‘device_ids’ or ‘device_group’

2.3.6#

Release Date: August 6, 2024

  • Fixed a bug where “Warning” status for deployments was not included in the portal enum

2.3.5#

Release Date: July 31, 2024

  • Updated the documentation to reflect revised edge device architecture

  • Fixed a bug in the Portal where groups were not displaying their member devices

2.3.4#

Release Date: July 23, 2024

  • Fixed a bug in Federated EDA where device level statistics are returned when only global specified

2.3.3#

Release Date: July 16, 2024

  • Documentation improvements

2.3.2#

Release Date: July 16, 2024

  • Fixed an issue where cloud resources are not torn down properly due to deprecated kubernetes resources

2.3.1#

Release Date: July 15, 2024

  • Fixed an issue with setting up MLOPS policies

2.3.0#

Release Date: July 12, 2024

The OctaiPipe version 2.3 release brings a number of new features which brings OctaiPipe to new types of devices as well as making FL more secure and robust to adversarial attacks. The new release brings you the following:

New features#

  • OctaiPipe Inference on Android phones. You can now run model inference on Android phones using the new OctaiOxide Library

  • Adversarial Fortification. A number of configurable features adding fortification against Byzantine and data poisoning attacks for all FL models

  • Save best round. You can now save the model from the FL round that performed the best on the evaluation dataset

  • Federated EDA. An exploratory data analysis suite that allows you to get metadata and column statistics from a set of devices, without moving raw data

  • Unsupervised federated learning. An implementation of the k-FED algorithm both for FL training and inference

  • Running multiple OctaiPipe devices on one machine for FL experimentation and simulation

Additional updates#

  • Added an octaipipe user docker image to make containers more secure

  • Added more efficient paging, sorting and filtering to the front end

  • Added the ability to set resource reservations and limits for OctaiPipe containers running at the edge

  • Added additional messages if an FL round fails to define where the error is coming from

  • Created a more cost-efficient and configurable Azure marketplace installation of OctaiPipe

  • Fixed a bug where FL did not re-authenticate after running for a long time

  • General updates to documentation and Tutorials

  • Smaller bug fixes and security improvements

Known issues#

The update to bring a more secure octaipipe user image works for all Linux based machines, except for Windows machines running Linux in WSL.

2.2.10#

Release Date: April 30, 2024

  • Updated documentation on Azure marketplace VM sizing

2.2.9#

Release Date: April 29, 2024

  • Fixed device_groups not being able to be assigned by themselves in FL config

  • Fixed mismatching Python versions for test, Jupyter and docker images. All now Python 3.9.19

2.2.8#

Release Date: April 23, 2024

  • VM sizes in Azure Marketplace install now configurable

2.2.7#

Release Date: April 20, 2024

  • Added benchmarking capabilities to OctaiPipe

  • Each action get it’s own deployment children record

  • Improved status updates for deployment children across policies

  • Documentation updates

2.2.6#

Release Date: April 17, 2024

  • Fixed docker compose file from device registration being http not https

  • Fixed docker compose network syntyax to latest version

  • Fixed missing tag for OctaiPipe Edge Client in docker hub

2.2.5#

Release Date: April 16, 2024

  • Device deployments are now tracked during FL experiments giving more info on active and dropped devices in debug logs.

  • FL minimum device threshold is enforced

  • Fixes a bug in logging OctaiSpec Pydantic blueprints

  • Docs updates

2.2.2#

Release Date: March 27, 2024

Bug fixes#

  • Fixed Portal path for tutorial

  • General documentation updates

2.2.1#

Release Date: March 25, 2024

Marketplace documentation#

Added how-to page for installing OctaiPipe from Azure Marketplace

2.2.0#

Release Date: March 21, 2024

Version 2.2 of OctaiPipe continues our journey enabling popular model types to be trained with a federated approach and reducing the footprint so they can run on increasingly constrained edge devices. This release contains the following major changes:

Federated XGBoost#

XGBoost (Extreme Gradient Boosting) is a powerful and popular machine learning algorithm combining multiple weak learners to create a strong predictive model. It often out-performs other algorithms in both speed and accuracy while using fewer resources and producing more explainable results than deep learning approaches.

OctaiPipe V2.2 introduces a novel approach to training XGBoost models in a federated way that minimises the number of interactions between edge devices and the Federated Learning server ensuring robustness to intermittent network connections.

A new tutorial explains how to define and run an XGBoost Federated Learning workload.

Initial support for Web Assembly#

Currently, OctaiPipe uses Docker containers to encapsulate data processing, model training and model inference workloads for execution on edge devices. This requires the edge devices to support Linux and Docker and results in edge workload containers of up to 1GB for common model types.

OctaiPipe V2.2 introduces the OctaiOxide inference image which uses Web Assembly (WASM) as a new encapsulation method for smaller and faster workloads. A new tutorial demonstrates this with SGD regressor and neural network models that are significantly smaller and faster than previously.

Improvements to Model monitoring and continuous learning#

This release builds on the MLOps capabilities added in V1.6 that enabled definition of monitoring policies for models deployed to edge devices. A new demonstration video will be released shortly showing how a policy can detect data drift and trigger a new federated learning workload before updating the model to reflect its new environment.

Support for self-service installation of OctaiPipe from the Azure marketplace#

Since V2.0, the OctaiPipe portal can be installed in customer Azure or AWS cloud environments. The Azure installation process has now been simplified so new customers can self-install OctaiPipe into their Azure subscription without the need for hands-on support. This allows organisations to rapidly evaluate how OctaiPipe can reduce costs and improve efficiency training, deploying and managing machine learning solutions on edge devices.

Continuous improvements to platform reliability, security and costs#

During every release cycle, OcaiPipe is updated to fix bugs, reduce complexity, increase security and reduce running costs. A comprehensive set of security and vulnerability tests are completed to ensure new capabilities are not degrading performance in this area. In V2.2, we have:

  • Removed the requirement to use a separate VPN connection to access the Jupyter notebooks as all OctaiPipe access is secured with a single sign on authority.

  • Moved the Portal to run in Kubernetes and optimised node sizes improving cloud agnosticism and further reducing running costs.

  • Optimised K8s node size to reduce costs.

  • Implement TLS certificate for FL communication rounds.

  • Recommend cost reduction techniques in documentation.

  • Future proof Azure deployment by moving Kubenetes to use Managed Identity.

2.1.7#

Release Date: February 1, 2024

  • Fixed documentation not being properly added to Docker image in Portal

2.1.6#

Release Date: January 31, 2024

  • Fixed bug where updating a model failed if the Portal endpoint did not have working SSL cert

2.1.5#

Release Date: January 31, 2024

  • Fixe bug where model load specs are overwritten if provided by user in FL

2.1.4#

Release Date: January 29, 2024

  • Fixed bug in NN zoo client where client did not allow uer to configure if Portal endpoint has working SSL cert

2.1.3#

Release Date: January 25, 2024

  • Fixed intermittent bug in monitoring database when service takes a long way to start

2.1.2#

Release Date: January 23, 2024

  • Model versioning also properly implemented for FL models

2.1.1#

Release Date: January 17, 2024

Fixed bug where FL workload setup connects directly to device to check column names for data to train on.

2.1.0#

Release Date: December 21, 2023

Introduction of the OctaiPipe Edge Client#

A light weight client that runs on the edge device and queries the database for incoming commands to run, such as deployments or FL training instructions

  • Improves scalability by running all commands on the device, meaning the resources of the development workspace do not slow down device deployments

  • Enhanced security by removing the need to store and use device credentials to connect to the device

  • Removed the need for ensuring devices are directly accessible from workspace through for example a VPN

API endpoint security#

The backend API endpoints have been locked down such users, devices, and the FL training processes can only access data about themselves. For example, a device can only access data about that device, and not other devices.

Introduction of OctaiPipe core and lite#

OctaiPipe core and lite allow users to run scaled down, lightweight versions of OctaiPipe, to be used on lower powered devices or in setting where minimal memory and CPU usage are of interest.

For example, using OctaiPipe Lite, a user can train a federated Sklearn model and deploy it to a device using an image 3 times smaller than previously possible.

MQTT data loading and writing#

The ability to read and write data using MQTT has been added to OctaiPipe. It is also possible to use OctaiPipe to spin up an MQTT broker on your edge devices.

Bug fixes#

  • Fixed problem when trying to log into OctaiPipe as a user from a local machine

  • Fixed issue where documentation would not build if versions of OctaiPipe and the Portal mismatched

2.0.3#

Release Date: October 4, 2023

Minor updates and bug fixes.

2.0.0#

Release Date: September 20, 2023

Web UI#

In v2.0, we have added a web front end that makes it faster and easier to manage devices, experiments, models and deployments. A new notifications page allows you to monitor ongoing deployments and experiments in one place so you can easily see when anything needs your attention.

The combination of the familiar and productive Jupyter notebook development environment and the new web management UI, further simplifies and automates the use of Federated Learning on Edge devices for OctaiPipe users.

Independent Security review#

As part of release v2, we coordinated with an external penetration testing team to ensure our platform is secure. We updated all deployed components to the latest security patches and enforced TLS certificate on our rest API. Automated penetrations tests are now part of our continuous build process.

Azure and AWS Availability#

OctaiPipe is now available in both Azure and AWS as a Platform as a Service (PaaS) solution. By delivering the federated learning server and database of devices, experiments, models, and deployments into your cloud infrastructure, it ensures you remain in control of your data and intellectual property. This further improves the data security and privacy of your system, driving data compliance.

Updated Examples and demo videos#

We have created videos showing how to use OctaiPipe for the following activities:

  • Data preprocessing

  • Database connection

  • Model inference

  • Experiment tracking

  • Federated Learning classification example

  • Federated Learning regression example

Known Issues#

The current version of Kubeflow has a UI bug resulting in notebooks displaying an error message after creation. Waiting for a few seconds and then refreshing the page should resolve the issue and display the notebook.

1.6.0#

Release Date: August 16, 2023

Introducing MLOps functionality and adding flexibility to your FL data sources.

MLOps#

  • Monitoring and automating your machine learning models in IoT settings is now seamless with your production deployments and very flexible!

  • Checkout our Tutorial - Using MLOps policies for how to get started.

usage/Tutorials/02_MLOps-policies/MLOps-policies

Breaking Changes#

  • FL Config Data Specs:

Used to be: datastore_type was the same for all devices:

input_data_specs:
  datastore_type: influxdb
  query_type: dataframe
  query_config:
    devices:
      - device: default
        query_template_path: ./configs/data/influx_query_def.txt
        start: "2022-11-10T00:00:00.000Z"
        stop: "2022-11-11T00:00:00.000Z"
      - device: FL-01
        query_template_path: ./configs/data/influx_query_1.txt
        start: "2022-11-10T00:00:00.000Z"
        stop: "2022-11-11T00:00:00.000Z"

Now : each device must have their own datastore_type:

input_data_specs:
  devices:
    - device: default
      datastore_type: influxdb
      query_type: dataframe
      query_template_path: ./configs/data/influx_query_def.txt
      query_config:
        start: "2022-11-10T00:00:00.000Z"
        stop: "2022-11-11T00:00:00.000Z"
        bucket: cmapss-bucket
        measurement: sensors-raw
        tags: {}
    - device: FL-01
      datastore_type: influxdb
      query_type: dataframe
      query_template_path: ./configs/data/influx_query_1.txt
      query_config:
        start: "2022-11-10T00:00:00.000Z"
        stop: "2022-11-11T00:00:00.000Z"
        bucket: cmapss-bucket
        measurement: sensors-raw
        tags: {}

This allows more flexibility in case your devices data storages are heterogeneous.

Known Issues#

Entering a blank custom notebook image in kubeflow, if left running can lead to permanent user 500 error. This can be resolved by deleting the notebook using kubectl. If you are using OctaiPipe from our infrastructure, contact your support specialist.

1.5.1#

Release Date: May 24, 2023

Introducing new Python Interface module, explicit device environment variables definition and other improvements.

User Interface#

  • All functionality is now assembled under one module which becomes your primary interface

  • octaideploy cli interface is deprecated, use octaipipe.deployments module instead

Env Variables Definition#

  • You can now supply a dictionary of environment variables to your devices via env key in the deployment config

  • If you don’t specify any datasource-related variables, octaipipe will, as previously, assume the variables you specify under datasources are set on the Devices

  • Limitation: for now it is impossible to give different devices different variables/variable values

Known Issues#

Entering a blank custom notebook image in kubeflow, if left running can lead to permanent user 500 error. This can be resolved by deleting the notebook using kubectl. If you are using OctaiPipe from our infrastructure, contact your support specialist.

1.4.0#

Release Date: April 02, 2023

This sees the major release of Cloud Deployment as well as a number of bug fixes and improvements.

Cloud Deployment#

  • Ability to deploy all pipeline steps to the cloud

  • Model deployment to production get API endpoint plus an option to run them periodically

  • Monitoring database to collect metrics from all of your cloud deployments

Known Issues#

Entering a blank custom notebook image in kubeflow, if left running can lead to permanent user 500 error. This can be resolved by deleting the notebook using kubectl. If you are using OctaiPipe from our infrastructure, contact your support specialist.

1.3.0#

Release Date: March 22, 2023

Release 1.2.10 is a minor release that includes bug fixes and improvements.

Minor Changes and Bug Fixes#

  • Logging for Grafana deployment now shows URL pointing to dashboard rather than Grafana main page

  • Grafana theme now defaults to white

  • Fixed a bug where an error is thrown if OCTAIPIPE_DEBUG logging is set to True

  • Added database credentials to OctaiPipe docker images

  • Moved torch installation to [local] to cut down default image size

  • Switched to using local data store for model training and evaluation tests to save time on pytests

  • Add njobs=-1 as default to sklearn random forest regressor and classifier

  • Added a “no scaling” option to base pytorch model for FL to allow for custom preprocessing step

  • Added pull request template to guide pull request creation

  • Solved an issue where DeviceClient would not be able to add a device to database if it was removed in the same session

  • ModelClient checks Experiment table for experiment before posting record with provided experiment ID. This prevent silent error of not being able to create record in database due to missing experiment record

  • Added OctaiPipe login, where credentials saved in ~/.octaipipe/oct_creds.env are loaded into environment when OctaiPipe is imported

  • Credentials in ~/.octaipipe/oct_creds.env and ~/.octaipipe/octaipipe.json are moved to FL server on deployment

Known Issues#

The current version of Kubeflow Pipelines results in an Internal Server Error when trying to retrieve outputs of a pipeline run

1.2.10#

Release Date: March 14, 2023

A small release comprising several minor fixes in code, tests and CI.

Minor Changes and Bug Fixes#

  • Stabilized Ubuntu version for the testing environment

  • Fixed a bug with train-val-test split writing out only the test data

  • Updated model types in the documentation

  • Removed hardcoding secrets in tests

  • Fixed a bug with experiment record not creating if FL fails

  • Changed default namespace for the deployments to colab

  • Secured OctaiClient calls in setup.py

  • pem files for device connection are now saved to their own folder in ~/.octaipipe

Known issues#

Latest Kubeflow pipelines version results in Internal Server Error when trying to pull pipeline output directly to Python/Jupyter Notebook using octaikube.utils.get_outputs()

1.2.2#

Release Date: February 1, 2023

Minor Changes and Bug Fixes#

  • Fixed a bug used to assume ‘start’ field in the query_config, which is incorrect for local source and other non-influx sources

1.2.1#

Release Date: December 23, 2022

This is a minor release that includes bug fixes and improvements.

Minor Changes and Bug Fixes#

  • Pem files that are used for device connection are stored.

  • Experiment record is now created even if the experiment fails

  • Fixed train-val-test splitting for local files

  • Torch added to setup

Known Issues#

Cannot return pipeline output directly in Python/Jupyter Notebook using ock.utils.get_outputs() due to internal Kubeflow error

1.2.0#

Release Date: December 22, 2022

In this release we are thrilled to introduce Federated Learning.

Federated Learning#

  • OctaiFL and associated classes implemented.

  • OctaiFL.run() sends client images to selected devices using OctaiDeploy and spins up an aggregation server in kubernetes.

  • In current version the run() method will wait until the training is done and then tear down resources.

  • Experimentation concept is added to OctaiClient, each FL run creates an experiment record with linked models.

  • Evaluation table was created to store model evaluation metrics status after each training cycle.

Minor Changes and Bug Fixes#

  • Feature lagger target label hotfix

  • Model client find id hotfix

  • Now uses a python3.9 virtual environment

  • Removal of prophet

  • Added sleep time to pytest

  • Requests retry hotfix

  • Removed mlprodict and pyquicksetup

  • Updated readme for xgboost

  • Added workflow and related dockerfile

  • Version for docs taken from env variable

  • Odbc build hotfix

  • Xgboost feature names hotfix

  • Pip install local for xgboost

  • Log file size limit dix

1.1.0#

Release Date: November 28, 2022

In this release we are excited to introduce AutoML pipelines. We also focus on improved security and deployment features.

AutoML#

New features and improvements#

  • Security enhancements to OctaiClient with API endpoints secured so that only users registered in an organization and t-dab admins can access.

  • Introduction of AutoML pipelines to OctaiPipe core.

  • Improved Custom step integration with OctaiClient.

Minor Changes and Bug Fixes#

  • Added more standard models into OctaiPipe Library

  • Fixed bugs with model filenames and logging in ModelBase

  • Other minor bugfixes

1.0.4#

Release Date: October 31, 2022

New features and improvements#

  • Introduction of pyproject.toml taking care of build dependencies installation

  • Hotfix in model_client.find_previous_version()

  • Hotfixes in test cases

1.0.0#

Release Date: October 20, 2022

The 1.0 release is the first step towards producing a strong IoT ML market offering. The MVP consists of the development platform, allowing us to develop, deploy and monitor ML solutions, with deployment available to edge or cloud environments.

The first users of the platform are T-DAB data scientists, using it on internal projects. This will provide valuable feedback to the development team for bug fixes, performance improvements and new features.

Core Pipeline Steps:#

  • Preprocessing

  • Feature Engineering

  • Model Training

  • Model Inference

  • Data Drift

Edge#

  • Edge Device registration & infrastructure installation

  • Running Pipelines on the edge devices

  • Deploying Models to Edge Devices

  • Grafana monitoring dashboard at the edge

Cloud#

  • Running Pipelines in Kubeflow (using OctaiKube)

  • Grafana monitoring dashboard on the cloud

OctaiClient Metadata#

  • Recording Model parameters to the OctaiClient DB

  • Recording deployment parameters to the OctaiClient DB