Infrastructure Cost Management#

Azure#

How to clean up Kubernetes resources#

OctaiPipe is designed to not leave hanging deployments and workloads behind, however things might go wrong in an unexpected way, so it’s important to know how you hand cleanup your kubernetes environment. Luckily, it can be achieved with a couple of simple commands.

Note

It’s important to note that the resources that you will see are shared across your organisation so make sure to check in with your colleagues before removing elements you do not recognise.

  1. See what is currently running in k8s

In your jupyter server’s terminal run

kubectl get all -n colab

That should return a list of everything that is currently running. You can notice the list is split into subsections based on resource type: jobs, services, pods, etc. You don’t need to know the details about these terms, but if you would like to know more, you can chekout k8s documentation.

In ideal situation, apart from intentional workloads there will only be monitoring database server that’s running, which will look something like this:

NAME                                                             READY   STATUS             RESTARTS   AGE
pod/monitor-database-demov22-production-image-688d7b65bf-tq8w2   1/1     Running            0          32h

NAME                                                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
service/monitor-database-demov22-production-service   LoadBalancer   X.X.X.X         X.X.X.X         8086:30080/TCP   2d9h

NAME                                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/monitor-database-demov22-production-image   1/1     1            1           2d9h

Other resources that you might see there are FL servers, monitoring observers and triggers and cloud deployments. If you are sure there are resources that shouldn’t be there, go to step 2.

2. Delete the resources You can copy the resource name from the terminal and run the following command to delete it:

kubectl delete -n colab <resource name>

For example:

kubectl delete -n colab pod/ct-4869f-policy1-trigger-cloud-28557830-h54n4

Note

There is an order for deleting resources:

  1. deployments

  2. jobs

  3. pods

  4. services

If you dont’ follow the order, some of the resources might be recreated after you delete them.

And in case there are several resources, you can just queue them at the end of the command separated by space.

kubectl delete -n colab <resource 1> <resource 2> <resource 3>

3. Delete unattached volumes A specific resource type that that is not shown by kubectl get all -n colab are volumes (pvc’s). To see the list of the volumes, run

kubectl get pvc -n colab

You should see the result lie below

NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
monitor-database-demov22-production-data   Bound    pvc-cd4901d1-4724-4312-a2c8-6906a76043ac   5Gi        RWO            default        2d18h
test-influxdb-db-data                      Bound    pvc-c037c4f1-9d05-4c19-a625-b1779d8a97d0   5Gi        RWO            default        2d18h

In example above both volumes are Bound, that means they are attached to other running resources and should be left as ther are. For those volumes that are not bound, delete them via this command:

kubectl delete pvc -n colab <volume name>

  1. Let us know if you have any issues

If you experience a recurring issues of dangling workloads or issues with deleting accidental resources, don’t hesitate to contact us at support@octaipipe.ai

How to remove unattached disks#

In order to manage costs, it is important to remove any unattached disks from your Azure account. Disks can be created when a new Jupyter Notebook is spun up in OctaiPipe and are not automatically deleted when the instance is deleted.

Login to Azure Portal - Navigate to the Azure Portal: https://portal.azure.com <https://portal.azure.com> - Sign in using your Azure account credentials.

Navigate to Disks - In the main search box at the top, type “Disks” and select it from the dropdown.

Filter Unattached Disks - In the Disks pane, you will see a list of all the disks. - From the filter options, select “Disk state” and choose “Unattached.”

Delete - Click into each disk and select “Delete” from the top menu.

Precautions - Ensure that there’s no valuable data on the disk that might be required in the future. - It’s recommended to have backups or snapshots of disks before deletion. - The disks attached to your Jupyter Notebook Servers will come up as unattached disks if the Notebook server is Stopped. Make sure to not delete the Notebook server disks as it will mean the Notebook server cannot restart and your data will be lost.