At Private AI, we are building a privacy suite centered around personally identifiable information (PII) detection and remediation in unstructured data, such as text. Users interact with our system via a REST API, but what makes us different is that we distribute our system as a container that our customers themselves run. It’s counterintuitive to send data to a 3rd party to preserve privacy, many of which reserve rights to said data for “service improvement”. That being said, he need to make sure to have the required container resources for it to work seamlessly.
To distribute our software, we rely heavily on Docker, which we find to be an amazing tool that allows developers to streamline their development lifecycle by containerizing their applications with their runtime. This results in a portable and lightweight container that feels like a Virtual Machine (VM) and that is expected to run the same way on any device.
The emphasis here is on “feels like” because they are not VMs, they are just processes running on the host machine. Due to this reason, it is not easy to determine the amount of resources available for a given container. In most cases, you do not need to check if the container has access to enough resources since the process that is running inside the container is fairly lightweight and containers are usually deployed internally, meaning the resources they run on can be tightly controlled.
Automated Container Resource Checks: Does your container have the required resources?
Given that Private AI's container runs on customer systems it is provisioned with a wide and somewhat unpredictable variety of hardware. Anything from Apple laptops for initial container testing, to CI systems like Github Actions and of course AWS cloud instances. Not having the required resources can affect the stability of your application, resulting in difficult to debug situations where a container can be killed suddenly by the host or it will not even run in the first place. This was especially common for customers running our container using Docker Desktop, which by default only provisions 2GB RAM. In order to improve customer experience, Private AI has implemented automated container resource checks at container startups.
Container resource checks can be a difficult and time-consuming task to perform. It is pretty important for machine learning applications, due to the large compute requirement and occasional specialized hardware requirements like GPUs. In this article, we share the best tips on how you can utilize automated container resource checks to improve your application’s reliability.
Looking under the hood: namespaces and cgroups features
The two important features that dictate how to implement these automated container resource checks are “namespaces” and “cgroups”. Just to give a brief explanation of what these features are, “namespaces” are a feature of the Linux kernel that partitions kernel resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources. On the other hand, “cgroups” (or control groups) is also a feature of the Linux kernel that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.
Without getting into too much detail, Docker uses these features in order to isolate the containers from the host machine and to manage the resources that are visible to that container.
The cgroups feature is especially important as these resources are managed by files that are located under the /sys/fs/cgroup directory. This means that you can search this directory to find which resources are available.
Do you have access to enough RAM at startup?
Now that you know where to look, let’s get into two different scenarios and how to find the amount of RAM that is available for your container.
Scenario 1
Let’s look at a container that is started up by the following command:
`docker run -m 2G -it ubuntu`
This will start an Ubuntu container for us in interactive mode and it will limit its memory to 2 gigabytes. As the user, we know that the memory is limited to 2 gigabytes but how can your application find this out and determine if it should continue running or stop its execution?
This is where the “cgroups” feature comes in handy. The container is able to access its cgroups files during runtime. Therefore, we can easily read the contents of /sys/fs/cgroup/memory/memory.limit_in_bytes in order to find the memory limit that is put on the container. In this example, we see that the contents of this file are “2147483648” bytes, which is equal to 2 gigabytes.
You can easily automate your resource check solution by reading the contents of this file at startup and then compare it with the minimum requirements you have for your application.
Scenario 2
On the other hand, let’s remove the “-m 2G” flag from the command in scenario 1 and see how we can automate resource checks in this case.
Here, we can again try to read the contents of /sys/fs/cgroup/memory/memory.limit_in_bytes to determine the memory limit. However, when we do this, we see that it is either a huge number that does not make any sense or it just says “max”. How can we automate memory checks in this case?
This is where Python’s psutil library comes in. This library implements many useful process and system utilities that can be used for system monitoring. In this scenario, we will use psutil’s virtual_memory() function to see if the host is able to allocate enough RAM instead of looking into whether the container has any memory limits. This function will return all the information about the memory but in our case, we are interested in the available memory information. We can use this value to determine if the host can allocate enough memory for our container. To showcase how to use this module, following is a small Python script that will print the available memory of the system to the terminal:
```python<br>import psutil<br>available_memory = psutil.virtual_memory().available<br>print(available_memory)<br>```
So, to have an automated resource check in this scenario, you can implement the above Python script to check whether the host can allocate the memory by comparing the available memory against a certain memory threshold that you have.
Does the container have access to any GPUs at startup?
So far we have talked a lot in detail about how to implement automated container resource checks for memory, but we can also perform a similar resource check for GPUs as well. However, things are different when it comes to GPU checks because of how Docker containers are able to access GPUs.
First, in order to use GPUs within your container, you need to have the GPU drivers installed on the host machine. After this, you also need to install the Nvidia Container Toolkit and the Nvidia Container Runtime in order to access the GPU drivers of the host within your container. After completing these steps, we are ready to dive into how we can perform automated resource checks for GPUs.
During the development of our resource check implementation, we have noticed that the nvidia-smi command is available in containers that are started with the “--gpus” flag and not available in containers that are started without the “--gpus” flag. Since we know that nvidia-smi is only available on containers that have access to the host machine’s GPU drivers, we check the availability of this command by running this command at startup:
`command -v nvidia-smi`
Final takeaways for automated container resource checks
It can be challenging to perform automated resource checks due to the isolated nature of containers, however the improved user experience is worth it. By understanding the underlying technologies of Docker, we found a solution that covers every resource including GPUs thanks to Nvidia’s Container Runtime and Toolkit. It is important for us that our software has access to the necessary resources in order to provide the best experience to our users.
Interested in receiving more tips on automated container resource checks? Sign up for Private AI’s mailing list to get notified about the latest information about machine learning model training and deployment.
Sign up for our Community API
The “get to know us” plan. Our full product, but limited to 75 API calls per day and hosted by us.
