runtime-rs: add support direct volume.

As block/direct volume use similar steps of device adding, so making full use of block volume code is a better way to handle direct volume. the only different point is that direct volume will use DirectVolume and get_volume_mount_info to parse mountinfo.json from the direct volume path. That's to say, direct volume needs the help of `kata-ctl direct-volume ...`. Details seen at Advanced Topics: [How to run Kata Containers with kinds of Block Volumes] docs/how-to/how-to-run-kata-containers-with-kinds-of-Block-Volumes.md Fixes: kata-containers#5656 Signed-off-by: alex.lyn <[email protected]>
microsoft · Jun 9, 2023 · 776a15e · 776a15e
1 parent abae114
commit 776a15e
Show file tree

Hide file tree

Showing 7 changed files with 363 additions and 99 deletions.
diff --git a/docs/how-to/README.md b/docs/how-to/README.md
@@ -45,3 +45,4 @@
 - [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md)
 - [How to run Kata Containers with AMD SEV-SNP](how-to-run-kata-containers-with-SNP-VMs.md)
 - [How to use EROFS to build rootfs in Kata Containers](how-to-use-erofs-build-rootfs.md)
+- [How to run Kata Containers with kinds of Block Volumes](how-to-run-kata-containers-with-kinds-of-Block-Volumes.md)
diff --git a/docs/how-to/how-to-run-kata-containers-with-kinds-of-Block-Volumes.md b/docs/how-to/how-to-run-kata-containers-with-kinds-of-Block-Volumes.md
@@ -0,0 +1,78 @@
+# A new way for Kata Containers to use Kinds of Block Volumes
+
+> **Note:** This guide is only available for runtime-rs with default Hypervisor Dragonball.
+> Now, other hypervisors are still ongoing, and it'll be updated when they're ready.
+
+
+## Background
+
+Currently, there is no widely applicable and convenient method available for users to use some kinds of backend storages, such as File on host based block volume, SPDK based volume or VFIO device based volume for Kata Containers, so we adopt [Proposal: Direct Block Device Assignment](https://github.com/kata-containers/kata-containers/blob/main/docs/design/direct-blk-device-assignment.md) to address it.
+
+## Solution
+
+According to the proposal, it requires to use the `kata-ctl direct-volume` command to add a direct assigned block volume device to the Kata Containers runtime. 
+
+And then with the help of method [get_volume_mount_info](https://github.com/kata-containers/kata-containers/blob/099b4b0d0e3db31b9054e7240715f0d7f51f9a1c/src/libs/kata-types/src/mount.rs#L95), get information from JSON file: `(mountinfo.json)` and parse them into structure [Direct Volume Info](https://github.com/kata-containers/kata-containers/blob/099b4b0d0e3db31b9054e7240715f0d7f51f9a1c/src/libs/kata-types/src/mount.rs#L70) which is used to save device-related information. 
+
+We only fill the `mountinfo.json`, such as `device` ,`volume_type`, `fs_type`, `metadata` and `options`, which correspond to the fields in [Direct Volume Info](https://github.com/kata-containers/kata-containers/blob/099b4b0d0e3db31b9054e7240715f0d7f51f9a1c/src/libs/kata-types/src/mount.rs#L70), to describe a device. 
+
+The JSON file `mountinfo.json` placed in a sub-path `/kubelet/kata-test-vol-001/volume001` which under fixed path `/run/kata-containers/shared/direct-volumes/`. 
+And the full path looks like: `/run/kata-containers/shared/direct-volumes/kubelet/kata-test-vol-001/volume001`, But for some security reasons. it is 
+encoded as `/run/kata-containers/shared/direct-volumes/L2t1YmVsZXQva2F0YS10ZXN0LXZvbC0wMDEvdm9sdW1lMDAx`.
+
+Finally, when running a Kata Containers witch `ctr run --mount type=X, src=Y, dst=Z,,options=rbind:rw`, the `type=X` should be specified a proprietary type specifically designed for some kind of volume. 
+
+Now, supported types: 
+
+- `directvol` for direct volume
+- `spdkvol` for SPDK volume (TBD)
+- `vfiovol` for VFIO device based volume (TBD)
+
+
+## Setup Device and Run a Kata-Containers
+
+### Direct Block Device Based Volume
+
+#### create raw block based backend storage
+
+> **Tips:** raw block based backend storage MUST be formatted with `mkfs`.
+
+```bash
+$ sudo dd if=/dev/zero of=/tmp/stor/rawdisk01.20g bs=1M count=20480
+$ sudo mkfs.ext4 /tmp/stor/rawdisk01.20g
+```
+
+#### setup direct block device for kata-containers
+
+```json
+{
+  "device": "/tmp/stor/rawdisk01.20g", 
+  "volume_type": "directvol", 
+  "fs_type": "ext4", 
+  "metadata":"{}", 
+  "options": []
+}
+```
+
+```bash
+$ sudo ./kata-ctl direct-volume add /kubelet/kata-direct-vol-002/directvol002 "{\"device\": \"/tmp/stor/rawdisk01.20g\", \"volume_type\": \"directvol\", \"fs_type\": \"ext4\", \"metadata\":"{}", \"options\": []}"
+$# /kubelet/kata-direct-vol-002/directvol002 <==> /run/kata-containers/shared/direct-volumes/W1lMa2F0ZXQva2F0YS10a2F0DAxvbC0wMDEvdm9sdW1lMDAx
+$ cat W1lMa2F0ZXQva2F0YS10a2F0DAxvbC0wMDEvdm9sdW1lMDAx/mountInfo.json 
+{"volume_type":"directvol","device":"/tmp/stor/rawdisk01.20g","fs_type":"ext4","metadata":{},"options":[]}
+```
+
+#### Run a Kata container with direct block device volume
+
+```bash
+$ # type=disrectvol,src=/kubelet/kata-direct-vol-002/directvol002,dst=/disk002,options=rbind:rw
+$sudo ctr run -t --rm --runtime io.containerd.kata.v2 --mount type=directvol,src=/kubelet/kata-direct-vol-002/directvol002,dst=/disk002,options=rbind:rw "$image" kata-direct-vol-xx05302045 /bin/bash
+```
+
+
+### SPDK Device Based Volume
+
+TBD
+
+### VFIO Device Based Volume
+
+TBD
diff --git a/src/runtime-rs/Cargo.lock b/src/runtime-rs/Cargo.lock
diff --git a/src/runtime-rs/crates/resource/src/rootfs/block_rootfs.rs b/src/runtime-rs/crates/resource/src/rootfs/block_rootfs.rs
@@ -10,7 +10,10 @@ use agent::Storage;
 use anyhow::{anyhow, Context, Result};
 use async_trait::async_trait;
 use hypervisor::{
-    device::{device_manager::DeviceManager, DeviceConfig},
+    device::{
+        device_manager::{do_handle_device, DeviceManager},
+        DeviceConfig, DeviceType,
+    },
     BlockConfig,
 };
 use kata_types::mount::Mount;
@@ -46,18 +49,10 @@ impl BlockRootfs {
             ..Default::default()
         };
 
-        let device_id = d
-            .write()
-            .await
-            .new_device(&DeviceConfig::BlockCfg(block_device_config.clone()))
-            .await
-            .context("failed to create deviec")?;
-
-        d.write()
+        // create and insert block device into Kata VM
+        let device_info = do_handle_device(d, &DeviceConfig::BlockCfg(block_device_config.clone()))
             .await
-            .try_add_device(device_id.as_str())
-            .await
-            .context("failed to add deivce")?;
+            .context("do handle device failed.")?;
 
         let mut storage = Storage {
             fs_type: rootfs.fs_type.clone(),
@@ -66,17 +61,11 @@ impl BlockRootfs {
             ..Default::default()
         };
 
-        // get complete device information
-        let dev_info = d
-            .read()
-            .await
-            .get_device_info(device_id.as_str())
-            .await
-            .context("failed to get device info")?;
-
-        if let DeviceConfig::BlockCfg(config) = dev_info {
-            storage.driver = config.driver_option;
-            storage.source = config.virt_path;
+        let mut device_id: String = "".to_owned();
+        if let DeviceType::Block(device) = device_info {
+            storage.driver = device.config.driver_option;
+            storage.source = device.config.virt_path;
+            device_id = device.device_id;
         }
 
         Ok(Self {