Device assignment allows a VM to have direct access to HW without host/hyp intervention. AVF uses vfio-platform for device assignment, and host kernel support is required.
This document explains how to setup and launch VM with device assignments.
For device assignment, a VM device assignment DTBO (a.k.a. VM DTBO) is required. VM DTBO is a device tree overlay which describes all assignable devices information. Information includes physical reg, IOMMU, device properties, and dependencies.
VM DTBO allows to pass extra properties of assignable platform devices to the VM (which can't be discovered from the HW) while keeping the VMM device-agnostic.
When the host boots, the bootloader provides VM DTBO to both Android and pvmfw.
When a VM boots, the VMM selectively applies the DTBO based from provided labels, describing the assigned devices.
VM DTBO should be included in the dtbo partition. It should be in its own entry, and not together with any host OS's. See DTB/DTBO Paritions for partition format.
DTBO is compiled from device tree source (DTS) with dtc tool. DTBO syntax explains basic syntax of DTS.
Here are details and requirements:
VM DTBO should describe assignable devices and their labels.
&{/}, so it can be overlaid onto VM DT. Assignable devices should be backed by physical device.&{/}) to prevent unexpected modification of VM DT.VM DTBO should describe a /host node which describes physical devices and physical IOMMUs. The /host node only describes information for verification of assigned devices, and wouldn't be applied to VM DT. Here are details:
<android,pvmfw,token> property. The property describes the IOMMU token. An IOMMU token is a hypervisor-specific <u64> which uniquely identifies a physical IOMMU. IOMMU token must be constant across the VM boot for provisioning by pvmfw remains valid. The token must be kept up-to-date across hypervisor updates.#iommu-cells = <1>)#iommu-cells values aren't supported for now.<android,pvmfw,target> property that references an overlayable node. The overlayable node contains the properties that would be included in VM DT.<reg> property to provide physical regions.<iommus> property. The property is a prop-encoded-array and contains a number of (iommu phandle, SID) pairs.VM DTBO may have dependencies via phandle references. When a device node is assigned, dependencies of the node are also applied to VM DT.
When dependencies are applied, siblings or children nodes of dependencies are ignored unless explicitly referenced.
Here's a simple example device tree source with four assignable devices nodes.
/dts-v1/; /plugin/; / { // host node describes physical devices and IOMMUs, and wouldn't be applied to VM DT host { #address-cells = <0x2>; #size-cells = <0x1>; rng { reg = <0x0 0x12f00000 0x1000>; iommus = <&iommu0 0x3>; android,pvmfw,target = <&rng>; }; light { reg = <0x0 0x00f00000 0x1000>, <0x0 0x00f10000 0x1000>; iommus = <&iommu1 0x4>, <&iommu2 0x5>; android,pvmfw,target = <&light>; }; led { reg = <0x0 0x12000000 0x1000>; iommus = <&iommu1 0x3>; android,pvmfw,target = <&led>; }; bus0 { #address-cells = <0x1>; #size-cells = <0x1>; backlight { reg = <0x300 0x100>; android,pvmfw,target = <&backlight>; }; }; iommu0: iommu0 { #iommu-cells = <0x1>; android,pvmfw,token = <0x0 0x12e40000>; }; iommu1: iommu1 { #iommu-cells = <0x1>; android,pvmfw,token = <0x0 0x40000>; }; iommu2: iommu2 { #iommu-cells = <0x1>; android,pvmfw,token = <0x0 0x50000>; }; }; }; // Beginning of the assignable devices. Assigned devices would be applied to VM DT &{/} { // We only allows to overlay to root node rng: rng { compatible = "android,rng"; android,rng,ignore-gctrl-reset; }; light: light { compatible = "android,light"; version = <0x1 0x2>; }; led: led { compatible = "android,led"; prop = <0x555>; }; bus0 { backlight: backlight { compatible = "android,backlight"; android,backlight,ignore-gctrl-reset; }; }; };
If you compile the above with dtc -@, then you‘ll get __symbols__ for free. __symbol__ has label of nodes, and it’s required for the next step.
// generated __symbols__ __symbols__ { iommu0 = "/host/iommu0"; iommu1 = "/host/iommu1"; iommu2 = "/host/iommu2"; rng = "/fragment@rng/__overlay__/rng"; light = "/fragment@sensor/__overlay__/light"; led = "/fragment@led/__overlay__/led"; backlight = "/fragment@backlight/__overlay__/bus0/backlight"; };
AVF requires assignable device information to unbind from the host device driver and bind to VFIO driver. The information should be provided in an XML file at /vendor/etc/avf/assignable_devices.xml.
Here's example.
<devices> <device> <kind>sensor</kind> <dtbo_label>light</dtbo_label> <sysfs_path>/sys/bus/platform/devices/16d00000.light</sysfs_path> </device> </devices>
<kind>: Device kind. Currently only used for debugging purposes and not used for device assignment.<dtbo_label>: Label in the VM DTBO (i.e. symbol in __symbols__). Must be non-empty and unique in the XML.<sysfs_path>: Sysfs path of the device in host, used to bind to the VFIO driver. Must be non-empty and unique in the XML.In order to query list of the devices that can be assigned to a pVM, run the following command:
adb shell /apex/com.android.virt/bin/vm info
All supported assignable devices will be located under the “Assignable devices:” section of the output.
Bootloader should provide VM DTBO to both Android and pvmfw.
Bootloader should provide the VM DTBO index with sysprop ro.boot.hypervisor.vm_dtbo_idx.. DTBO index represents DTBO location in dtbo.img.
For protected VM, bootloader must provide VM DTBO to the pvmfw. pvmfw sanitizes incoming device tree with the VM DTBO.
For more detail about providing VM DTBO in pvmfw, see: pvmfw/README.md
We don't support client API yet in Android V, but you can use CLI to test device assignment. Note that host kernel support is required.
Specify --devices ${sysfs_path} when booting VM. The parameter can be repeated multiple times for specifying multiple devices.
Here's an example:
adb shell /apex/com.android.virt/bin/vm run-microdroid --devices /sys/bus/platform/devices/16d00000.light