[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [virtio-dev] [PATCH 0/2] introduce virtio-ism: internal shared memory device
å 2022/10/18 16:33, Gerry åé:
2022å10æ18æ 14:54ïJason Wang <jasowang@redhat.com> åéïOn Mon, Oct 17, 2022 at 8:31 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:On Mon, 17 Oct 2022 16:17:31 +0800, Jason Wang <jasowang@redhat.com> wrote:Adding Stefan.On Mon, Oct 17, 2022 at 3:47 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:Hello everyone, # BackgroundNowadays, there is a common scenario to accelerate communication between different VMs and containers, including light weight virtual machine based containers. One way to achieve this is to colocate them on the same host. However, the performance of inter-VM communication through network stack is not optimal and may also waste extra CPU cycles. This scenario has been discussedmany times, but still no generic solution available [1] [2] [3].With pci-ivshmem + SMC(Shared Memory Communications: [4]) based PoC[5], We found that by changing the communication channel between VMs from TCP to SMCwith shared memory, we can achieve superior performance for a common socket-based application[5]: Â- latency reduced by about 50% Â- throughput increased by about 300% Â- CPU consumption reduced by about 50%Since there is no particularly suitable shared memory management solution matches the need for SMC(See ## Comparison with existing technology), and virtio is the standard for communication in the virtualization world, we want to implement a virtio-ism device based on virtio, which can support on-demand memory sharing across VMs, containers or VM-container. To match the needs of SMC,the virtio-ism device need to support:1. Dynamic provision: shared memory regions are dynamically allocated andÂÂprovisioned. 2. Multi-region management: the shared memory is divided into regions,ÂÂand a peer may allocate one or more regions from the same shared memoryÂÂdevice.3. Permission control: The permission of each region can be set seperately.Looks like virtio-ROCE https://lore.kernel.org/all/20220511095900.343-1-xieyongji@bytedance.com/T/ and virtio-vhost-user can satisfy the requirement?# Virtio ism deviceISM devices provide the ability to share memory between different guests on a host. A guest's memory got from ism device can be shared with multiple peers at the same time. This shared relationship can be dynamically created and released.The shared memory obtained from the device is divided into multiple ism regions for share. ISM device provides a mechanism to notify other ism region referrersof content update events. # Usage (SMC as example) Maybe there is one of possible use cases:1. SMC calls the interface ism_alloc_region() of the ism driver to return theÂÂlocation of a memory region in the PCI space and a token.2. The ism driver mmap the memory region and return to SMC with the token3. SMC passes the token to the connected peer 3. the peer calls the ism driver interface ism_attach_region(token) to ÂÂget the location of the PCI space of the shared memory # About hot plugging of the ism deviceÂÂHot plugging of devices is a heavier, possibly failed, time-consuming, andÂÂless scalable operation. So, we don't plan to support it for now. # Comparison with existing technology ## ivshmem or ivshmem 2.0 of QemuÂÂ1. ivshmem 1.0 is a large piece of memory that can be seen by all devices thatÂÂuse this VM, so the security is not enough.ÂÂ2. ivshmem 2.0 is a shared memory belonging to a VM that can be read-only by all ÂÂother VMs that use the ivshmem 2.0 shared memory device, which also does notÂÂmeet our needs in terms of security. ## vhost-pci and virtiovhostuserÂÂDoes not support dynamic allocation and therefore not suitable for SMC.I think this is an implementation issue, we can support VHOST IOTLB message then the regions could be added/removed on demand.1. After the attacker connects with the victim, if the attacker does notÂÂdereference memory, the memory will be occupied under virtiovhostuser. In the ÂÂcase of ism devices, the victim can directly release the reference, and theÂÂmaliciously referenced region only occupies the attacker's resourcesLet's define the security boundary here. E.g do we trust the device or not? If yes, in the case of virtiovhostuser, can we simple do VHOST_IOTLB_UNMAP then we can safely release the memory from the attacker.Thanks, Jason:) In our the design, there are several roles involved: 1) a virtio-ism-smc front-end driver 2) a Virtio-ism backend device driver and its associated vmm 3) a global device manager 4) a group of remote/peer virtio-ism backend devices/vmms 5) a group of remote/peer virtio-ism-smc front-end drivers Among which , we treat 1, 2 and 3 as trusted, 4 and 5 as untrusted.
It looks to me VIRTIO_ISM_PERM_MANAGE violates what you've described here. E.g what happens if 1 grant this permission to 5?
Because 4 and 5 are trusted, we canât guarantee that IOTLB Invalidate requests have been executed as expected.
Interesting, I wonder how this is guaranteed by ISM. Anything that can work for ISM but not IOTLB? Note that the only difference for me is the device API. We can hook anything that works for ISM to IOTLB.
Say when disconnecting an SMC connection, a malicious peer may ignore the IOTLB invalidation request and keep access the shared memory region.We have considered the IOTLB based design but encountered several issues:1) It depends on the way to provision guest vm memory. We need a memory resource descriptor to support vhost-user IOTLB messages, thus canât support anonymous memory based vm.
Hypervisor (Qemu) is free to hook IOTLB message to any kind of memory backend, isn't? E.g Qemu can choose to implement IOTLB by its own instead of forwarding it to another VM.
2) Lack of fine-grain access control of memory resource descriptor. When send a memory resource descriptor to an untrusted peer, we canât enforce region based access control. Memfd supports file level seal operations, but still lack of region based permission control. Hugetlbfs based fd doesnât support seal at all.
So in the above, you said 4 and 5 are untrusted. If yes how you can enforce regioned based access control (the memory is still mapped by the untrsuted VMM)? And again, virtio-vhost-user is not limited to memfd/hugetlbfs, it can do want you've done in your protoype (hooking to /dev/shm).
3) Lack of reliable way to reclaim granted access permissions from untrusted peers, as stated above.
It would be better to explain how this "reclaim" works.
4) How implement resource accounting. Say a vm has shared some memory regions from peers, and those peers exited unexpectedly, then those shared memory will be accounted to the victim vm, and may cause unexpected OOM.Based on the above consideration, we adopted another design and introduced the device manager to solve above issues:1) the device manager is the owner of memory buffers.
I don't see the definition "device manager" in your proposal, this needs to be clarified in both the spec and the changelog or the cover letter.
2) the device manager creates a memfd for each memory buffer/region, and configure SEALs according to requested access permissions.
Ok, but this seems not what you've implemented in your qemu prototype?
3) When a guest vm reclaims a shared memory buffer, the device manager will provision a new memfd to the guest vm.
How can this be done for the untrusted peers?
And it will take the responsibility to reclaim the old buffer from peer and eventually release the old buffer. 4) Simplify the control communication channel. Every guest vm only needs to talk with the device manager and no need to discover and communicate with other peers.
Not sure but it's better not mandate any model in the application layer. Thanks
Thanks, Gerry2. The ism device of a VM can be shared with multiple (1000+) VMs at the sameÂÂtime, which is a challenge for virtiovhostuserPlease elaborate more the the challenges, anything make virtiovhostuser different?3. The sharing relationship of ism is dynamically increased, and virtiovhostuserÂÂdetermines the sharing relationship at startup.Not necessarily with IOTLB API?4. For security issues, the device under virtiovhostuser may mmap more memory,ÂÂwhile ism only maps one region to other devicesWith VHOST_IOTLB_MAP, the map could be done per region. ThanksThanks.Thanks# Design ÂÂThis is a structure diagram based on ism sharing between two vms. ÂÂÂ|-------------------------------------------------------------------------------------------------------------|ÂÂÂ| |------------------------------------------------| ÂÂÂÂÂÂ|------------------------------------------------| | ÂÂÂ| | Guest ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂ| Guest ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| | ÂÂÂ| | ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| | ÂÂÂ| | ÂÂ---------------- ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂ---------------- ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| | ÂÂÂ| | ÂÂ| ÂÂÂdriver ÂÂÂ| ÂÂÂÂ[M1] ÂÂ[M2] ÂÂ[M3] ÂÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂ| ÂÂÂdriver ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂ[M2] ÂÂ[M3] ÂÂÂÂ| | ÂÂÂ| | ÂÂ---------------- ÂÂÂÂÂÂ| ÂÂÂÂÂ| ÂÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂ---------------- ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂ| ÂÂÂÂÂ| | ÂÂÂ| | ÂÂÂ|cq| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|map ÂÂ|map ÂÂ|map ÂÂÂ| ÂÂÂÂÂÂ| ÂÂÂ|cq| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|map ÂÂ|map ÂÂ| | ÂÂÂ| | ÂÂÂ| Â| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂ| ÂÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂÂ| Â| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂ| ÂÂÂÂÂ| | ÂÂÂ| | ÂÂÂ| Â| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ------------------- ÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂÂ| Â| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ-------------------- ÂÂÂ| | ÂÂÂ| |----|--|----------------| Âdevice memory Â|-----| ÂÂÂÂÂÂ|----|--|----------------| Âdevice memory ÂÂ|----| | ÂÂÂ| | ÂÂÂ| Â| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ------------------- ÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂÂ| Â| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ-------------------- ÂÂÂ| | ÂÂÂ| | ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| | ÂÂÂ| | ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| | ÂÂÂ| | Qemu ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂ| Qemu ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| | ÂÂÂ| |--------------------------------+---------------| ÂÂÂÂÂÂ|-------------------------------+----------------| | ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|------------------------------+------------------------| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ-------------------------- ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| M1 | ÂÂ| M2 | ÂÂ| M3 | ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ-------------------------- ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂ| ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ| ÂÂÂ| HOST ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ|ÂÂÂ--------------------------------------------------------------------------------------------------------------- # POC codeÂÂKernel: https://github.com/fengidri/linux-kernel-virtio-ism/commits/ismÂÂQemu: https://github.com/fengidri/qemu/commits/ism If there are any problems, please point them out. Hope to hear from you, thank you. [1] https://projectacrn.github.io/latest/tutorials/enable_ivshmem.html [2] https://dl.acm.org/doi/10.1145/2847562 [3] https://hal.archives-ouvertes.fr/hal-00368622/document [4] https://lwn.net/Articles/711071/[5] https://lore.kernel.org/netdev/20220720170048.20806-1-tonylu@linux.alibaba.com/T/Xuan Zhuo (2): ÂReserve device id for ISM device Âvirtio-ism: introduce new device virtio-ism content.tex ÂÂÂ| ÂÂ3 + virtio-ism.tex | 340 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 343 insertions(+) create mode 100644 virtio-ism.tex -- 2.32.0.3.g01195cf9f ---------------------------------------------------------------------To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org <mailto:virtio-dev-unsubscribe@lists.oasis-open.org> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org <mailto:virtio-dev-help@lists.oasis-open.org>--------------------------------------------------------------------- To unsubscribe, e-mail:virtio-dev-unsubscribe@lists.oasis-open.org For additional commands, e-mail:virtio-dev-help@lists.oasis-open.org
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]