Home
Reading
Searching
Subscribe
Sponsors
Statistics
Posting
Contact
Spam
Lists
Links
About
Hosting
Filtering
Features Download
Marketing
Archives
FAQ
Blog
 
Gmane
From: Xiao Guangrong <xiaoguangrong <at> linux.vnet.ibm.com>
Subject: Re: [RFC PATCH 00/12] KVM: MMU: locklessly wirte-protect
Newsgroups: gmane.linux.kernel
Date: Tuesday 30th July 2013 13:11:46 UTC (over 3 years ago)
On 07/30/2013 09:01 PM, Xiao Guangrong wrote:
> Background
> ==========
> Currently, when mark memslot dirty logged or get dirty page, we need to
> write-protect large guest memory, it is the heavy work, especially, we
need to
> hold mmu-lock which is also required by vcpu to fix its page table fault
and
> mmu-notifier when host page is being changed. In the extreme cpu / memory
used
> guest, it becomes a scalability issue.
> 
> This patchset introduces a way to locklessly write-protect guest memory.
> 
> Idea
> ==========
> There are the challenges we meet and the ideas to resolve them.
> 
> 1) How to locklessly walk rmap?
> The first idea we got to prevent "desc" being freed when we are walking
the
> rmap is using RCU. But when vcpu runs on shadow page mode or nested mmu
mode,
> it updates the rmap really frequently.
> 
> So we uses SLAB_DESTROY_BY_RCU to manage "desc" instead, it allows the
object
> to be reused more quickly. We also store a "nulls" in the last "desc"
> (desc->more) which can help us to detect whether the "desc" is moved to
anther
> rmap then we can re-walk the rmap if that happened. I learned this idea
from
> nulls-list.
> 
> Another issue is, when a spte is deleted from the "desc", another spte in
the
> last "desc" will be moved to this position to replace the deleted one. If
the
> deleted one has been accessed and we do not access the replaced one, the
> replaced one is missed when we do lockless walk.
> To fix this case, we do not backward move the spte, instead, we forward
move
> the entry: when a spte is deleted, we move the entry in the first desc to
that
> position.
> 
> 2) How to locklessly access shadow page table?
> It is easy if the handler is in the vcpu context, in that case we can use
> walk_shadow_page_lockless_begin() and walk_shadow_page_lockless_end()
that
> disable interrupt to stop shadow page be freed. But we are on the ioctl
context
> and the paths we are optimizing for have heavy workload, disabling
interrupt is
> not good for the system performance.
> 
> We add a indicator into kvm struct (kvm->arch.rcu_free_shadow_page), then
use
> call_rcu() to free the shadow page if that indicator is set. Set/Clear
the
> indicator are protected by slot-lock, so it need not be atomic and does
not
> hurt the performance and the scalability.
> 
> 3) How to locklessly write-protect guest memory?
> Currently, there are two behaviors when we write-protect guest memory,
one is
> clearing the Writable bit on spte and the another one is dropping spte
when it
> points to large page. The former is easy we only need to atomicly clear a
bit
> but the latter is hard since we need to remove the spte from rmap. so we
unify
> these two behaviors that only make the spte readonly. Making large spte
> readonly instead of nonpresent is also good for reducing jitter.
> 
> And we need to pay more attention on the order of making spte writable,
adding
> spte into rmap and setting the corresponding bit on dirty bitmap since
> kvm_vm_ioctl_get_dirty_log() write-protects the spte based on the dirty
bitmap,
> we should ensure the writable spte can be found in rmap before the dirty
bitmap
> is visible. Otherwise, we cleared the dirty bitmap and failed to
write-protect
> the page.
> 
> Performance result
> ====================
> Host: CPU: Intel(R) Xeon(R) CPU           X5690  @ 3.47GHz x 12
> Mem: 36G
> 
> The benchmark i used and will be attached:

The benchmarks have been attached in this mail.

Thanks!
 
CD: 4ms