From d4e9a345900fea2ff92731538d733cc6afbe7a4e Mon Sep 17 00:00:00 2001 From: Hsuan Hsu Date: Sun, 24 May 2020 00:41:58 +0800 Subject: [PATCH] cpu-kvm,arch-arm: Improve KvmCPU tick event scheduling The memory-mapped timer emulated by gem5 is driven by the underlying gem5 tick, which means that we must align the tick with the host time to make the timer interrupt fire at a nearly native rate. In each KVM execution round, the number of ticks incremented is directly calculated from the number of instructions executed. However, when a guest CPU switches to idle state, KVM seems to stay in kernel- space until the POSIX timer set up in user-space raises an expiration signal, instead of trapping to user-space immediately; and somehow the instruction count is just too low to match the elapsed host time. This makes the gem5 tick increment very slowly when the guest is idle and drastically slow down workloads being sensitive to the guest time which is driven by timer interrupt. Before switching to KVM to execute the guest code, gem5 programs the POSIX timer to expire according to the remaining ticks before the next event in the event queue. Based on this, we can come up with the following solution: If KVM returns to user-space due to POSIX timer expiration, it must be time to process the next gem5 event, so we just fast-forward the tick (by scheduling the next CPU tick event) to that event directly without calculating from the instruction count. There is one more related issue needed to be solved. The KVM exit reason, KVM_EXIT_INTR, was treated as the case where the KVM execution was disturbed by POSIX timer expiration. However, there exists a case where the exit reason is KVM_EXIT_INTR but the POSIX timer has not expired. Its cause is still unknown, but it can be observed via the "old_value" argument returned by timer_settime() when disarming the POSIX timer. In addition, it seems to happen often when a guest CPU is not in idle state. When this happens, the above tick event scheduling incorrectly treats KVM_EXIT_INTR as POSIX timer expiration and fast- forwards the tick to process the next event too early. This makes the guest feel external events come too fast, and will sometimes cause trouble. One example is the VSYNC interrupt from HDLCD. The guest seems to get stuck in VSYNC handling if the KVM CPU is not given enough time between each VSYNC interrupt to complete a service. (Honestly I did not dig in to see how the guest handled the VSYNC interrupt and how the above situation became trouble. I just observed from the debug trace of GIC & HDLCD & timer, and made this conclusion.) This change also uses a workaround to detect POSIX timer expiration correctly to make the guest work with HDLCD. JIRA: https://gem5.atlassian.net/browse/GEM5-663 Change-Id: I6159238a36fc18c0c881d177a742d8a7745a23ca Signed-off-by: Hsuan Hsu Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/30919 Reviewed-by: Andreas Sandberg Maintainer: Andreas Sandberg Tested-by: kokoro --- src/cpu/kvm/base.cc | 9 +++++++-- src/cpu/kvm/timer.cc | 15 ++++++++++++--- src/cpu/kvm/timer.hh | 5 +++++ 3 files changed, 24 insertions(+), 5 deletions(-) diff --git a/src/cpu/kvm/base.cc b/src/cpu/kvm/base.cc index 18aead84e..0afab1e84 100644 --- a/src/cpu/kvm/base.cc +++ b/src/cpu/kvm/base.cc @@ -691,8 +691,13 @@ BaseKvmCPU::tick() } // Schedule a new tick if we are still running - if (_status != Idle && _status != RunningMMIOPending) - schedule(tickEvent, clockEdge(ticksToCycles(delay))); + if (_status != Idle && _status != RunningMMIOPending) { + if (_kvmRun->exit_reason == KVM_EXIT_INTR && runTimer->expired()) + schedule(tickEvent, clockEdge(ticksToCycles( + curEventQueue()->nextTick() - curTick() + 1))); + else + schedule(tickEvent, clockEdge(ticksToCycles(delay))); + } } Tick diff --git a/src/cpu/kvm/timer.cc b/src/cpu/kvm/timer.cc index e7b185f5d..538fd569f 100644 --- a/src/cpu/kvm/timer.cc +++ b/src/cpu/kvm/timer.cc @@ -122,10 +122,19 @@ PosixKvmTimer::disarm() struct itimerspec ts; memset(&ts, 0, sizeof(ts)); - DPRINTF(KvmTimer, "Disarming POSIX timer\n"); - - if (timer_settime(timer, 0, &ts, NULL) == -1) + if (timer_settime(timer, 0, &ts, &prevTimerSpec) == -1) panic("PosixKvmTimer: Failed to disarm timer\n"); + + DPRINTF(KvmTimer, "Disarmed POSIX timer: %is%ins left\n", + prevTimerSpec.it_value.tv_sec, + prevTimerSpec.it_value.tv_nsec); +} + +bool +PosixKvmTimer::expired() +{ + return (prevTimerSpec.it_value.tv_nsec == 0 && + prevTimerSpec.it_value.tv_sec == 0); } Tick diff --git a/src/cpu/kvm/timer.hh b/src/cpu/kvm/timer.hh index 32222e53b..376ba7c1a 100644 --- a/src/cpu/kvm/timer.hh +++ b/src/cpu/kvm/timer.hh @@ -93,6 +93,9 @@ class BaseKvmTimer * signals upon timeout. */ virtual void disarm() = 0; + virtual bool expired() { + return true; + } /** * Determine the resolution of the timer in ticks. This method is @@ -193,6 +196,7 @@ class PosixKvmTimer : public BaseKvmTimer void arm(Tick ticks); void disarm(); + bool expired() override; protected: Tick calcResolution(); @@ -200,6 +204,7 @@ class PosixKvmTimer : public BaseKvmTimer private: clockid_t clockID; timer_t timer; + struct itimerspec prevTimerSpec; }; /** -- 2.30.2