From: Gabe Black <gabeblack@google.com>
Date: Tue, 4 Aug 2020 07:33:55 +0000 (-0700)
Subject: scons,fastmodel: Limit how many instances of simgen can run at once.
X-Git-Tag: v20.1.0.0~339
X-Git-Url: https://git.libre-soc.org/?a=commitdiff_plain;h=7a94f5cbf747c7f72914afeb72d9f8e8721a9465;p=gem5.git

scons,fastmodel: Limit how many instances of simgen can run at once.

Each instance of simgen uses a license. If there are only so many to
go around, running many instances at once could exhaust the pool of
licenses and break the build.

The number of licenses may be less than the number of regular build
steps we want to do in parallel, but may be greater than zero. To
limit them to at most n in parallel where n might be less than j
and/or more than 1, we create a group of license slots, assign simgen
invocations to a slot, and then use scons's side effect mechanism to
ensure no two invocations in the same slot run at the same time.

This may be a suboptimal packing if the commands take significantly
different amounts of time to run since the slots are preallocated and
not demand allocated, but the difference shouldn't normally matter in
practice, and scons doesn't provide a better mechanism for partially
serializing certain build steps.

Change-Id: Ifae58b48ae1b989c1915444bf7564f352f042305
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/32124
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Gabe Black <gabeblack@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
---

diff --git a/src/arch/arm/fastmodel/SConscript b/src/arch/arm/fastmodel/SConscript
index c9b08b1c1..398037354 100644
--- a/src/arch/arm/fastmodel/SConscript
+++ b/src/arch/arm/fastmodel/SConscript
@@ -36,6 +36,7 @@
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 from __future__ import print_function
+from itertools import cycle
 
 Import('*')
 
@@ -242,6 +243,10 @@ class ProjectFileParser(Grammar):
         t[0] = t[1]
 
 
+license_count = int(env['ARMLMD_LICENSE_COUNT'])
+arm_licenses = list((Value(i) for i in range(license_count)))
+license_cycle = cycle(arm_licenses)
+
 class ArmFastModelComponent(object):
     def __init__(self, project_file, *extra_deps):
         project_file = File(project_file)
@@ -312,6 +317,14 @@ class ArmFastModelComponent(object):
         sources = [project_file]
         sources.extend(extra_deps)
         env.Command(lib_nodes + self.headers, sources, simgen_action)
+        # Distribute simgen actions among ARM license slots. All actions which
+        # have a given license as a "side effect" will be serialized relative
+        # to each other, meaning the number of licenses being used concurrently
+        # will never be larger than the number of license nodes.
+        #
+        # This allocation is fixed and may not be as optimal as a dynamic one,
+        # but the difference is probably not significant.
+        env.SideEffect(next(license_cycle), lib_nodes[0])
 
     def prepare_env(self, env):
         env.Append(LIBPATH=self.libpaths)
diff --git a/src/arch/arm/fastmodel/SConsopts b/src/arch/arm/fastmodel/SConsopts
index 35e38952d..74165fe89 100644
--- a/src/arch/arm/fastmodel/SConsopts
+++ b/src/arch/arm/fastmodel/SConsopts
@@ -41,6 +41,8 @@ sticky_vars.AddVariables(
      os.environ.get('MAXCORE_HOME', '')),
     ('ARMLMD_LICENSE_FILE', 'ARM license file location',
      os.environ.get('ARMLMD_LICENSE_FILE', '')),
+    ('ARMLMD_LICENSE_COUNT',
+     'The maximum number of ARM licenses to use concurrently', 1),
     ('SIMGEN', 'simgen executable', os.environ.get('SIMGEN', default_simgen)),
 )