ext: testlib loading tests from multiple directories

We currently run regressions with the following command line

./main.py run [...] <directory>

Where <directory> is the positional argument pointing to the tests root
directory: Testlib will walk through the directory and load every
testsuite it encounters in its path.

./main.py run [...] <directory1> <directory2> ...

Allowing testlib to load tests from multiple directories will make it
possible to load out of tree regressions (living in an EXTRAS repository
for example)

JIRA: https://gem5.atlassian.net/browse/GEM5-905

Change-Id: I802d8753a18f4dfb00347252f031b5438e9be672
Signed-off-by: Giacomo Travaglini <giacomo.travaglini@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/40136
Reviewed-by: Bobby R. Bruce <bbruce@ucdavis.edu>
Maintainer: Bobby R. Bruce <bbruce@ucdavis.edu>
Tested-by: kokoro <noreply+kokoro@google.com>

diff --git a/TESTING.md b/TESTING.md
index 17aeff94c4a59e4ccc117a5c3c975bf633aa2e5a..88d1f295718ad0a445dfaad7b52a355d6ca6b8b8 100644 (file)
--- a/TESTING.md
+++ b/TESTING.md
@@ -63,6 +63,20 @@ cd tests
 The above is the *minumum* you should run before posting a patch to
 https://gem5-review.googlesource.com
 
+## Running tests from multiple directories
+
+The command line above will walk the directory tree starting from the cwd
+(tests), and it will run every test it encounters in its path. It is possible
+to specify multiple root directories by providing several positional
+arguments:
+
+```shell
+./main.py run <directory1> <directory2> [...]
+```
+
+This will load every test in directory1 and directory2 (and their
+subdirectories).
+
 ## Specifying a subset of tests to run
 
 You can use the tag query interface to specify the exact tests you want to run.

diff --git a/ext/testlib/configuration.py b/ext/testlib/configuration.py
index f2d93d631d4c24350140ec2125577f22a2b12ec1..1fffab46ffe87bd90e0e4cf7bd1c1f42f8c44e72 100644 (file)
--- a/ext/testlib/configuration.py
+++ b/ext/testlib/configuration.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2020 ARM Limited
+# Copyright (c) 2020-2021 ARM Limited
 # All rights reserved
 #
 # The license below extends only to copyright in the software and shall
@@ -493,10 +493,11 @@ def define_common_args(config):
     # A list of common arguments/flags used across cli parsers.
     common_args = [
         Argument(
-            'directory',
-            nargs='?',
-            default=os.getcwd(),
-            help='Directory to start searching for tests in'),
+            'directories',
+            nargs='*',
+            default=[os.getcwd()],
+            help='Space separated list of directories to start searching '
+                 'for tests in'),
         Argument(
             '--exclude-tags',
             action=StorePositionalTagsAction,
@@ -646,7 +647,7 @@ class RunParser(ArgParser):
 
         common_args.uid.add_to(parser)
         common_args.skip_build.add_to(parser)
-        common_args.directory.add_to(parser)
+        common_args.directories.add_to(parser)
         common_args.build_dir.add_to(parser)
         common_args.base_dir.add_to(parser)
         common_args.bin_path.add_to(parser)
@@ -703,7 +704,7 @@ class ListParser(ArgParser):
             help='Quiet output (machine readable).'
         ).add_to(parser)
 
-        common_args.directory.add_to(parser)
+        common_args.directories.add_to(parser)
         common_args.bin_path.add_to(parser)
         common_args.isa.add_to(parser)
         common_args.variant.add_to(parser)
@@ -722,7 +723,7 @@ class RerunParser(ArgParser):
         super(RerunParser, self).__init__(parser)
 
         common_args.skip_build.add_to(parser)
-        common_args.directory.add_to(parser)
+        common_args.directories.add_to(parser)
         common_args.build_dir.add_to(parser)
         common_args.base_dir.add_to(parser)
         common_args.bin_path.add_to(parser)

diff --git a/ext/testlib/main.py b/ext/testlib/main.py
index d2ae5a9f6d8c71994c13aeb3eef6d035b0146b14..6087a8e598479bae922f02c0ac28b9729e12a311 100644 (file)
--- a/ext/testlib/main.py
+++ b/ext/testlib/main.py
@@ -205,7 +205,10 @@ def load_tests():
     testloader = loader_mod.Loader()
     log.test_log.message(terminal.separator())
     log.test_log.message('Loading Tests', bold=True)
-    testloader.load_root(configuration.config.directory)
+
+    for root in configuration.config.directories:
+        testloader.load_root(root)
+
     return testloader
 
 def do_list():