Clean usage of options in regressions (#8190)
authorAndrew Reynolds <andrew.j.reynolds@gmail.com>
Wed, 2 Mar 2022 18:59:56 +0000 (12:59 -0600)
committerGitHub <noreply@github.com>
Wed, 2 Mar 2022 18:59:56 +0000 (18:59 +0000)
commit4ca634218d48a79983ee789fb132d3ffa5ea872f
tree10818c13830c500e8d1878baff8efbc5a91ead33
parentd07ce1107a4431e4d3e584ae731fe19b51c19631
Clean usage of options in regressions (#8190)

We now support tester annotations, thus it is best practice not to use check-unsat-cores or check-models explicitly in the regressions.

This changes regressions to not use these options, where these options are either:

removed entirely (the benchmark was fixed in the meantime)
changed to -q (the benchmark succeeds with a warning)
changed to produce- instead of check- if the option on the benchmark is required to ensure a configuration of models/unsat-cores is run. This is done for unsat-cores on sat benchmarks and models on unsat benchmarks.
replaced by DISABLE-TESTER: model.
one spurious regression is deleted, which was identical to another + check-models.
It also makes some fixes to enable more benchmarks succeed with check-models, and adds a warning when fmf-fun is combined with check-model.
148 files changed:
src/smt/check_models.cpp
src/theory/datatypes/theory_datatypes.cpp
src/theory/quantifiers/sygus_inst.cpp
test/regress/CMakeLists.txt
test/regress/regress0/arith/bug569.smt2
test/regress/regress0/arith/integers/ackermann4.smt2
test/regress/regress0/arith/integers/issue6146-stale-vars.smt2
test/regress/regress0/arith/issue4367.smt2
test/regress/regress0/arrays/issue3813-massign-assert.smt2
test/regress/regress0/arrays/issue4414.smt2
test/regress/regress0/arrays/issue4927-unsat-cores.smt2
test/regress/regress0/arrays/issue5720.smt2
test/regress/regress0/aufbv/issue3687-check-models-small.smt2
test/regress/regress0/bug421b.smt2 [deleted file]
test/regress/regress0/bug528a.smt2
test/regress/regress0/bv/ackermann1.smt2
test/regress/regress0/bv/ackermann3.smt2
test/regress/regress0/bv/ackermann4.smt2
test/regress/regress0/bv/bv_to_int_5281.smt2
test/regress/regress0/bv/bv_to_int_bvmul2.smt2
test/regress/regress0/bv/int_to_bv_model.smt2
test/regress/regress0/bv/int_to_bv_model2.smt2
test/regress/regress0/bv/proj-issue343.smt2
test/regress/regress0/bv/test-bv_intro_pow2.smt2
test/regress/regress0/cores/issue4971-0.smt2
test/regress/regress0/cores/issue4971-1.smt2
test/regress/regress0/cores/issue4971-2.smt2
test/regress/regress0/cores/issue4971-3.smt2
test/regress/regress0/cores/issue5238.smt2
test/regress/regress0/fmf/bug-041417-set-options.cvc.smt2
test/regress/regress0/fmf/bug782.smt2
test/regress/regress0/fmf/fd-false.smt2
test/regress/regress0/fmf/issue3661-ccard-dec.smt2
test/regress/regress0/fmf/sort-infer-typed-082718.smt2
test/regress/regress0/fmf/tail_rec.smt2
test/regress/regress0/int-to-bv/basic.smt2
test/regress/regress0/int-to-bv/neg-consts.smt2
test/regress/regress0/issue5736.smt2
test/regress/regress0/issue5743.smt2
test/regress/regress0/issue5947.smt2
test/regress/regress0/issue6741.smt2
test/regress/regress0/nl/issue3003.smt2
test/regress/regress0/nl/issue3411.smt2
test/regress/regress0/nl/issue3652.smt2
test/regress/regress0/nl/issue3729-cm-solved-tf.smt2
test/regress/regress0/nl/issue3991.smt2
test/regress/regress0/nl/issue5740-2-mod00.smt2
test/regress/regress0/nl/nta/exp-neg2-unsat-unsound.smt2
test/regress/regress0/nl/nta/real-pi.smt2
test/regress/regress0/nl/sin-cos-346-b-chunk-0169.smt2
test/regress/regress0/nl/sqrt2-value.smt2
test/regress/regress0/push-pop/bug654-dd.smt2
test/regress/regress0/push-pop/issue2137.min.smt2
test/regress/regress0/push-pop/quant-fun-proc-unfd.smt2
test/regress/regress0/quantifiers/issue5693-prenex.smt2
test/regress/regress0/sep/issue3720-check-model.smt2
test/regress/regress0/sep/nemp.smt2
test/regress/regress0/sep/nspatial-simp.smt2
test/regress/regress0/sep/simple-080420-const-sets.smt2
test/regress/regress0/sep/skolem_emp.smt2
test/regress/regress0/sep/wand-crash.smt2
test/regress/regress0/seq/issue4370-bool-terms.smt2
test/regress/regress0/seq/nth-update.smt2
test/regress/regress0/seq/rev.smt2
test/regress/regress0/seq/seq-nth.smt2
test/regress/regress1/bug520.smt2
test/regress/regress1/bv2int-isabelle.smt2
test/regress/regress1/datatypes/issue-variant-dt-zero.smt2
test/regress/regress1/decision/issue5785.smt2
test/regress/regress1/fmf-fun-dbu.smt2
test/regress/regress1/fmf/bug651.smt2
test/regress/regress1/fmf/bug723-irrelevant-funs.smt2
test/regress/regress1/fmf/bug764.smt2
test/regress/regress1/fmf/constr-ground-to.smt2
test/regress/regress1/fmf/fmf-fun-no-elim-ext-arith.smt2
test/regress/regress1/fmf/fmf-fun-no-elim-ext-arith2.smt2
test/regress/regress1/fmf/issue916-fmf-or.smt2
test/regress/regress1/ho/nested_lambdas-AGT034^2.smt2
test/regress/regress1/ho/nested_lambdas-sat-SYO056^1-delta.smt2
test/regress/regress1/issue3990-sort-inference.smt2
test/regress/regress1/model-blocker-values.smt2
test/regress/regress1/nl/approx-sqrt.smt2
test/regress/regress1/nl/bug698.smt2
test/regress/regress1/nl/exp-approx.smt2
test/regress/regress1/nl/exp-soundness-bound.smt2
test/regress/regress1/nl/issue3300-approx-sqrt-witness.smt2
test/regress/regress1/nl/issue3647.smt2
test/regress/regress1/nl/issue4791-llr.smt2
test/regress/regress1/nl/sin1-deq-sat.smt2
test/regress/regress1/nl/sin1-sat.smt2
test/regress/regress1/nl/solve-eq-small-qf-nra.smt2
test/regress/regress1/push-pop/quant-fun-proc-unmacro.smt2
test/regress/regress1/push-pop/quant-fun-proc.smt2
test/regress/regress1/quantifiers/issue3316.smt2
test/regress/regress1/quantifiers/issue3317.smt2
test/regress/regress1/quantifiers/issue3628.smt2
test/regress/regress1/quantifiers/issue3664.smt2
test/regress/regress1/quantifiers/issue3765.smt2
test/regress/regress1/quantifiers/issue4620-erq-witness-unsound.smt2
test/regress/regress1/quantifiers/issue4685-wrewrite.smt2
test/regress/regress1/quantifiers/issue5469-aext.smt2
test/regress/regress1/quantifiers/issue5470-aext.smt2
test/regress/regress1/quantifiers/issue5766-wrong-sel-trigger.smt2
test/regress/regress1/quantifiers/issue6607-witness-te.smt2
test/regress/regress1/quantifiers/issue6638-sygus-inst.smt2
test/regress/regress1/quantifiers/issue6775-vts-int.smt2
test/regress/regress1/quantifiers/qid-debug-inst.smt2
test/regress/regress1/quantifiers/quaternion_ds1_symm_0428.fof.smt2
test/regress/regress1/quantifiers/sygus-inst-ufnia-sat-t3_rw1505.smt2
test/regress/regress1/quantifiers/symmetric_unsat_7.smt2
test/regress/regress1/sep/crash1220.smt2
test/regress/regress1/sep/dispose-list-4-init.smt2
test/regress/regress1/sep/fmf-nemp-2.smt2
test/regress/regress1/sep/loop-1220.smt2
test/regress/regress1/sep/sep-neg-1refine.smt2
test/regress/regress1/sep/sep-neg-nstrict2.smt2
test/regress/regress1/sep/sep-neg-simple.smt2
test/regress/regress1/sep/sep-neg-swap.smt2
test/regress/regress1/sep/sep-nterm-again.smt2
test/regress/regress1/sep/sep-simp-unc.smt2
test/regress/regress1/sep/simple-neg-sat.smt2
test/regress/regress1/sep/wand-0526-sat.smt2
test/regress/regress1/sep/wand-false.smt2
test/regress/regress1/sep/wand-nterm-simp.smt2
test/regress/regress1/sep/wand-nterm-simp2.smt2
test/regress/regress1/sep/wand-simp-sat.smt2
test/regress/regress1/sep/wand-simp-sat2.smt2
test/regress/regress1/strings/issue3657-unexpectedUnsatCVC4.smt2
test/regress/regress1/strings/issue5611-deq-norm-emp.smt2
test/regress/regress1/strings/strings-leq-trans-unsat.smt2
test/regress/regress1/sygus/issue3648.smt2
test/regress/regress2/bug765.smt2
test/regress/regress2/bv_to_int_bitwise.smt2
test/regress/regress2/bv_to_int_bvuf_to_intuf_smtlib.smt2
test/regress/regress2/bv_to_int_mask_array_1.smt2
test/regress/regress2/bv_to_int_quantifiers_bvand.smt2
test/regress/regress2/fp/issue7056.smt2
test/regress/regress2/issue3687-check-models.smt2
test/regress/regress2/nl/ufnia-factor-open-proof.smt2
test/regress/regress2/push-pop/DRAGON_4_e2_2799_e3_1915.lus.ic3.1.min.smt2
test/regress/regress2/quantifiers/exp-trivially-dd3.smt2
test/regress/regress2/strings/issue3203.smt2
test/regress/regress3/bv_to_int_and_or.smt2
test/regress/regress3/bv_to_int_check_bvsge_bvashr0_4bit.smt2.minimized.smt2
test/regress/regress3/bv_to_int_input_mouser_detect.c.smt2.minimized.smt2
test/regress/regress3/bv_to_int_quant2.smt2
test/regress/regress3/issue4476-ext-rew.smt2
test/regress/regress3/issue4714.smt2