Run automated tests
What are automated tests?
A large part of the source code of OmniOpt2 is to make sure that everything works as expected. This code executes real test cases and looks at the results to check if they are as expected. Many things in OmniOpt2 get tested automatically to see if they work properly. All test related files are in the folder .tests.
Why would I want to run those?
OmniOpt2 is supposed to be run on a wide variety of Linux systems. Not every system specific thing can be caught, though, since I cannot test it manually on all the available Linux-distributions. If you encounter problems in OmniOpt2, I may ask you to run those tests and submit the output to me, so that I can debug it thoroughly.
You may have made a change to OmniOpt2 and want to see if it still runs and you haven't broken anything.
How to run tests?
To run all tests, which takes a lot of time, run:
./.tests/mainPossible options:
| Option | Meaning |
|---|---|
| --max_eval=(INT) | How many evaluations should be tried for each test (the lower, the faster) |
| --num_random_steps=(INT) | Number of random steps that should be tried (the lower, the faster) |
| --num_parallel_jobs=(INT) | How many parallel jobs should be started (ignored on non-sbatch-systems) |
| --run_with_coverage | Allows to use coverage instead of python3 for coverage testing unit test coverage |
| --exit_on_first_error | Exit on first error |
| --gpus=(INT) | How many GPUs you want for each worker/need to allocate an sbatch job |
| --debug | Enables debug mode |
| --no_plots | Disables plot tests |
| --quick | Only runs quick tests (faster) |
| --reallyquick | Only runs really quick tests (fastest) |
Example on the quickest useful test
When this succeeds without any errors, you can be reasonably sure that OmniOpt2 will properly do the following things under normal circumstances:
- Run a simple run (one random step and 2 steps in total, so both model, SOBOL and BOTORCH_MODULAR get tested)
- Continue a run
- Continue an already continued run
- Test the of the number of results for all these jobs
- Plot scripts create svg files that contain strings that are to be expected
- Basic documentation tests are done
./.tests/main --num_random_steps=1 --max_eval=2 --reallyquick