✅ Run automated tests

What are automated tests?

A large part of the source code of OmniOpt2 is to make sure that everything works as expected. This code executes real test cases and looks at the results to check if they are as expected. Many things in OmniOpt2 get tested automatically to see if they work properly. All test related files are in the folder .tests.

Why would I want to run those?

OmniOpt2 is supposed to be run on a wide variety of Linux systems. Not every system specific thing can be caught, though, since I cannot test it manually on all the available Linux-distributions. If you encounter problems in OmniOpt2, I may ask you to run those tests and submit the output to me, so that I can debug it thoroughly.

You may have made a change to OmniOpt2 and want to see if it still runs and you haven't broken anything.

How to run tests?

To run all tests, which takes a lot of time, run:

./.tests/main

Possible options:

--exit_on_first_error

bool

False

--help

bool

False

--run_with_coverage

bool

False

--quick

bool

False

--reallyquick

bool

False

--superquick

bool

False

--debug

bool

False

--no_linkchecker

bool

False

--no_linter

bool

False

--skip_test_job_nr

bool

False

--skip_first_n_tests

int

--skip_worker_check

bool

False

--no_plots

bool

False

Example on the quickest useful test

When this succeeds without any errors, you can be reasonably sure that OmniOpt2 will properly do the following things under normal circumstances:

• Run a simple run (one random step and 2 steps in total, so both model, SOBOL and BOTORCH_MODULAR get tested)
• Continue a run
• Continue an already continued run
• Test the of the number of results for all these jobs
• Plot scripts create svg files that contain strings that are to be expected
• Basic documentation tests are done

./.tests/main --num_random_steps=1 --max_eval=2 --reallyquick

All test scripts

adding_jobs_to_new_job

Test if adding jobs from existing jobs to new jobs works

all_shellscripts_with_help_have_help_page

See if all bash scripts have --help.

bandit

Tests for security-related python bugs and improvement suggestions.

check_if_results_csv_has_the_right_headers

Start an OmniOpt2 run and see if the results.csv has the proper results.

check_state_file_documentation

Test if all state files that are written are documented.

continued_constraint

Testing if continuing a constrained run works.

current_runs_plot_test

Plots all current projects in runs and tests if the plot script succeeded.

deadcode

Find unreachable and unused code.

doc_has_outdated_params

Test if any documentation page has outdated options for .omniopt.py/omniopt.

find_add_arguments_with_double_quotes

Find arguments in .omniopt.py that have double quotes and fail if they do, since other script that parse it need them to be single quotes.

find_double_add_arguments_help

Find arguments added multiple times in .omniopt.py.

find_double_function_names

Find function names that are defined twice or more.

find_double_function_thats_already_in_helper

Find functions that are defined in any .py and also .helpers.py, so that the they can be moved to the .helpers.py compeletly.

find_functions_that_are_defined_multiple_times

Find functions that are defined multiple times and could be moved to .helpers.py.

find_functions_with_missing_types

Find python-scripts with missing function return types.

find_missing_env_vars_documentation

Find environment variables that change how OmniOpt works that are not documented.

find_missing_toc_in_md_files

Find help documentation files missing the Table-of-Contents.

find_typos_in_bash

Finds typos in bash scripts.

find_typos_in_gui

Find typos in the GUI (PHP).

find_typos_in_js

Find typos in JavaScript-files.

find_typos_in_php

Find typos in php scripts.

find_typos_in_python

Find typos in python scripts.

find_undocumented_exit_codes

Check if all used exit codes are documented.

find_untyped_cli_args

Find CLI arguments that are not typed properly for mypy and beartype.

find_unused_but_documented_exit_codes

Find exit codes in documentation that are not used in OmniOpt2.

find_unused_functions_in_omniopt

Find functions in .omniopt.py that are unused.

find_unused_params

find parameters that are defined in OmniOpt but never used.

find_unused_params_in_plots

Find argparse parameter for plots that go unused.

find_unused_share_functions

Find functions in the share_functions.php that are not used anywhere.

find_wrong_order_definitions

Find definitions of functions that are in the wrong order.

find_wrongly_typed_cli_args

Find CLI arguments that have the wrong types.

flake8

Runs linter 'flake8' on python.

folder_structure

Tests if all files that are in the run subfolders are documented properly.

install_via_pip

Install via github and pip and test if it works properly.

js_syntax

Check syntax of JS scripts.

linkchecker

Checks all links on the site.

lint_python

Run python-linters only.

lizard

Test scripts for cyclomatic complexity.

missing_plot_metadata

Tests if al plot scripts have the metadata for the help pages, i.e. # EXPECTED FILES and # DESCRIPTION.

multiplication_constraint

Test non-standard constraints.

mypy

Run mypy to find variables that are not typed properly.

optimization_example

Simple optimization example based on the shekel-function.

--fixed_param

bool

False

--fail_or_not

0 or 1

--random_sem

0 or 1

--nr_results

int

--help

bool

False

--debug

bool

False

optimization_example_full_search

Optimization example based on the shekel-function.

--help

bool

False

--debug

bool

False

orchestrator

Tests if the orchestrator runs properly.

--nosuccess

bool

False

--help

bool

False

--debug

bool

False

php_files

Syntax check all PHP files.

php_search

Test if the search.php compiles and delivers proper results.

phpcs

Run a PHP linter.

plots

Test all plot scripts for different jobs.

post_job_pareto_front_calculation

Run an omniopt2 job with 2 result objectives, but cancel it, and test if the pareto front is calculated correctly afterwards.

pyflakes

Run pyflakes linter on .py files.

pylint

Runs pylint linter on python files.

radon

Find functions with too large cyclomatic complexity.

share

Tests the share page for PHP-syntax errors and other simple-to-test stuff.

share_tests

Run different tests for the share functions.

shellcheck

Lints bash files.

single_quotes_in_configloader

Tests if the config-loader in .omniopt.py has only single quotes, as double-quotes cause other scripts that parse it to fail.

state_files

Test if all state_files are properly documented.

unit_tests

Runs some basic unit-tests on different scripts.

wronggoing_stuff

Test if OmniOpt2 reacts properly if stuff goes wrong.

--nosuccess

bool

False

--help

bool

False

--debug

bool

False

linter

Run different linters.

--quick

bool

False

--help

bool

False

--dont_fail_on_error

bool

False

optimization_example_all_float

Very simple single-objective-optimization-problem based on the shekel-function where all inputs are float.

--random_sem

0 or 1

--x

float

--y

float

--z

float

--a

float

--help

bool

False

--debug

bool

False

start_simple_optimization_run

Start a simple optimization based on the shekel-function.

--max_eval

int

--mem_gb

int

--num_parallel_jobs

int

--num_random_steps

int

--model

str

--gridsearch

bool

False

--gpus

int

--time

int

--allow_failure

bool

False

--force_local_execution

bool

False

--all_float

bool

False

--flame_graph

bool

False

--one_param

bool

False

--two_params

bool

False

--nr_results

int

--seed

int

--additional_parameter

str

--alternate_min_max

bool

False

--force_choice_for_ranges

bool

False

--follow

bool

False

--generate_all_jobs_at_once

bool

False

--verbose

bool

False

--revert_to_random_when_seemingly_exhausted

bool

False

--testname

str

--show_ram_every_n_seconds

int

--random_sem

bool

False

--help

bool

False

--debug

bool

False

test_seed

Test if the seed is properly used and results are consistent and deterministic.

--seed

int

--gpus

int

--hash

str

--help

bool

False

train_example_network

Starts an OmniOpt2 run that trains an example neural network.

--help

bool

False

--debug

bool

False

--data

str

--num_parallel_jobs

int

--worker_timeout

int

--mem_gb

int

--time

int

--num_random_steps

int

--max_eval

int

--min_dense

int

--max_dense

int

--min_dense_units

int

--max_dense_units

int

--min_conv_filters

int

--max_conv_filters

int

--min_conv

int

--max_conv

int

--min_epochs

int

--max_epochs

int

--min_height

int

--max_height

int

--min_width

int

--max_width

int

--min_learning_rate

float

--max_learning_rate

float

--validation_split

float

--result_names

str

--follow

bool

False

--gridsearch

bool

False

--generate_all_jobs_at_once

bool

False

--revert_to_random_when_seemingly_exhausted

bool

False

external_generator

Test external generator.

--mem_gb

int

--time

int

--worker_timeout

int

--max_eval

int

--num_parallel_jobs

int

--gpus

int

--num_random_steps

int

--external_generator

str

--help

bool

False

randomforest

Test if random forest works properly.

multi_results_different_models

Test multi-objective-optimization with different models.

tpe

Test the Tree Parzen Estimator model (TPE).

--max_eval

--num_random_steps

--num_parallel_jobs

--mem_gb

--testname

--gpus

0 or 1

--additional

--help

bool

False

all_oo_options_are_in_gui_and_vice_versa

Find GUI options that are not in OmniOpt2 or vice versa, so they always reflect each other.

main

Main test script, coordinates complete test runs

--exit_on_first_error

bool

False

--help

bool

False

--run_with_coverage

bool

False

--quick

bool

False

--reallyquick

bool

False

--superquick

bool

False

--debug

bool

False

--no_linkchecker

bool

False

--no_linter

bool

False

--skip_test_job_nr

bool

False

--skip_first_n_tests

int

--skip_worker_check

bool

False

--no_plots

bool

False

transforms

Test transforms.

--max_eval

--num_random_steps

--num_parallel_jobs

--mem_gb

--testname

--gpus

0 or 1

--additional

--help

bool

False