ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
โ Exception in parallel job handling: Can only mark failed a trial that is currently running.
It seems like the search space was exhausted. You were able to get 4% of the jobs you requested (got: 1, submitted: 2, failed: 0, requested: 23) after main ran
========================================================================
Current git-hash: 3bc86dcb6d9e1e752cf34d003103a8f1015b5b05 (last fully tested stable version 6 commits ago [9043d91dbe370183252bdf120ecb2c226402839d, 8689])
Using coverage run -p because $RUN_WITH_COVERAGE is set
[WARNING 10-21 08:25:08] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
Run-UUID: 5ee330b6-0d3d-47da-8222-4be840dc104b
####### ## ## ## ## #### ####### ######## ######## #######
## ## ### ### ### ## ## ## ## ## ## ## ## ##
## ## #### #### #### ## ## ## ## ## ## ## ##
## ## ## ### ## ## ## ## ## ## ## ######## ## #######
## ## ## ## ## #### ## ## ## ## ## ##
## ## ## ## ## ### ## ## ## ## ## ##
####### ## ## ## ## #### ####### ## ## #########
omniopt --max_nr_of_zero_results 3 --live_share --send_anonymized_usage_stats --partition=alpha --gpus=0 --experiment_name=test_wronggoing_stuff --mem_gb=1 --time=120 --worker_timeout=1 --max_eval=23 --num_parallel_jobs=10 --run_program=Li8udGVzdHMvdGVzdF93cm9uZ2dvaW5nX3N0dWZmLmJpbi9iaW4vJShwcm9ncmFtKQ== --parameter program choice divide_by_0,simple_ok,perl,exit_code_137,result_but_exit_code_stdout_stderr,exit_code_stdout,signal_but_has_output,exit_code_no_output,exit_code_stdout_stderr,module_not_found,signal,empty,file_does_not_exist,sleep_10_and_result_exit_252,no_chmod_x,no_shebang,perl_module_fail,segfault,wrong_arch,python_name_error,syntax_error,tensorflow_syntax_error,force_keyboard_interrupt --num_random_steps=1 --follow
Run-folder: /home/runner/work/OmniOpt/OmniOpt/runs/test_wronggoing_stuff/0See https://imageseg.scads.de/omniax/share?user_id=runner&experiment_name=test_wronggoing_stuff&run_nr=8 for live-results.You have 4 CPUs available for the main process.No CUDA devices found.Generation strategy: SOBOL for 1 step and then BOTORCH_MODULAR for 22 steps.
Run-Program: ./.tests/test_wronggoing_stuff.bin/bin/%(program)
Experiment parameters
โโโโโโโโโโโณโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โName โType โValues โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โprogramโchoiceโdivide_by_0, simple_ok, perl, exit_code_137, result_but_exit_code_stdout_stderr, exit_code_stdout, signal_but_has_output, exit_code_no_output, exit_code_stdout_stderr, โ
โโโmodule_not_found, signal, empty, file_does_not_exist, sleep_10_and_result_exit_252, no_chmod_x, no_shebang, perl_module_fail, segfault, wrong_arch, python_name_error, โ
โโโsyntax_error, tensorflow_syntax_error, force_keyboard_interrupt โ
โโโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Result-Names
โโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โResult-NameโMin or max?โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โRESULT โ minโ
โโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.โ Exception in parallel job handling: Can only mark failed a trial that is currently running.
It seems like the search space was exhausted. You were able to get 4% of the jobs you requested (got: 1, submitted: 2, failed: 0, requested: 23) after main ran Failed Jobs parameters
โโโโโโโโโโณโโโโโโโโโณโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โparam1โparam2โparam3โerror_description โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โvalue1โvalue2โvalue3โSome error occurred during execution (this is not a real error!).โ
โโโโโโโโโโดโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Best RESULT, min (total: 1)
โโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโ
โprogram โRESULT โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โsignal_but_has_outputโ16579.0โ
โโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโ
Not printing sixel graphics in CI
Runtime Infos
โโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โNumber of evaluationsโMin time โMax time โAverage timeโMedian timeโ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 2 โ 6.00 sec โ 6.00 sec โ 6.00 sec โ 6.00 sec โ
โโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
2025-10-21 08:25:10 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, Started OmniOpt2 run...
2025-10-21 08:25:10 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, getting new HP set (no sbatch)
2025-10-21 08:25:10 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, requested 1 jobs, got 1, 0.04 s/job
2025-10-21 08:25:10 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, eval #1/1 start
2025-10-21 08:25:10 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, starting new job
2025-10-21 08:25:10 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, started new job
2025-10-21 08:25:19 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, new result: 16579.0
2025-10-21 08:25:20 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, finishing jobs, finished 1 job
2025-10-21 08:25:20 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:20 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, requested 1 jobs, got 1, 0.31 s/job
2025-10-21 08:25:20 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, eval #1/1 start
2025-10-21 08:25:20 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, starting new job
2025-10-21 08:25:20 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, started new job
2025-10-21 08:25:29 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, job_failed
2025-10-21 08:25:30 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:30 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:30 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:30 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:31 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:31 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:31 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:31 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:32 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:32 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:32 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:32 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:33 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:33 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:33 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:33 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:34 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:34 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:34 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:34 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:34 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, requested 1 jobs, got 0, n/a
2025-10-21 08:25:34 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, found 1 zero-jobs (max: 3)
2025-10-21 08:25:35 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:35 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:35 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:35 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:36 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:36 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:36 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:36 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:36 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:37 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:37 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:37 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:38 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:38 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:38 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:38 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:39 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:39 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:39 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:39 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:39 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, requested 1 jobs, got 0, n/a
2025-10-21 08:25:39 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, found 2 zero-jobs (max: 3)
2025-10-21 08:25:40 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:40 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:40 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:40 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:41 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:41 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:41 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:41 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:42 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:42 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:42 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:42 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:43 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:43 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:43 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:43 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:44 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:44 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:44 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:44 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, getting new HP set (no sbatch)
2025-10-21 08:25:44 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, requested 1 jobs, got 0, n/a
2025-10-21 08:25:44 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, found 3 zero-jobs (max: 3)
2025-10-21 08:25:44 (5712e15c-e90b-4422-a9f5-b0efe0d2ed45): SOBOL, best RESULT: 16579, 3 empty jobs (>= 3)
Job submission durations
โโโโโโโโโโโณโโโโโโโโโโณโโโโโโโณโโโโโโโโโโโโโโโ
โ Batch โ Seconds โ Jobs โ Time per job โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 1 โ 0.096 โ 1 โ 0.096 โ
โ 2 โ 0.063 โ 1 โ 0.063 โ
โโโโโโโโโโโผโโโโโโโโโโผโโโโโโโผโโโโโโโโโโโโโโโค
โ Average โ 0.079 โ โ โ
โ Median โ 0.079 โ โ โ
โ Total โ 0.159 โ โ โ
โ Max โ 0.096 โ โ โ
โ Min โ 0.063 โ โ โ
โโโโโโโโโโโดโโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโโโ
Model generation times
โโโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโณโโโโโโโโโโโโโโโ
โ Iteration โ Seconds โ Jobs โ Time per job โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 1 โ 0.035 โ 1 โ 0.035 โ
โ 2 โ 0.308 โ 1 โ 0.308 โ
โ 3 โ 4.846 โ 0 โ 0.000 โ
โ 4 โ 5.030 โ 0 โ 0.000 โ
โ 5 โ 4.850 โ 0 โ 0.000 โ
โโโโโโโโโโโโโผโโโโโโโโโโผโโโโโโโผโโโโโโโโโโโโโโโค
โ Average โ 3.014 โ โ โ
โ Median โ 4.846 โ โ โ
โ Total โ 15.069 โ โ โ
โ Max โ 5.030 โ โ โ
โ Min โ 0.035 โ โ โ
โโโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโโโ