ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
โ Exception in parallel job handling: Can only mark failed a trial that is currently running.
It seems like the search space was exhausted. You were able to get 4% of the jobs you requested (got: 1, submitted: 2, failed: 0, requested: 23) after main ran
========================================================================
Current git-hash: 2bfcf4ebab7e0b53d16282a8de94a4c409ec3fbf (last fully tested stable version 2 commits ago [9043d91dbe370183252bdf120ecb2c226402839d, 8689])
Using coverage run -p because $RUN_WITH_COVERAGE is set
[WARNING 10-21 07:10:47] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
Run-UUID: c4828bdb-d008-4b7c-8b8c-86f63b0a9932
_____ _ _____ ______
/ ___ \ (_)/ ___ \ _ (_____ \
| | | |____ ____ _| | | |____ | |_ ____) )
| | | | \| _ \| | | | | _ \| _) /_____/
| |___| | | | | | | | | |___| | | | | |__ _______
\_____/|_|_|_|_| |_|_|\_____/| ||_/ \___|_______)
|_|
omniopt --max_nr_of_zero_results 3 --live_share --send_anonymized_usage_stats --partition=alpha --gpus=0 --experiment_name=test_wronggoing_stuff --mem_gb=1 --time=120 --worker_timeout=1 --max_eval=23 --num_parallel_jobs=10 --run_program=Li8udGVzdHMvdGVzdF93cm9uZ2dvaW5nX3N0dWZmLmJpbi9iaW4vJShwcm9ncmFtKQ== --parameter program choice divide_by_0,simple_ok,perl,exit_code_137,result_but_exit_code_stdout_stderr,exit_code_stdout,signal_but_has_output,exit_code_no_output,exit_code_stdout_stderr,module_not_found,signal,empty,file_does_not_exist,sleep_10_and_result_exit_252,no_chmod_x,no_shebang,perl_module_fail,segfault,wrong_arch,python_name_error,syntax_error,tensorflow_syntax_error,force_keyboard_interrupt --num_random_steps=1 --follow
Run-folder: /home/runner/work/OmniOpt/OmniOpt/runs/test_wronggoing_stuff/0See https://imageseg.scads.de/omniax/share?user_id=runner&experiment_name=test_wronggoing_stuff&run_nr=2 for live-results.You have 4 CPUs available for the main process.No CUDA devices found.Generation strategy: SOBOL for 1 step and then BOTORCH_MODULAR for 22 steps.
Run-Program: ./.tests/test_wronggoing_stuff.bin/bin/%(program)
Experiment parameters
โโโโโโโโโโโณโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โName โType โValues โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โprogramโchoiceโdivide_by_0, simple_ok, perl, exit_code_137, result_but_exit_code_stdout_stderr, exit_code_stdout, signal_but_has_output, exit_code_no_output, exit_code_stdout_stderr, โ
โโโmodule_not_found, signal, empty, file_does_not_exist, sleep_10_and_result_exit_252, no_chmod_x, no_shebang, perl_module_fail, segfault, wrong_arch, python_name_error, โ
โโโsyntax_error, tensorflow_syntax_error, force_keyboard_interrupt โ
โโโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Result-Names
โโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โResult-NameโMin or max?โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โRESULT โ minโ
โโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.โ Exception in parallel job handling: Can only mark failed a trial that is currently running.
It seems like the search space was exhausted. You were able to get 4% of the jobs you requested (got: 1, submitted: 2, failed: 0, requested: 23) after main ran Failed Jobs parameters
โโโโโโโโโโณโโโโโโโโโณโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โparam1โparam2โparam3โerror_description โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โvalue1โvalue2โvalue3โSome error occurred during execution (this is not a real error!).โ
โโโโโโโโโโดโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Best RESULT, min (total: 1)
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโ
โprogram โRESULTโ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โsleep_10_and_result_exit_252โ1234.0โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโ
Not printing sixel graphics in CI
Runtime Infos
โโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โNumber of evaluationsโMin time โ Max time โAverage timeโMedian timeโ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 2 โ 6.00 sec โ 12.00 sec โ 9.00 sec โ 9.00 sec โ
โโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
2025-10-21 07:10:48 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, Started OmniOpt2 run...
2025-10-21 07:10:48 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, getting new HP set (no sbatch)
2025-10-21 07:10:48 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, requested 1 jobs, got 1, 0.03 s/job
2025-10-21 07:10:48 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, eval #1/1 start
2025-10-21 07:10:48 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, starting new job
2025-10-21 07:10:48 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, started new job
2025-10-21 07:11:03 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, new result: 1234.0
2025-10-21 07:11:05 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, finishing jobs, finished 1 job
2025-10-21 07:11:05 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:05 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, requested 1 jobs, got 1, 0.30 s/job
2025-10-21 07:11:05 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, eval #1/1 start
2025-10-21 07:11:05 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, starting new job
2025-10-21 07:11:05 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, started new job
2025-10-21 07:11:14 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, job_failed
2025-10-21 07:11:14 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:15 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:15 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:15 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:15 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:16 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:16 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:16 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:16 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:17 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:17 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:17 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:17 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:17 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:18 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:18 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:18 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:18 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:19 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:19 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:19 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, requested 1 jobs, got 0, n/a
2025-10-21 07:11:19 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, found 1 zero-jobs (max: 3)
2025-10-21 07:11:19 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:19 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:20 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:20 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:20 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:20 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:21 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:21 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:21 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:21 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:22 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:22 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:22 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:22 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:23 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:23 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:23 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:23 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:24 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:24 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:24 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, requested 1 jobs, got 0, n/a
2025-10-21 07:11:24 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, found 2 zero-jobs (max: 3)
2025-10-21 07:11:24 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:24 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:25 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:25 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:25 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:25 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:26 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:26 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:26 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:26 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:26 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:27 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:27 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:27 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:27 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:28 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:28 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:28 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:28 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:29 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, getting new HP set (no sbatch)
2025-10-21 07:11:29 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, requested 1 jobs, got 0, n/a
2025-10-21 07:11:29 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, found 3 zero-jobs (max: 3)
2025-10-21 07:11:29 (0c5e32b0-57d5-4b52-a927-eaf33c6eebb1): SOBOL, best RESULT: 1234, 3 empty jobs (>= 3)
Job submission durations
โโโโโโโโโโโณโโโโโโโโโโณโโโโโโโณโโโโโโโโโโโโโโโ
โ Batch โ Seconds โ Jobs โ Time per job โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 1 โ 0.093 โ 1 โ 0.093 โ
โ 2 โ 0.058 โ 1 โ 0.058 โ
โโโโโโโโโโโผโโโโโโโโโโผโโโโโโโผโโโโโโโโโโโโโโโค
โ Average โ 0.076 โ โ โ
โ Median โ 0.076 โ โ โ
โ Total โ 0.151 โ โ โ
โ Max โ 0.093 โ โ โ
โ Min โ 0.058 โ โ โ
โโโโโโโโโโโดโโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโโโ
Model generation times
โโโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโณโโโโโโโโโโโโโโโ
โ Iteration โ Seconds โ Jobs โ Time per job โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 1 โ 0.033 โ 1 โ 0.033 โ
โ 2 โ 0.299 โ 1 โ 0.299 โ
โ 3 โ 4.739 โ 0 โ 0.000 โ
โ 4 โ 4.886 โ 0 โ 0.000 โ
โ 5 โ 4.744 โ 0 โ 0.000 โ
โโโโโโโโโโโโโผโโโโโโโโโโผโโโโโโโผโโโโโโโโโโโโโโโค
โ Average โ 2.940 โ โ โ
โ Median โ 4.739 โ โ โ
โ Total โ 14.702 โ โ โ
โ Max โ 4.886 โ โ โ
โ Min โ 0.033 โ โ โ
โโโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโโโ