⚠ Job (task=0) failed during processing with trace:
----------------------
Traceback (most recent call last):
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/submission.py", line 55, in process_job
result = delayed.result()
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/utils.py", line 137, in result
self._result = self.function(*self.args, **self.kwargs)
File "<@beartype(__main__.evaluate) at 0x7f301ec085e0>", line 36, in evaluate
File "/home/runner/work/OmniOpt/OmniOpt/.omniopt.py", line 4803, in evaluate
final_result = _run_multi_evaluation(parameters, program_string, nr_evals, trial_index, submit_time, queue_time, return_in_case_of_error)
File "<@beartype(__main__._run_multi_evaluation) at 0x7f301ec08550>", line 178, in _run_multi_evaluation
beartype.roar.BeartypeCallHintParamViolation: Function __main__._run_multi_evaluation() parameter return_in_case_of_error={'RESULT1': 99999999999999999999999999999999999999999999999999999999999, 'RESULT2': -9999999...} violates type hint dict[str, typing.Union[float, typing.Tuple[float, float]]], as dict key str 'RESULT1' value int 99999999999999999999999999999999999999999999999999999999999 not tuple or float.
----------------------
You can check full logs with 'job.stderr(0)' and 'job.stdout(0)'or at paths:
- /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0/single_runs/75133/75133_0_log.err
- /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0/single_runs/75133/75133_0_log.out
⚠ Job (task=0) failed during processing with trace:
----------------------
Traceback (most recent call last):
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/submission.py", line 55, in process_job
result = delayed.result()
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/utils.py", line 137, in result
self._result = self.function(*self.args, **self.kwargs)
File "<@beartype(__main__.evaluate) at 0x7f301ec085e0>", line 36, in evaluate
File "/home/runner/work/OmniOpt/OmniOpt/.omniopt.py", line 4803, in evaluate
final_result = _run_multi_evaluation(parameters, program_string, nr_evals, trial_index, submit_time, queue_time, return_in_case_of_error)
File "<@beartype(__main__._run_multi_evaluation) at 0x7f301ec08550>", line 178, in _run_multi_evaluation
beartype.roar.BeartypeCallHintParamViolation: Function __main__._run_multi_evaluation() parameter return_in_case_of_error={'RESULT1': 99999999999999999999999999999999999999999999999999999999999, 'RESULT2': -9999999...} violates type hint dict[str, typing.Union[float, typing.Tuple[float, float]]], as dict key str 'RESULT1' value int 99999999999999999999999999999999999999999999999999999999999 not tuple or float.
----------------------
You can check full logs with 'job.stderr(0)' and 'job.stdout(0)'or at paths:
- /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0/single_runs/75145/75145_0_log.err
- /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0/single_runs/75145/75145_0_log.out
Error in get_calculated_frontier: No full data points with both objectives found in /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0.
Best RESULT1 could not be determined
========================================================================
Current git-hash: 9549d52ee597a3bd0ab7313d9a232666aa4c243f (last fully tested stable version 20 commits ago [1431d695e1d9911d740a43d6b86abbd58f76a3b1, 9230])
Using coverage run -p because $RUN_WITH_COVERAGE is set
[WARNING 02-19 10:53:18] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
Run-UUID: 5effbaff-e311-4356-a748-09cb3059723a
@@@@@@ @@@@@@@@@@ @@@ @@@ @@@ @@@@@@ @@@@@@@ @@@@@@@ @@@@@@
@@@@@@@@ @@@@@@@@@@@ @@@@ @@@ @@@ @@@@@@@@ @@@@@@@@ @@@@@@@ @@@@@@@@
@@! @@@ @@! @@! @@! @@!@!@@@ @@! @@! @@@ @@! @@@ @@! @@@
!@! @!@ !@! !@! !@! !@!!@!@! !@! !@! @!@ !@! @!@ !@! @!@
@!@ !@! @!! !!@ @!@ @!@ !!@! !!@ @!@ !@! @!@@!@! @!! !!@
!@! !!! !@! ! !@! !@! !!! !!! !@! !!! !!@!!! !!! !!:
!!: !!! !!: !!: !!: !!! !!: !!: !!! !!: !!: !:!
:!: !:! :!: :!: :!: !:! :!: :!: !:! :!: :!: :!:
::::: :: ::: :: :: :: :: ::::: :: :: :: :: :::::
: : : : : :: : : : : : : : :: : :::
omniopt --live_share --send_anonymized_usage_stats --partition alpha --experiment_name=multiple_evals_per_arm --mem_gb=4 --time=1200 --worker_timeout=5 --max_eval 2 --num_parallel_jobs 1 --gpus=0 --run_program Li8udGVzdHMvb3B0aW1pemF0aW9uX2V4YW1wbGUgIC0taW50X3BhcmFtPSclKGludF9wYXJhbSknIC0tZmxvYXRfcGFyYW09JyUoZmxvYXRfcGFyYW0pJyAtLWNob2ljZV9wYXJhbT0nJShjaG9pY2VfcGFyYW0pJyAtLWludF9wYXJhbV90d289JyUoaW50X3BhcmFtX3R3byknIC0tbnJfcmVzdWx0cz0y --parameter int_param range -100 10 int --parameter float_param range -100 10 float --parameter choice_param choice 1,2,4,8,16,hallo --parameter int_param_two range -100 10 int --num_random_steps 1 --model BOTORCH_MODULAR --auto_exclude_defective_hosts --result_names RESULT1=min RESULT2=max --generate_all_jobs_at_once --revert_to_random_when_seemingly_exhausted --nr_evals_per_arm=2 --show_generate_time_table
Run-folder: /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0See https://imageseg.scads.de/omniax/share?user_id=runner&experiment_name=multiple_evals_per_arm&run_nr=0 for live-results.You have 4 CPUs available for the main process.No CUDA devices found.Generation strategy: SOBOL for 1 step and then BOTORCH_MODULAR for 1 step.
Run-Program: ./.tests/optimization_example --int_param='%(int_param)' --float_param='%(float_param)' --choice_param='%(choice_param)' --int_param_two='%(int_param_two)' --nr_results=2
Experiment parameters
┏━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃Name ┃Type ┃Lower bound┃Upper bound┃Values ┃Type ┃Log Scale?┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│int_param │range │-100 │10 ││int │No │
│float_param │range │-100 │10 ││float│No │
│choice_param │choice│││1, 2, 4, 8, 16, hallo│││
│int_param_two│range │-100 │10 ││int │No │
└───────────────┴────────┴─────────────┴─────────────┴───────────────────────┴───────┴────────────┘
Result-Names
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃Result-Name┃Min or max?┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│RESULT1 │ min│
│RESULT2 │ max│
└─────────────┴─────────────┘
⚠ Job (task=0) failed during processing with trace:
----------------------
Traceback (most recent call last):
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/submission.py", line 55, in process_job
result = delayed.result()
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/utils.py", line 137, in result
self._result = self.function(*self.args, **self.kwargs)
File "<@beartype(__main__.evaluate) at 0x7f301ec085e0>", line 36, in evaluate
File "/home/runner/work/OmniOpt/OmniOpt/.omniopt.py", line 4803, in evaluate
final_result = _run_multi_evaluation(parameters, program_string, nr_evals, trial_index, submit_time, queue_time, return_in_case_of_error)
File "<@beartype(__main__._run_multi_evaluation) at 0x7f301ec08550>", line 178, in _run_multi_evaluation
beartype.roar.BeartypeCallHintParamViolation: Function __main__._run_multi_evaluation() parameter return_in_case_of_error={'RESULT1': 99999999999999999999999999999999999999999999999999999999999, 'RESULT2': -9999999...} violates type hint dict[str, typing.Union[float, typing.Tuple[float, float]]], as dict key str 'RESULT1' value int 99999999999999999999999999999999999999999999999999999999999 not tuple or float.
----------------------
You can check full logs with 'job.stderr(0)' and 'job.stdout(0)'or at paths:
- /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0/single_runs/75133/75133_0_log.err
- /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0/single_runs/75133/75133_0_log.out
⚠ Job (task=0) failed during processing with trace:
----------------------
Traceback (most recent call last):
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/submission.py", line 55, in process_job
result = delayed.result()
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/utils.py", line 137, in result
self._result = self.function(*self.args, **self.kwargs)
File "<@beartype(__main__.evaluate) at 0x7f301ec085e0>", line 36, in evaluate
File "/home/runner/work/OmniOpt/OmniOpt/.omniopt.py", line 4803, in evaluate
final_result = _run_multi_evaluation(parameters, program_string, nr_evals, trial_index, submit_time, queue_time, return_in_case_of_error)
File "<@beartype(__main__._run_multi_evaluation) at 0x7f301ec08550>", line 178, in _run_multi_evaluation
beartype.roar.BeartypeCallHintParamViolation: Function __main__._run_multi_evaluation() parameter return_in_case_of_error={'RESULT1': 99999999999999999999999999999999999999999999999999999999999, 'RESULT2': -9999999...} violates type hint dict[str, typing.Union[float, typing.Tuple[float, float]]], as dict key str 'RESULT1' value int 99999999999999999999999999999999999999999999999999999999999 not tuple or float.
----------------------
You can check full logs with 'job.stderr(0)' and 'job.stdout(0)'or at paths:
- /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0/single_runs/75145/75145_0_log.err
- /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0/single_runs/75145/75145_0_log.outError in get_calculated_frontier: No full data points with both objectives found in /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0.
The data seems to be empty. Cannot plot pareto frontier.
Failed Jobs parameters
┏━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃param1┃param2┃param3┃error_description ┃
┡━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│value1│value2│value3│Some error occurred during execution (this is not a real error!).│
└────────┴────────┴────────┴───────────────────────────────────────────────────────────────────┘
Best RESULT1 could not be determinedModel generation times
┏━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┓
┃Iteration┃Seconds┃Jobs┃Time per job┃
┡━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━┩
│ 1 │ 0.030 │ 1 │ 0.030 │
│ 2 │ 0.039 │ 1 │ 0.039 │
├───────────┼─────────┼──────┼──────────────┤
│ Average │ 0.035 │ │ │
│ Median │ 0.035 │ │ │
│ Total │ 0.070 │ │ │
│ Max │ 0.039 │ │ │
│ Min │ 0.030 │ │ │
└───────────┴─────────┴──────┴──────────────┘
Job submission durations
┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┓
┃ Batch┃Seconds┃Jobs┃Time per job┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━┩
│ 1 │ 0.123 │ 1 │ 0.123 │
│ 2 │ 0.053 │ 1 │ 0.053 │
├─────────┼─────────┼──────┼──────────────┤
│ Average │ 0.088 │ │ │
│ Median │ 0.088 │ │ │
│ Total │ 0.175 │ │ │
│ Max │ 0.123 │ │ │
│ Min │ 0.053 │ │ │
└─────────┴─────────┴──────┴──────────────┘
2026-02-19 10:53:20 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, Started OmniOpt2 run...
2026-02-19 10:53:20 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, getting new HP set (no sbatch)
2026-02-19 10:53:20 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, requested 1 jobs, got 1, 0.03 s/job
2026-02-19 10:53:20 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, eval #1/1 start
2026-02-19 10:53:20 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, starting new job
2026-02-19 10:53:20 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, started new job
2026-02-19 10:53:23 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, job_failed
2026-02-19 10:53:24 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, failed: 1, finishing jobs, finished 1 job
2026-02-19 10:53:24 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, failed: 1, getting new HP set (no sbatch)
2026-02-19 10:53:24 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, failed: 1, requested 1 jobs, got 1, 0.04 s/job
2026-02-19 10:53:24 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, failed: 1, eval #1/1 start
2026-02-19 10:53:24 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, failed: 1, starting new job
2026-02-19 10:53:24 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, failed: 1, started new job
2026-02-19 10:53:27 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, failed: 1, job_failed
2026-02-19 10:53:27 (b104b2dd-66be-4e2f-a72a-78254507174b): SOBOL, failed: 2, finishing jobs, finished 1 job
Job submission durations
┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┓
┃ Batch ┃ Seconds ┃ Jobs ┃ Time per job ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━┩
│ 1 │ 0.123 │ 1 │ 0.123 │
│ 2 │ 0.053 │ 1 │ 0.053 │
├─────────┼─────────┼──────┼──────────────┤
│ Average │ 0.088 │ │ │
│ Median │ 0.088 │ │ │
│ Total │ 0.175 │ │ │
│ Max │ 0.123 │ │ │
│ Min │ 0.053 │ │ │
└─────────┴─────────┴──────┴──────────────┘
Model generation times
┏━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┓
┃ Iteration ┃ Seconds ┃ Jobs ┃ Time per job ┃
┡━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━┩
│ 1 │ 0.030 │ 1 │ 0.030 │
│ 2 │ 0.039 │ 1 │ 0.039 │
├───────────┼─────────┼──────┼──────────────┤
│ Average │ 0.035 │ │ │
│ Median │ 0.035 │ │ │
│ Total │ 0.070 │ │ │
│ Max │ 0.039 │ │ │
│ Min │ 0.030 │ │ │
└───────────┴─────────┴──────┴──────────────┘
submitit INFO (2026-02-19 10:53:21,053) - Starting with JobEnvironment(job_id=75133, hostname=runnervmn36qa, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-02-19 10:53:21,053) - Loading pickle: /home/runner/work/OmniOpt/OmniOpt/runs/multiple_evals_per_arm/0/single_runs/75133/75133_submitted.pkl
Trial-Index: 0
submitit ERROR (2026-02-19 10:53:22,654) - Submitted job triggered an exception
submitit ERROR (2026-02-19 10:53:22,654) - Submitted job triggered an exception
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/_submit.py", line 11, in <module>
submitit_main()
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/submission.py", line 76, in submitit_main
process_job(args.folder)
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/submission.py", line 69, in process_job
raise error
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/submission.py", line 55, in process_job
result = delayed.result()
File "/home/runner/.omniax_venvs/Python_3.10.12/x86_64/lib/python3.10/site-packages/submitit/core/utils.py", line 137, in result
self._result = self.function(*self.args, **self.kwargs)
File "<@beartype(__main__.evaluate) at 0x7f301ec085e0>", line 36, in evaluate
File "/home/runner/work/OmniOpt/OmniOpt/.omniopt.py", line 4803, in evaluate
final_result = _run_multi_evaluation(parameters, program_string, nr_evals, trial_index, submit_time, queue_time, return_in_case_of_error)
File "<@beartype(__main__._run_multi_evaluation) at 0x7f301ec08550>", line 178, in _run_multi_evaluation
beartype.roar.BeartypeCallHintParamViolation: Function __main__._run_multi_evaluation() parameter return_in_case_of_error={'RESULT1': 99999999999999999999999999999999999999999999999999999999999, 'RESULT2': -9999999...} violates type hint dict[str, typing.Union[float, typing.Tuple[float, float]]], as dict key str 'RESULT1' value int 99999999999999999999999999999999999999999999999999999999999 not tuple or float.