========================================================================
To cancel, press CTRLc, then run 'scancel 1170025'
⠋Importing logging...⠋Importing warnings...⠋Importing argparse...⠋Importing datetime...⠋Importing dataclass...⠋Importing socket...⠋Importing stat...⠋Importing pwd...⠋Importing base64...⠋Importing json...⠋Importing yaml...⠋Importing toml...⠋Importing csv...⠋Importing ast...⠋Importing rich.table...⠋Importing rich print...⠋Importing rich.pretty...⠋Importing pformat...⠋Importing rich.prompt...⠋Importing types.FunctionType...⠋Importing typing...⠋Importing ThreadPoolExecutor...⠋Importing submitit.LocalExecutor...⠋Importing submitit.Job...⠋Importing importlib.util...⠋Importing platform...⠋Importing inspect frame info...⠋Importing pathlib.Path...⠋Importing uuid...⠋Importing cowsay...⠋Importing shutil...⠋Importing itertools.combinations...⠋Importing os.listdir...⠋Importing os.path...⠋Importing PIL.Image...⠋Importing sixel...⠋Importing subprocess...⠋Importing tqdm...⠋Importing beartype...⠋Importing statistics...⠋Trying to import pyfiglet...⠙Importing helpers...⠋Parsing arguments...⠸Importing torch...⠋Importing numpy...
[WARNING 10-25 00:51:34] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
⠧Importing ax...⠋Importing ax.core.generator_run...⠋Importing Cont_X_trans and Y_trans from ax.adapter.registry...⠋Importing ax.core.arm...⠋Importing ax.core.objective...⠋Importing ax.core.Metric...⠋Importing ax.exceptions.core...⠋Importing ax.exceptions.generation_strategy...⠋Importing CORE_DECODER_REGISTRY...⠋Trying ax.generation_strategy.generation_node...⠋Importing GenerationStep, GenerationStrategy from generation_strategy...⠋Importing GenerationNode from generation_node...⠋Importing ExternalGenerationNode...⠋Importing MaxTrials...⠋Importing GeneratorSpec...⠋Importing Models from ax.generation_strategy.registry...⠋Importing get_pending_observation_features...⠋Importing load_experiment...⠋Importing save_experiment...⠋Importing save_experiment_to_db...⠋Importing TrialStatus...⠋Importing Data...⠋Importing Experiment...⠋Importing parameter types...⠋Importing TParameterization...⠋Importing pandas...⠋Importing AxClient and ObjectiveProperties...⠋Importing RandomForestRegressor...⠋Importing botorch...⠋Importing submitit...⠋Importing ax logger...⠋Importing SQL-Storage-Stuff...
Run-UUID: dbdef305-0b00-45d0-a82e-0edd9ffc80fb
.oOOOo. .oOOOo.
.O o. o .O o. .oOOo.
O o O o O O
o O o O oOo o
O o `oOOoOO. 'OoOo. O O o .oOo. o O'
o O O o o o O o o O O o O O
`o O' o O O O o O `o O' o O o .O
`OoooO' O o o o O o' `OoooO' oOoO' `oO oOoOoO
O
o'
⠋Writing worker creation log...
omniopt --live_share --send_anonymized_usage_stats --partition alpha --experiment_name=adding_old_jobs_to_new_jobs --mem_gb=4 --time=60 --worker_timeout=5 --max_eval 2 --num_parallel_jobs 1 --gpus=1 --run_program Li8udGVzdHMvb3B0aW1pemF0aW9uX2V4YW1wbGUgLS1pbnRfcGFyYW09JyUoaW50X3BhcmFtKScgLS1mbG9hdF9wYXJhbT0nJShmbG9hdF9wYXJhbSknIC0tY2hvaWNlX3BhcmFtPSclKGNob2ljZV9wYXJhbSknICAtLWludF9wYXJhbV90d289JyUoaW50X3BhcmFtX3R3byknIC0tbnJfcmVzdWx0cz0x --parameter int_param range -100 10 int --parameter float_param range -100 10 float --parameter choice_param choice 1,2,4,8,16,hallo --parameter int_param_two range -100 10 int --follow --num_random_steps 1 --model BOTORCH_MODULAR --auto_exclude_defective_hosts --load_data_from_existing_jobs runs/adding_old_jobs_to_new_jobs//0 runs/adding_old_jobs_to_new_jobs//1
⠋Disabling logging...⠋Setting run folder...⠋Creating folder /data/horse/ws/s3811141-oo_crap/omniopt/runs/adding_old_jobs_to_new_jobs/2...⠋Writing revert_to_random_when_seemingly_exhausted file ...⠋Writing username state file...⠋Writing result names file...⠋Writing result min/max file...⠋Saving state files...Run-folder: /data/horse/ws/s3811141-oo_crap/omniopt/runs/adding_old_jobs_to_new_jobs/2⠋Writing live_share file if it is present...⠋Writing job_start_time file...⠸Writing git information⠋Checking max_eval...⠋Calculating number of steps...⠋Adding excluded nodes...⠋Initializing ax_client...⠋Setting orchestrator...See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=adding_old_jobs_to_new_jobs&run_nr=20 for live-results.You have 1 CPUs available for the main process.Using CUDA device NVIDIA H100.Generation strategy: SOBOL for 1 step and then BOTORCH_MODULAR for 1 step.
Run-Program: ./.tests/optimization_example --int_param='%(int_param)' --float_param='%(float_param)' --choice_param='%(choice_param)' --int_param_two='%(int_param_two)' --nr_results=1
Experiment parameters
┏━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃Name ┃Type ┃Lower bound┃Upper bound┃Values ┃Type ┃Log Scale?┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│int_param │range │-100 │10 ││int │No │
│float_param │range │-100 │10 ││float│No │
│choice_param │choice│││1, 2, 4, 8, 16, hallo│││
│int_param_two│range │-100 │10 ││int │No │
└───────────────┴────────┴─────────────┴─────────────┴───────────────────────┴───────┴────────────┘
Result-Names
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃Result-Name┃Min or max?┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│RESULT │ min│
└─────────────┴─────────────┘
⠋Write files and show overviewInserted 2 jobs from runs/adding_old_jobs_to_new_jobs/0/results.csv⠋Loading job 1/2 from runs/adding_old_jobs_to_new_jobs/0/results.csv into ax_client, result: {'RESULT': -38454.8226699435}: Saved results.csvInserted 2 jobs from runs/adding_old_jobs_to_new_jobs/1/results.csv⠙Loading job 1/2 from runs/adding_old_jobs_to_new_jobs/1/results.csv into ax_client, result: {'RESULT': -93640.4547308622}: Saved results.csv
SOBOL, best RESULT: -270752.52, waiting for 1 job, finished 1 job : 10it [01:59, 11.99s/it]
Best RESULT, min (total: 2 + inserted jobs: 4)
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃OO_Info_int_param┃OO_Info_int_param_two┃OO_Info_float_param┃OO_Info_choice_param┃OO_Info_SLURM_JOB_ID┃int_param┃float_param┃choice_param┃int_param_two┃RESULT ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│-100.0 │-66.0 │-100.0 │1.0 │1170072.0 │-100 │-100.0 │1 │-66 │-270752.52│
└───────────────────┴───────────────────────┴─────────────────────┴──────────────────────┴──────────────────────┴───────────┴─────────────┴──────────────┴───────────────┴────────────┘
Runtime Infos
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃Number of evaluations┃Min time ┃Max time ┃Average time┃Median time┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ 2 │ 6.00 sec │ 7.00 sec │ 6.50 sec │ 6.50 sec │
└───────────────────────┴───────────┴───────────┴──────────────┴─────────────┘
2025-10-25 00:51:43 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, Started OmniOpt2 run...
2025-10-25 00:51:45 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, getting new HP set
2025-10-25 00:51:45 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, requested 1 jobs, got 1, 1.41 s/job
2025-10-25 00:51:48 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, eval #1/1 start
2025-10-25 00:51:48 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, starting new job
2025-10-25 00:51:48 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, unknown 1, started new job
2025-10-25 00:51:49 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, unknown 1, waiting for 1 job
2025-10-25 00:51:51 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, pending 1, waiting for 1 job
2025-10-25 00:53:11 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, pending 1, new result: RESULT: -131923.572806
2025-10-25 00:53:16 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, waiting for 1 job, finished 1 job
2025-10-25 00:53:18 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, getting new HP set
2025-10-25 00:53:18 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, requested 1 jobs, got 1, 1.10 s/job
2025-10-25 00:53:20 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, eval #1/1 start
2025-10-25 00:53:21 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, starting new job
2025-10-25 00:53:21 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, unknown 1, started new job
2025-10-25 00:53:22 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, unknown 1, waiting for 1 job
2025-10-25 00:53:26 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, running 1, waiting for 1 job
2025-10-25 00:53:40 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -165259.928217499, running 1, new result: RESULT: -270752.520000
2025-10-25 00:53:42 (95a3e8a9-1115-4eb8-b7ba-28bc490f8ffb): SOBOL, best RESULT: -270752.52, waiting for 1 job, finished 1 job
Job submission durations
┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┓
┃ Batch ┃ Seconds ┃ Jobs ┃ Time per job ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━┩
│ 1 │ 0.808 │ 1 │ 0.808 │
│ 2 │ 0.451 │ 1 │ 0.451 │
├─────────┼─────────┼──────┼──────────────┤
│ Average │ 0.630 │ │ │
│ Median │ 0.630 │ │ │
│ Total │ 1.259 │ │ │
│ Max │ 0.808 │ │ │
│ Min │ 0.451 │ │ │
└─────────┴─────────┴──────┴──────────────┘
Model generation times
┏━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┓
┃ Iteration ┃ Seconds ┃ Jobs ┃ Time per job ┃
┡━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━┩
│ 1 │ 1.410 │ 1 │ 1.410 │
│ 2 │ 1.098 │ 1 │ 1.098 │
├───────────┼─────────┼──────┼──────────────┤
│ Average │ 1.254 │ │ │
│ Median │ 1.254 │ │ │
│ Total │ 2.509 │ │ │
│ Max │ 1.410 │ │ │
│ Min │ 1.098 │ │ │
└───────────┴─────────┴──────┴──────────────┘