OmniOpt2

Experiment overview

Setting	Value
Model for non-random steps	BOTORCH_MODULAR
Max. nr. evaluations	500
Number random steps	20
Nr. of workers (parameter)	20
Main process memory (GB)	100
Worker memory (GB)	40

Job Summary per Generation Node

Generation Node	Total	FAILED	RUNNING
SOBOL	20	1	19

Experiment parameters

Name	Type	Lower bound	Upper bound	Values	Type	Log Scale?
epochs	range	10	30		int	No
lr	range	1e-06	0.001		float	No
batch_size	range	8	1024		int	No
hidden_size	range	8	4096		int	No
dropout	range	0	0.5		float	No
activation	fixed			leaky_relu
num_dense_layers	range	1	4		int	No
init	fixed			normal
weight_decay	fixed			0

Number of evaluations

Failed	Succeeded	Running	Total
1	0	19	20

Result names and types

name	min/max
VAL_ACC	max
RUNTIME	min

Last progressbar status

2025-09-03 14:08:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed

Git-Version

Commit: d1319821974a6b8830eaedcce29a62bdd0f28d54 (8506)

trial_index,submit_time,queue_time,worker_generator_uuid,start_time,end_time,run_time,program_string,VAL_ACC,RUNTIME,exit_code,signal,hostname,OO_Info_SLURM_JOB_ID,arm_name,trial_status,generation_node,epochs,lr,batch_size,hidden_size,dropout,num_dense_layers,activation,init,weight_decay
0,1756900915,53,07a3908a-56df-4580-9653-b5dc72534ea3,1756900968,1756900981,13,python3 .tests/mnist/train --epochs 22 --learning_rate 0.00075914370673894893 --batch_size 892 --hidden_size 1671 --dropout 0.00648493180051445961 --activation leaky_relu --num_dense_layers 3 --init normal --weight_decay 0,,,1,,c73,715634,0_0,FAILED,SOBOL,22,0.000759143706738948931267429998,892,1671,0.006484931800514459609985351562,3,leaky_relu,normal,0
1,1756900915,56,07a3908a-56df-4580-9653-b5dc72534ea3,1756900971,1756900995,24,python3 .tests/mnist/train --epochs 18 --learning_rate 0.00047653036180697389 --batch_size 59 --hidden_size 2248 --dropout 0.46610866999253630638 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0,,,1,,c137,715631,1_0,RUNNING,SOBOL,18,0.000476530361806973886545923946,59,2248,0.466108669992536306381225585938,1,leaky_relu,normal,0
2,1756900915,65,07a3908a-56df-4580-9653-b5dc72534ea3,1756900980,1756901023,43,python3 .tests/mnist/train --epochs 11 --learning_rate 0.00050981890391465286 --batch_size 756 --hidden_size 795 --dropout 0.28195986803621053696 --activation leaky_relu --num_dense_layers 2 --init normal --weight_decay 0,,,1,,c59,715635,2_0,RUNNING,SOBOL,11,0.000509818903914652861615597335,756,795,0.281959868036210536956787109375,2,leaky_relu,normal,0
3,1756900915,92,07a3908a-56df-4580-9653-b5dc72534ea3,1756901007,1756901020,13,python3 .tests/mnist/train --epochs 29 --learning_rate 0.00022733075708150866 --batch_size 452 --hidden_size 3412 --dropout 0.24543185904622077942 --activation leaky_relu --num_dense_layers 4 --init normal --weight_decay 0,,,1,,c130,715643,3_0,RUNNING,SOBOL,29,0.000227330757081508656866283835,452,3412,0.2454318590462207794189453125,4,leaky_relu,normal,0
4,1756900915,52,07a3908a-56df-4580-9653-b5dc72534ea3,1756900967,1756900991,24,python3 .tests/mnist/train --epochs 26 --learning_rate 0.00069654003607016069 --batch_size 246 --hidden_size 3742 --dropout 0.16659668926149606705 --activation leaky_relu --num_dense_layers 3 --init normal --weight_decay 0,,,1,,c101,715633,4_0,RUNNING,SOBOL,26,0.0006965400360701606884486492,246,3742,0.166596689261496067047119140625,3,leaky_relu,normal,0
5,1756900915,55,07a3908a-56df-4580-9653-b5dc72534ea3,1756900970,1756900994,24,python3 .tests/mnist/train --epochs 15 --learning_rate 0.00003856711840257049 --batch_size 931 --hidden_size 226 --dropout 0.36077966727316379547 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0,,,1,,c46,715637,5_0,RUNNING,SOBOL,15,0.000038567118402570488614432287,931,226,0.36077966727316379547119140625,1,leaky_relu,normal,0
6,1756900915,94,07a3908a-56df-4580-9653-b5dc72534ea3,1756901009,1756901015,6,python3 .tests/mnist/train --epochs 15 --learning_rate 0.00094574345167819428 --batch_size 362 --hidden_size 2829 --dropout 0.3872135956771671772 --activation leaky_relu --num_dense_layers 2 --init normal --weight_decay 0,,,1,,c137,715642,6_0,RUNNING,SOBOL,15,0.000945743451678194278244016235,362,2829,0.387213595677167177200317382812,2,leaky_relu,normal,0
7,1756900915,93,07a3908a-56df-4580-9653-b5dc72534ea3,1756901008,1756901014,6,python3 .tests/mnist/train --epochs 25 --learning_rate 0.00028788813546486202 --batch_size 558 --hidden_size 1362 --dropout 0.08536463556811213493 --activation leaky_relu --num_dense_layers 4 --init normal --weight_decay 0,,,1,,c46,715650,7_0,RUNNING,SOBOL,25,0.000287888135464862016804632461,558,1362,0.085364635568112134933471679688,4,leaky_relu,normal,0
8,1756900915,81,07a3908a-56df-4580-9653-b5dc72534ea3,1756900996,1756901027,31,python3 .tests/mnist/train --epochs 23 --learning_rate 0.00060206411137711264 --batch_size 270 --hidden_size 353 --dropout 0.49069864256307482719 --activation leaky_relu --num_dense_layers 4 --init normal --weight_decay 0,,,1,,c9,715641,8_0,RUNNING,SOBOL,23,0.000602064111377112638756603502,270,353,0.490698642563074827194213867188,4,leaky_relu,normal,0
9,1756900915,53,07a3908a-56df-4580-9653-b5dc72534ea3,1756900968,1756900992,24,python3 .tests/mnist/train --epochs 17 --learning_rate 0.00013212820798531175 --batch_size 586 --hidden_size 3869 --dropout 0.0445017707534134388 --activation leaky_relu --num_dense_layers 2 --init normal --weight_decay 0,,,1,,c130,715632,9_0,RUNNING,SOBOL,17,0.000132128207985311745724724131,586,3869,0.044501770753413438796997070312,2,leaky_relu,normal,0
10,1756900915,94,07a3908a-56df-4580-9653-b5dc72534ea3,1756901009,1756901015,6,python3 .tests/mnist/train --epochs 13 --learning_rate 0.00085224256213102501 --batch_size 138 --hidden_size 1234 --dropout 0.20349261723458766937 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0,,,1,,c73,715648,10_0,RUNNING,SOBOL,13,0.000852242562131025013262564105,138,1234,0.20349261723458766937255859375,1,leaky_relu,normal,0
11,1756900915,65,07a3908a-56df-4580-9653-b5dc72534ea3,1756900980,1756901023,43,python3 .tests/mnist/train --epochs 28 --learning_rate 0.00038242533200420444 --batch_size 974 --hidden_size 2702 --dropout 0.261261780746281147 --activation leaky_relu --num_dense_layers 3 --init normal --weight_decay 0,,,1,,c59,715636,11_0,RUNNING,SOBOL,28,0.000382425332004204441248285118,974,2702,0.261261780746281147003173828125,3,leaky_relu,normal,0
12,1756900915,60,07a3908a-56df-4580-9653-b5dc72534ea3,1756900975,1756900993,18,python3 .tests/mnist/train --epochs 30 --learning_rate 0.00091605766268540185 --batch_size 673 --hidden_size 2376 --dropout 0.33814278990030288696 --activation leaky_relu --num_dense_layers 4 --init normal --weight_decay 0,,,1,,c41,715638,12_0,RUNNING,SOBOL,30,0.000916057662685401848101651723,673,2376,0.338142789900302886962890625,4,leaky_relu,normal,0
13,1756900915,76,07a3908a-56df-4580-9653-b5dc72534ea3,1756900991,1756901027,36,python3 .tests/mnist/train --epochs 11 --learning_rate 0.00032053126609139153 --batch_size 471 --hidden_size 1799 --dropout 0.12662675511091947556 --activation leaky_relu --num_dense_layers 2 --init normal --weight_decay 0,,,1,,c105,715639,13_0,RUNNING,SOBOL,11,0.000320531266091391526737441886,471,1799,0.126626755110919475555419921875,2,leaky_relu,normal,0
14,1756900915,69,07a3908a-56df-4580-9653-b5dc72534ea3,1756900984,1756901015,31,python3 .tests/mnist/train --epochs 19 --learning_rate 0.00066575672778394085 --batch_size 793 --hidden_size 3284 --dropout 0.12144447257742285728 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0,,,1,,c92,715640,14_0,RUNNING,SOBOL,19,0.000665756727783940846672472436,793,3284,0.121444472577422857284545898438,1,leaky_relu,normal,0
15,1756900915,92,07a3908a-56df-4580-9653-b5dc72534ea3,1756901007,1756901019,12,python3 .tests/mnist/train --epochs 21 --learning_rate 0.00007035662622004747 --batch_size 95 --hidden_size 667 --dropout 0.41377108776941895485 --activation leaky_relu --num_dense_layers 3 --init normal --weight_decay 0,,,1,,c101,715644,15_0,RUNNING,SOBOL,21,0.000070356626220047471689626739,95,667,0.413771087769418954849243164062,3,leaky_relu,normal,0
16,1756900917,114,07a3908a-56df-4580-9653-b5dc72534ea3,1756901031,1756901056,25,python3 .tests/mnist/train --epochs 20 --learning_rate 0.00063404246163740758 --batch_size 619 --hidden_size 2992 --dropout 0.22102874284610152245 --activation leaky_relu --num_dense_layers 2 --init normal --weight_decay 0,,,1,,c80,715647,16_0,RUNNING,SOBOL,20,0.000634042461637407579744252217,619,2992,0.221028742846101522445678710938,2,leaky_relu,normal,0
17,1756900917,109,07a3908a-56df-4580-9653-b5dc72534ea3,1756901026,1756901051,25,python3 .tests/mnist/train --epochs 20 --learning_rate 0.00010106458863150329 --batch_size 300 --hidden_size 1520 --dropout 0.30638576904311776161 --activation leaky_relu --num_dense_layers 4 --init normal --weight_decay 0,,,1,,c83,715645,17_0,RUNNING,SOBOL,20,0.000101064588631503287936147517,300,1520,0.306385769043117761611938476562,4,leaky_relu,normal,0
18,1756900917,114,07a3908a-56df-4580-9653-b5dc72534ea3,1756901031,1756901056,25,python3 .tests/mnist/train --epochs 10 --learning_rate 0.00088336781060136872 --batch_size 996 --hidden_size 3645 --dropout 0.44169881567358970642 --activation leaky_relu --num_dense_layers 3 --init normal --weight_decay 0,,,1,,c55,715649,18_0,RUNNING,SOBOL,10,0.000883367810601368716807990644,996,3645,0.4416988156735897064208984375,3,leaky_relu,normal,0
19,1756900917,114,07a3908a-56df-4580-9653-b5dc72534ea3,1756901031,1756901049,18,python3 .tests/mnist/train --epochs 30 --learning_rate 0.0003502636425653473 --batch_size 180 --hidden_size 133 --dropout 0.03090231027454137802 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0,,,1,,c82,715646,19_0,RUNNING,SOBOL,30,0.000350263642565347302435668775,180,133,0.030902310274541378021240234375,1,leaky_relu,normal,0

To cancel, press CTRL c, then run 'scancel 715486'
Starting OmniOpt with MemRay.
Check later on with 'memray flamegraph 53241279-0e12-41a5-a201-76e7617cff8e.bin'
Collecting memray
  Downloading memray-1.18.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (8.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.2/8.2 MB 39.1 MB/s eta 0:00:00
Requirement already satisfied: jinja2>=2.9 in /home/s3811141/.omniax_venvs/Python_3.11.3/x86_64/lib/python3.11/site-packages (from memray) (3.1.6)
Requirement already satisfied: rich>=11.2.0 in /home/s3811141/.omniax_venvs/Python_3.11.3/x86_64/lib/python3.11/site-packages (from memray) (14.1.0)
Collecting textual>=0.41.0
  Using cached textual-6.1.0-py3-none-any.whl (707 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /home/s3811141/.omniax_venvs/Python_3.11.3/x86_64/lib/python3.11/site-packages (from jinja2>=2.9->memray) (3.0.2)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/s3811141/.omniax_venvs/Python_3.11.3/x86_64/lib/python3.11/site-packages (from rich>=11.2.0->memray) (4.0.0)
Requirement already satisfied: pygments=2.13.0 in /home/s3811141/.omniax_venvs/Python_3.11.3/x86_64/lib/python3.11/site-packages (from rich>=11.2.0->memray) (2.19.2)
Requirement already satisfied: platformdirs=3.6.0 in /home/s3811141/.omniax_venvs/Python_3.11.3/x86_64/lib/python3.11/site-packages (from textual>=0.41.0->memray) (4.4.0)
Requirement already satisfied: typing-extensions=4.4.0 in /home/s3811141/.omniax_venvs/Python_3.11.3/x86_64/lib/python3.11/site-packages (from textual>=0.41.0->memray) (4.14.1)
Requirement already satisfied: mdurl~=0.1 in /home/s3811141/.omniax_venvs/Python_3.11.3/x86_64/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=11.2.0->memray) (0.1.2)
Collecting linkify-it-py=1
  Using cached linkify_it_py-2.0.3-py3-none-any.whl (19 kB)
Collecting mdit-py-plugins>=0.5.0
  Using cached mdit_py_plugins-0.5.0-py3-none-any.whl (57 kB)
Collecting uc-micro-py
  Using cached uc_micro_py-1.0.3-py3-none-any.whl (6.2 kB)
Installing collected packages: uc-micro-py, mdit-py-plugins, linkify-it-py, textual, memray
Successfully installed linkify-it-py-2.0.3 mdit-py-plugins-0.5.0 memray-1.18.0 textual-6.1.0 uc-micro-py-1.0.3

[notice] A new release of pip available: 22.3.1 -> 25.2
[notice] To update, run: pip install --upgrade pip
Writing profile results into 53241279-0e12-41a5-a201-76e7617cff8e.bin
⠋ Importing logging...
⠋ Importing warnings...
⠋ Importing argparse...
⠋ Importing datetime...
⠋ Importing dataclass...
⠋ Importing socket...
⠋ Importing stat...
⠋ Importing pwd...
⠋ Importing base64...
⠋ Importing json...
⠋ Importing yaml...
⠋ Importing toml...
⠋ Importing csv...
⠋ Importing ast...
⠋ Importing rich.table...
⠋ Importing rich print...
⠋ Importing rich.pretty...
⠋ Importing rich.prompt...
⠋ Importing types.FunctionType...
⠋ Importing typing...
⠋ Importing ThreadPoolExecutor...
⠋ Importing submitit.LocalExecutor...
⠋ Importing submitit.Job...
⠋ Importing importlib.util...
⠋ Importing platform...
⠋ Importing inspect frame info...
⠋ Importing pathlib.Path...
⠋ Importing uuid...
⠋ Importing cowsay...
⠋ Importing shutil...
⠋ Importing itertools.combinations...
⠋ Importing os.listdir...
⠋ Importing os.path...
⠋ Importing PIL.Image...
⠋ Importing sixel...
⠋ Importing subprocess...
⠋ Importing tqdm...
⠸ Importing beartype...
⠋ Importing statistics...
⠋ Trying to import pyfiglet...
⠧ Importing helpers...
⠋ Parsing arguments...
⠹ Importing torch...
⠋ Importing numpy...
⠏ Importing ax...⚠ Memray support for Greenlet is experimental ⚠
Please report any issues at https://github.com/bloomberg/memray/issues


[WARNING 09-03 14:01:12] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
⠼ Importing ax...
⠋ Importing ax.core.generator_run...
⠋ Importing Cont_X_trans and Y_trans from ax.adapter.registry...
⠋ Importing ax.core.arm...
⠋ Importing ax.core.objective...
⠋ Importing ax.core.Metric...
⠋ Importing ax.exceptions.core...
⠋ Importing ax.exceptions.generation_strategy...
⠋ Importing CORE_DECODER_REGISTRY...
⠋ Trying ax.generation_strategy.generation_node...
⠋ Importing GenerationStep, GenerationStrategy from generation_strategy...
⠋ Importing GenerationNode from generation_node...
⠋ Importing ExternalGenerationNode...
⠋ Importing MaxTrials...
⠋ Importing GeneratorSpec...
⠋ Importing Models from ax.generation_strategy.registry...
⠋ Importing get_pending_observation_features...
⠋ Importing load_experiment...
⠋ Importing save_experiment...
⠋ Importing save_experiment_to_db...
⠋ Importing TrialStatus...
⠋ Importing Data...
⠋ Importing Experiment...
⠋ Importing parameter types...
⠋ Importing TParameterization...
⠋ Importing pandas...
⠋ Importing AxClient and ObjectiveProperties...
⠋ Importing RandomForestRegressor...
⠋ Importing botorch...
⠋ Importing submitit...
⠋ Importing ax logger...
⠋ Importing SQL-Storage-Stuff...
Run-UUID: 53241279-0e12-41a5-a201-76e7617cff8e
 .d88888b.                      d8b .d88888b.         888    .d8888b.  
d88P" "Y88b                     Y8Pd88P" "Y88b        888   d88P  Y88b 
888     888                        888     888        888          888 
888     88888888b.d88b. 88888b. 888888     88888888b. 888888     .d88P 
888     888888 "888 "88b888 "88b888888     888888 "88b888    .od888P"  
888     888888  888  888888  888888888     888888  888888   d88P"      
Y88b. .d88P888  888  888888  888888Y88b. .d88P888 d88PY88b. 888"       
 "Y88888P" 888  888  888888  888888 "Y88888P" 88888P"  "Y888888888888  
                                              888                      
                                              888                      
                                              888                      

⠋ Writing worker creation log...
omniopt --partition=alpha --experiment_name=mnist_multi --mem_gb=40 --time=1440 --worker_timeout=120 --max_eval=500 --num_parallel_jobs=20 --gpus=1 --num_random_steps=20 --follow --live_share --send_anonymized_usage_stats --result_names VAL_ACC=max RUNTIME=min --run_program=cHl0aG9uMyAudGVzdHMvbW5pc3QvdHJhaW4gLS1lcG9jaHMgJWVwb2NocyAtLWxlYXJuaW5nX3JhdGUgJWxyIC0tYmF0Y2hfc2l6ZSAlYmF0Y2hfc2l6ZSAtLWhpZGRlbl9zaXplICVoaWRkZW5fc2l6ZSAtLWRyb3BvdXQgJWRyb3BvdXQgLS1hY3RpdmF0aW9uICVhY3RpdmF0aW9uIC0tbnVtX2RlbnNlX2xheWVycyAlbnVtX2RlbnNlX2xheWVycyAtLWluaXQgJWluaXQgLS13ZWlnaHRfZGVjYXkgJXdlaWdodF9kZWNheQo= --cpus_per_task=1 --nodes_per_job=1 --revert_to_random_when_seemingly_exhausted --model=BOTORCH_MODULAR --n_estimators_randomforest=100 --run_mode=local --occ_type=euclid --main_process_gb=100 --max_nr_of_zero_results=50 --slurm_signal_delay_s=0 --max_failed_jobs=0 --max_attempts_for_generation=20 --num_restarts=20 --raw_samples=1024 --max_abandoned_retrial=20 --max_num_of_parallel_sruns=16 --number_of_generators=1 --generate_all_jobs_at_once --parameter epochs range 10 30 int false --parameter lr range 0.000001 0.001 float false --parameter batch_size range 8 1024 int false --parameter hidden_size range 8 4096 int false --parameter dropout range 0 0.5 float false --parameter activation fixed leaky_relu --parameter num_dense_layers range 1 4 int false --parameter init fixed normal --parameter weight_decay fixed 0
⠋ Disabling logging...
⠋ Setting run folder...
⠋ Creating folder /data/horse/ws/s3811141-oo_test_multi/omniopt/runs/mnist_multi/2...
⠋ Writing revert_to_random_when_seemingly_exhausted file ...
⠋ Writing username state file...
⠋ Writing result names file...
⠋ Writing result min/max file...
⠋ Saving state files...
Run-folder: /data/horse/ws/s3811141-oo_test_multi/omniopt/runs/mnist_multi/2
⠋ Writing live_share file if it is present...
⠋ Writing job_start_time file...
⠏ Writing git information
⠋ Checking max_eval...
⠋ Calculating number of steps...
⠋ Adding excluded nodes...
⠋ Setting global generation strategy
⠋ Initializing ax_client...
⠋ Setting orchestrator...
See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=mnist_multi&run_nr=1 for live-results.
⠇ Initializing live share...
⠇ Initializing live share...
You have 1 CPUs available for the main process. Using CUDA device NVIDIA H100.
Generation strategy: SOBOL for 20 steps and then BOTORCH_MODULAR for 480 steps.
Run-Program: python3 .tests/mnist/train --epochs %epochs --learning_rate %lr --batch_size %batch_size --hidden_size %hidden_size --dropout %dropout --activation %activation --num_dense_layers %num_dense_layers --init %init --weight_decay %weight_decay

                                  Experiment parameters                                   
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃ Name             ┃ Type  ┃ Lower bound ┃ Upper bound ┃ Values     ┃ Type  ┃ Log Scale? ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│ epochs           │ range │ 10          │ 30          │            │ int   │ No         │
│ lr               │ range │ 1e-06       │ 0.001       │            │ float │ No         │
│ batch_size       │ range │ 8           │ 1024        │            │ int   │ No         │
│ hidden_size      │ range │ 8           │ 4096        │            │ int   │ No         │
│ dropout          │ range │ 0           │ 0.5         │            │ float │ No         │
│ activation       │ fixed │             │             │ leaky_relu │       │            │
│ num_dense_layers │ range │ 1           │ 4           │            │ int   │ No         │
│ init             │ fixed │             │             │ normal     │       │            │
│ weight_decay     │ fixed │             │             │ 0          │       │            │
└──────────────────┴───────┴─────────────┴─────────────┴────────────┴───────┴────────────┘
        Result-Names         
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Result-Name ┃ Min or max? ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ VAL_ACC     │         min │
│ RUNTIME     │         min │
└─────────────┴─────────────┘
⠸ Write files and show overview
SOBOL, failed: 1 ('VAL_ACC: ' not found, 'RUNTIME: ' not found), completed 19 = ∑19/20, job_failed                         :   0%|░░░░░░░░░░| 0/500 [06:51

2025-09-03 14:01:53 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, Started OmniOpt2 run...
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #1/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #2/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #3/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #4/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #5/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #6/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #7/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #8/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #9/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #10/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #11/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #12/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #13/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #14/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #15/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #16/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #17/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #18/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #19/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, getting new HP set #20/20
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, requested 20 jobs, got 20, 0.06 s/job
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #1/20 start                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #2/20 start                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #3/20 start                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #4/20 start                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #5/20 start                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #6/20 start                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #7/20 start                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #8/20 start                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #9/20 start                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #10/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #11/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #12/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #13/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #14/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #15/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #16/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #17/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #18/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #19/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, eval #20/20 start                    
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, starting new job                     
2025-09-03 14:01:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 4 = ∑4/20, started new job   
2025-09-03 14:01:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 3 = ∑3/20, started new job   
2025-09-03 14:01:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 2 = ∑2/20, started new job   
2025-09-03 14:01:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 5 = ∑5/20, started new job   
2025-09-03 14:01:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 5 = ∑5/20, started new job   
2025-09-03 14:01:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 5 = ∑5/20, starting new job  
2025-09-03 14:01:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 5 = ∑5/20, starting new job  
2025-09-03 14:01:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 5 = ∑5/20, starting new job  
2025-09-03 14:01:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 5 = ∑5/20, starting new job  
2025-09-03 14:01:58 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 6 = ∑6/20, started new job   
2025-09-03 14:01:58 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 8 = ∑8/20, started new job   
2025-09-03 14:01:58 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 8 = ∑8/20, started new job   
2025-09-03 14:01:58 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 9 = ∑9/20, started new job   
2025-09-03 14:01:58 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 10 = ∑10/20, started new job 
2025-09-03 14:01:58 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, unknown 11 = ∑11/20, started new job 
2025-09-03 14:01:59 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/1 = ∑12/20, started new job
2025-09-03 14:01:59 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/2 = ∑13/20, started new job
2025-09-03 14:01:59 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/3 = ∑14/20, started new job
2025-09-03 14:01:59 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/4 = ∑15/20, started new job
2025-09-03 14:01:59 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/5 = ∑16/20, started new job
2025-09-03 14:01:59 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/6 = ∑17/20, started new job
2025-09-03 14:01:59 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/7 = ∑18/20, started new job
2025-09-03 14:02:00 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, started new job
2025-09-03 14:02:00 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, started new job
2025-09-03 14:02:01 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:02 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:03 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:03 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:04 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:04 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:05 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:06 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending/unknown 11/9 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:06 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:07 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:07 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:08 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:08 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:09 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:10 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:10 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:11 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:12 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:13 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:14 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:14 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:15 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:15 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:16 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:16 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:17 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:18 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:18 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:19 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:19 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:20 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:20 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:21 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:21 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:22 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:22 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:23 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:23 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:24 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:25 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:25 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:26 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:27 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:27 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs          
2025-09-03 14:02:30 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:31 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:31 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:32 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:32 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:33 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:34 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:34 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:35 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:35 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:36 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:36 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:37 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:38 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:38 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:39 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:39 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:40 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:47 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:47 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:48 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:48 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:49 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:50 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:50 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:51 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:51 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:52 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:52 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:53 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:54 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:54 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:55 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:57 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:58 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:59 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:02:59 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:03:00 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2025-09-03 14:03:05 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, running/completed/pending 10/1/9 = ∑20/20, job_failed
2025-09-03 14:08:40 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2025-09-03 14:08:41 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, waiting for 19 jobs                
2025-09-03 14:08:44 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:44 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:45 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:45 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:45 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:44 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:45 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:45 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:45 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:45 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:45 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:45 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed                         
2025-09-03 14:08:46 (07a3908a-56df-4580-9653-b5dc72534ea3): SOBOL, failed: 1 ('VAL_ACC: <FLOAT>' not found, 'RUNTIME: <FLOAT>' not found), completed 19 = ∑19/20, job_failed

Arguments Overview

Key	Value
config_yaml	None
config_toml	None
config_json	None
num_random_steps	20
max_eval	500
run_program	[['cHl0aG9uMyAudGVzdHMvbW5pc3QvdHJhaW4gLS1lcG9jaHMgJWVwb2NocyAtLWxlYXJuaW5nX3JhdGUgJWxyIC0tYmF0Y2hfc2l6ZSAlYmF0Y2hfc2l6ZSAtLWhpZGRlbl9zaXplICVoaWRkZW5f…
experiment_name	mnist_multi
mem_gb	40
parameter	[['epochs', 'range', '10', '30', 'int', 'false'], ['lr', 'range', '0.000001', '0.001', 'float', 'false'], ['batch_size', 'range', '8', '1024', 'int',
	'false'], ['hidden_size', 'range', '8', '4096', 'int', 'false'], ['dropout', 'range', '0', '0.5', 'float', 'false'], ['activation', 'fixed',
	'leaky_relu'], ['num_dense_layers', 'range', '1', '4', 'int', 'false'], ['init', 'fixed', 'normal'], ['weight_decay', 'fixed', '0']]
continue_previous_job	None
experiment_constraints	None
run_dir	runs
seed	None
verbose_tqdm	False
model	BOTORCH_MODULAR
gridsearch	False
occ	False
show_sixel_scatter	False
show_sixel_general	False
show_sixel_trial_index_result	False
follow	True
send_anonymized_usage_stats	True
ui_url	None
root_venv_dir	/home/s3811141
exclude	None
main_process_gb	100
max_nr_of_zero_results	50
abbreviate_job_names	False
orchestrator_file	None
checkout_to_latest_tested_version	False
live_share	True
disable_tqdm	False
disable_previous_job_constraint	False
workdir
occ_type	euclid
result_names	['VAL_ACC=max', 'RUNTIME=min']
minkowski_p	2
signed_weighted_euclidean_weights
generation_strategy	None
generate_all_jobs_at_once	True
revert_to_random_when_seemingly_exhausted	True
load_data_from_existing_jobs	[]
n_estimators_randomforest	100
max_attempts_for_generation	20
external_generator	None
username	None
max_failed_jobs	0
num_cpus_main_job	None
calculate_pareto_front_of_job	[]
show_generate_time_table	False
force_choice_for_ranges	False
max_abandoned_retrial	20
share_password	None
dryrun	False
db_url	None
run_program_once	None
worker_generator_path	None
save_to_database	False
range_max_difference	1000000
skip_search	False
dont_warm_start_refitting	False
refit_on_cv	False
fit_out_of_design	False
fit_abandoned	False
dont_jit_compile	False
num_restarts	20
raw_samples	1024
max_num_of_parallel_sruns	16
no_transform_inputs	False
no_normalize_y	False
transforms	[]
number_of_generators	1
num_parallel_jobs	20
worker_timeout	120
slurm_use_srun	False
time	1440
partition	alpha
reservation	None
force_local_execution	False
slurm_signal_delay_s	0
nodes_per_job	1
cpus_per_task	1
account	None
gpus	1
dependency	None
run_mode	local
verbose	False
verbose_break_run_search_table	False
debug	False
flame_graph	False
memray	False
no_sleep	False
tests	False
show_worker_percentage_table_at_end	False
auto_exclude_defective_hosts	False
run_tests_that_fail_on_taurus	False
raise_in_eval	False
show_ram_every_n_seconds	0
show_generation_and_submission_sixel	False
just_return_defaults	False
prettyprint	False
runtime_debug	False
debug_stack_regex
debug_stack_trace_regex	None
show_func_name	False

submitit INFO (2025-09-03 14:02:26,122) - Starting with JobEnvironment(job_id=715631, hostname=c137, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2025-09-03 14:02:26,125) - Loading pickle: /data/horse/ws/s3811141-oo_test_multi/omniopt/runs/mnist_multi/2/single_runs/715631/715631_submitted.pkl
ERROR: Could not install packages due to an OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/.torch_venv_821a313f65/lib/python3.11/site-packages/pip/_vendor/certifi/cacert.pem

WARNING: There was an error checking the latest version of pip.
Traceback (most recent call last):
  File "/data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/train", line 51, in ensure_venv_and_rich
    import torchvision
ModuleNotFoundError: No module named 'torchvision'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/train", line 57, in ensure_venv_and_rich
    subprocess.check_call([str(PYTHON_BIN), "-m", "pip", "install", "-q", "rich", "torch", "torchvision"])
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['.tests/mnist/.torch_venv_821a313f65/bin/python', '-m', 'pip', 'install', '-q', 'rich', 'torch', 'torchvision']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/train", line 65, in <module>
    ensure_venv_and_rich()
  File "/data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/train", line 59, in ensure_venv_and_rich
    shutil.rmtree(VENV_PATH)
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 732, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 3 more times]
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 666, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 664, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: '__pycache__'

Parameters: {"epochs": 18, "lr": 0.0004765303618069739, "batch_size": 59, "hidden_size": 2248, "dropout": 0.4661086699925363, "num_dense_layers": 1, "activation": "leaky_relu", "init": "normal", "weight_decay": 0}
Debug-Infos: 
========
DEBUG INFOS START:
Program-Code: python3 .tests/mnist/train --epochs 18 --learning_rate 0.00047653036180697389 --batch_size 59 --hidden_size 2248 --dropout 0.46610866999253630638 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0 
pwd: /data/horse/ws/s3811141-oo_test_multi/omniopt
File: .tests/mnist/train
UID: 2105408
GID: 200270
SLURM_JOB_ID: 715631
Status-Change-Time: 1756834716.0
Size: 12883 Bytes
Permissions: -rwxr-xr-x
Owner: s3811141
Last access: 1756900979.0
Last modification: 1756830959.0
Hostname: c137
========
DEBUG INFOS END

python3 .tests/mnist/train --epochs 18 --learning_rate 0.00047653036180697389 --batch_size 59 --hidden_size 2248 --dropout 0.46610866999253630638 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0 
stdout was empty
stderr:
 ERROR: Could not install packages due to an OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/.torch_venv_821a313f65/lib/python3.11/site-packages/pip/_vendor/certifi/cacert.pem

WARNING: There was an error checking the latest version of pip.
Traceback (most recent call last):
  File "/data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/train", line 51, in ensure_venv_and_rich
    import torchvision
ModuleNotFoundError: No module named 'torchvision'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/train", line 57, in ensure_venv_and_rich
    subprocess.check_call([str(PYTHON_BIN), "-m", "pip", "install", "-q", "rich", "torch", "torchvision"])
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['.tests/mnist/.torch_venv_821a313f65/bin/python', '-m', 'pip', 'install', '-q', 'rich', 'torch', 'torchvision']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/train", line 65, in <module>
    ensure_venv_and_rich()
  File "/data/horse/ws/s3811141-oo_test_multi/omniopt/.tests/mnist/train", line 59, in ensure_venv_and_rich
    shutil.rmtree(VENV_PATH)
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 732, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 3 more times]
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 666, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/software/genoa/r24.04/Python/3.11.3-GCCcore-12.3.0/lib/python3.11/shutil.py", line 664, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: '__pycache__'

Result: {'VAL_ACC': None, 'RUNTIME': None}
Final-results: {'VAL_ACC': None, 'RUNTIME': None}
EXIT_CODE: 1
submitit INFO (2025-09-03 14:03:16,009) - Job completed successfully
submitit INFO (2025-09-03 14:03:16,011) - Exiting after successful completion