OmniOpt2

Experiment overview

Setting	Value
Model for non-random steps	BOTORCH_MODULAR
Max. nr. evaluations	500
Number random steps	20
Nr. of workers (parameter)	20
Main process memory (GB)	8
Worker memory (GB)	10

Job Summary per Generation Node

Generation Node	Total	FAILED	RUNNING
SOBOL	7	5	2

Experiment parameters

Name	Type	Lower bound	Upper bound	Values	Type	Log Scale?
epochs	range	10	200		int	No
lr	range	1e-05	0.1		float	No
batch_size	range	8	2048		int	No
hidden_size	range	8	2048		int	No
dropout	range	0	0.5		float	No
activation	fixed			leaky_relu
num_dense_layers	range	1	4		int	No
init	fixed			normal
weight_decay	range	0	1		float	No

Number of evaluations

Failed	Succeeded	Running	Total
5	0	2	7

Result names and types

name	min/max
VAL_ACC	max

Last progressbar status

2025-07-31 16:43:28: Sobol, failed: 5 ('VAL_ACC: <FLOAT>' not found), running 1∑1 (5%/20), getting new HP set

Git-Version

Commit: f9547a580b93e0983ebff52a5b7750569294ad57

trial_index,submit_time,queue_time,start_time,end_time,run_time,program_string,VAL_ACC,exit_code,signal,hostname,OO_Info_SLURM_JOB_ID,arm_name,trial_status,generation_node,epochs,lr,batch_size,hidden_size,dropout,num_dense_layers,weight_decay,activation,init
0,1753972675,21,1753972696,1753972702,6,python3 .tests/mnist/train --epochs 10 --learning_rate 0.06942201295375824421 --batch_size 318 --hidden_size 1494 --dropout 0.06775018572807312012 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0.61139845848083496094,,1,,c150,531140,0_0,FAILED,SOBOL,10,0.069422012953758244213631201092,318,1494,0.0677501857280731201171875,1,0.6113984584808349609375,leaky_relu,normal
1,1753972717,8,1753972725,1753972731,6,python3 .tests/mnist/train --epochs 192 --learning_rate 0.01052411921357736084 --batch_size 1662 --hidden_size 12 --dropout 0.26961048133671283722 --activation leaky_relu --num_dense_layers 4 --init normal --weight_decay 0.39579123072326183319,,1,,c150,531142,1_0,FAILED,SOBOL,192,0.010524119213577360840150554111,1662,12,0.26961048133671283721923828125,4,0.39579123072326183319091796875,leaky_relu,normal
2,1753972804,12,1753972816,1753972822,6,python3 .tests/mnist/train --epochs 150 --learning_rate 0.09792287419735454246 --batch_size 918 --hidden_size 1577 --dropout 0.45615548733621835709 --activation leaky_relu --num_dense_layers 3 --init normal --weight_decay 0.97387396544218063354,,1,,c137,531145,2_0,FAILED,SOBOL,150,0.097922874197354542458171522412,918,1577,0.456155487336218357086181640625,3,0.973873965442180633544921875,leaky_relu,normal
3,1753972839,5,1753972844,1753972851,7,python3 .tests/mnist/train --epochs 69 --learning_rate 0.03201941623316147123 --batch_size 1246 --hidden_size 1013 --dropout 0.12737619783729314804 --activation leaky_relu --num_dense_layers 2 --init normal --weight_decay 0.00234625302255153656,,1,,c137,531150,3_0,FAILED,SOBOL,69,0.032019416233161471230239669694,1246,1013,0.127376197837293148040771484375,2,0.00234625302255153656005859375,leaky_relu,normal
4,1753972916,20,1753972936,1753972942,6,python3 .tests/mnist/train --epochs 94 --learning_rate 0.07578967309001834185 --batch_size 1960 --hidden_size 747 --dropout 0.21482571214437484741 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0.82084993179887533188,,1,,c137,531152,4_0,FAILED,SOBOL,94,0.075789673090018341850004901517,1960,747,0.214825712144374847412109375,1,0.820849931798875331878662109375,leaky_relu,normal
5,,,,,,,,,,,,5_0,RUNNING,SOBOL,109,0.041166075762640690072213800477,243,1844,0.38531420938670635223388671875,4,0.171710425056517124176025390625,leaky_relu,normal
6,,,,,,,,,,,,6_0,RUNNING,SOBOL,161,0.053561658112024894407010577879,1285,279,0.325688947923481464385986328125,3,0.718390106223523616790771484375,leaky_relu,normal

To cancel, press CTRL c, then run 'scancel 531136'
⠋ Importing logging...
⠋ Importing warnings...
⠋ Importing argparse...
⠋ Importing datetime...
⠋ Importing dataclass...
⠋ Importing hashlib...
⠋ Importing socket...
⠋ Importing stat...
⠋ Importing pwd...
⠋ Importing signal...
⠋ Importing base64...
⠋ Importing json...
⠋ Importing yaml...
⠋ Importing toml...
⠋ Importing csv...
⠋ Importing ast...
⠋ Importing rich.table...
⠋ Importing rich print...
⠋ Importing rich.pretty...
⠋ Importing rich.prompt...
⠋ Importing types.FunctionType...
⠋ Importing typing...
⠋ Importing ThreadPoolExecutor...
⠋ Importing submitit.LocalExecutor...
⠋ Importing submitit.Job...
⠋ Importing importlib.util...
⠋ Importing inspect...
⠋ Importing platform...
⠋ Importing inspect frame info...
⠋ Importing pathlib.Path...
⠋ Importing uuid...
⠋ Importing traceback...
⠋ Importing cowsay...
⠋ Importing psutil...
⠋ Importing shutil...
⠋ Importing itertools.combinations...
⠋ Importing os.listdir...
⠋ Importing os.path...
⠋ Importing PIL.Image...
⠋ Importing sixel...
⠋ Importing subprocess...
⠋ Importing tqdm...
⠇ Importing beartype...
⠋ Importing statistics...
⠋ Trying to import pyfiglet...
⠹ Importing helpers...
⠋ Parsing arguments...
⠸ Importing torch...
⠋ Importing numpy...
⠋ Importing collections...
⠸ Importing ax...
⠋ Importing ax.core.generator_run...
⠋ Importing Cont_X_trans and Y_trans from ax.modelbridge.registry...
⠋ Importing ax.core.arm...
⠋ Importing ax.core.objective...
⠋ Importing ax.core.Metric...
⠋ Importing ax.exceptions.core...
⠋ Importing ax.exceptions.generation_strategy...
⠋ Importing CORE_DECODER_REGISTRY...
⠋ Trying ax.generation_strategy.generation_node...
⠋ Importing GenerationStep, GenerationStrategy from generation_strategy...
⠋ Importing GenerationNode from generation_node...
⠋ Importing ExternalGenerationNode...
⠋ Importing MaxTrials...
⠋ Importing GeneratorSpec...
⠋ Importing Models from ax.modelbridge.registry...
⠋ Importing get_pending_observation_features...
⠋ Importing load_experiment...
⠋ Importing save_experiment...
⠋ Importing save_experiment_to_db...
⠋ Importing TrialStatus...
⠋ Importing Data...
⠋ Importing Experiment...
⠋ Importing parameter types...
⠋ Importing TParameterization...
⠋ Importing pandas...
⠋ Importing AxClient and ObjectiveProperties...
⠋ Importing RandomForestRegressor...
⠋ Importing botorch...
⠋ Importing submitit...
⠋ Importing ax logger...
⠋ Importing SQL-Storage-Stuff...
Run-UUID: a0ce8e37-d6f7-4153-8bcd-acf36dc0fa44
 _____                           _____           __       ___     
/\  __`\                      __/\  __`\        /\ \__  /'___`\   
\ \ \/\ \    ___ ___     ___ /\_\ \ \/\ \  _____\ \ ,_\/\_\ /\ \  
 \ \ \ \ \ /' __` __`\ /' _ `\/\ \ \ \ \ \/\ '__`\ \ \/\/_/// /__ 
  \ \ \_\ \/\ \/\ \/\ \/\ \/\ \ \ \ \ \_\ \ \ \L\ \ \ \_  // /_\ \
   \ \_____\ \_\ \_\ \_\ \_\ \_\ \_\ \_____\ \ ,__/\ \__\/\______/
    \/_____/\/_/\/_/\/_/\/_/\/_/\/_/\/_____/\ \ \/  \/__/\/_____/ 
                                             \ \_\                
                                              \/_/                

⠋ Writing worker creation log...
omniopt --partition=alpha --experiment_name=mnist_gpu_noall --mem_gb=10 --time=2880 --worker_timeout=120 --max_eval=500 --num_parallel_jobs=20 --gpus=1 --num_random_steps=20 --follow --live_share --send_anonymized_usage_stats --result_names VAL_ACC=max --run_program=cHl0aG9uMyAudGVzdHMvbW5pc3QvdHJhaW4gLS1lcG9jaHMgJWVwb2NocyAtLWxlYXJuaW5nX3JhdGUgJWxyIC0tYmF0Y2hfc2l6ZSAlYmF0Y2hfc2l6ZSAtLWhpZGRlbl9zaXplICVoaWRkZW5fc2l6ZSAtLWRyb3BvdXQgJWRyb3BvdXQgLS1hY3RpdmF0aW9uICVhY3RpdmF0aW9uIC0tbnVtX2RlbnNlX2xheWVycyAlbnVtX2RlbnNlX2xheWVycyAtLWluaXQgJWluaXQgLS13ZWlnaHRfZGVjYXkgJXdlaWdodF9kZWNheQ== --cpus_per_task=1 --nodes_per_job=1 --revert_to_random_when_seemingly_exhausted --model=BOTORCH_MODULAR --n_estimators_randomforest=100 --run_mode=local --occ_type=euclid --main_process_gb=8 --max_nr_of_zero_results=50 --slurm_signal_delay_s=0 --max_failed_jobs=0 --max_attempts_for_generation=20 --num_restarts=20 --raw_samples=1024 --max_abandoned_retrial=20 --max_num_of_parallel_sruns=16 --parameter epochs range 10 200 int false --parameter lr range 0.00001 0.1 float false --parameter batch_size range 8 2048 int false --parameter hidden_size range 8 2048 int false --parameter dropout range 0 0.5 float false --parameter activation fixed leaky_relu --parameter num_dense_layers range 1 4 int false --parameter init fixed normal --parameter weight_decay range 0 1 float false --ui_url aHR0cHM6Ly9pbWFnZXNlZy5zY2Fkcy5kZS9vbW5pYXgvZ3VpP3BhcnRpdGlvbj1hbHBoYSZleHBlcmltZW50X25hbWU9bW5pc3RfZ3B1X25vYWxsJnJlc2VydmF0aW9uPSZhY2NvdW50PSZtZW1fZ2I9MTAmdGltZT0yODgwJndvcmtlcl90aW1lb3V0PTEyMCZtYXhfZXZhbD01MDAmbnVtX3BhcmFsbGVsX2pvYnM9MjAmZ3B1cz0xJm51bV9yYW5kb21fc3RlcHM9MjAmZm9sbG93PTEmbGl2ZV9zaGFyZT0xJnNlbmRfYW5vbnltaXplZF91c2FnZV9zdGF0cz0xJmNvbnN0cmFpbnRzPSZyZXN1bHRfbmFtZXM9VkFMX0FDQyUzRG1heCZydW5fcHJvZ3JhbT1weXRob24zJTIwLnRlc3RzJTJGbW5pc3QlMkZ0cmFpbiUyMC0tZXBvY2hzJTIwJTI1ZXBvY2hzJTIwLS1sZWFybmluZ19yYXRlJTIwJTI1bHIlMjAtLWJhdGNoX3NpemUlMjAlMjViYXRjaF9zaXplJTIwLS1oaWRkZW5fc2l6ZSUyMCUyNWhpZGRlbl9zaXplJTIwLS1kcm9wb3V0JTIwJTI1ZHJvcG91dCUyMC0tYWN0aXZhdGlvbiUyMCUyNWFjdGl2YXRpb24lMjAtLW51bV9kZW5zZV9sYXllcnMlMjAlMjVudW1fZGVuc2VfbGF5ZXJzJTIwLS1pbml0JTIwJTI1aW5pdCUyMC0td2VpZ2h0X2RlY2F5JTIwJTI1d2VpZ2h0X2RlY2F5JmNwdXNfcGVyX3Rhc2s9MSZub2Rlc19wZXJfam9iPTEmc2VlZD0mZHJ5cnVuPTAmZGVidWc9MCZyZXZlcnRfdG9fcmFuZG9tX3doZW5fc2VlbWluZ2x5X2V4aGF1c3RlZD0xJmdyaWRzZWFyY2g9MCZtb2RlbD1CT1RPUkNIX01PRFVMQVImZXh0ZXJuYWxfZ2VuZXJhdG9yPSZuX2VzdGltYXRvcnNfcmFuZG9tZm9yZXN0PTEwMCZpbnN0YWxsYXRpb25fbWV0aG9kPWNsb25lJnJ1bl9tb2RlPWxvY2FsJmRpc2FibGVfdHFkbT0wJnZlcmJvc2VfdHFkbT0wJmZvcmNlX2xvY2FsX2V4ZWN1dGlvbj0wJmF1dG9fZXhjbHVkZV9kZWZlY3RpdmVfaG9zdHM9MCZzaG93X3NpeGVsX2dlbmVyYWw9MCZzaG93X3NpeGVsX3RyaWFsX2luZGV4X3Jlc3VsdD0wJnNob3dfc2l4ZWxfc2NhdHRlcj0wJnNob3dfd29ya2VyX3BlcmNlbnRhZ2VfdGFibGVfYXRfZW5kPTAmb2NjPTAmb2NjX3R5cGU9ZXVjbGlkJm5vX3NsZWVwPTAmc2x1cm1fdXNlX3NydW49MCZ2ZXJib3NlX2JyZWFrX3J1bl9zZWFyY2hfdGFibGU9MCZhYmJyZXZpYXRlX2pvYl9uYW1lcz0wJm1haW5fcHJvY2Vzc19nYj04Jm1heF9ucl9vZl96ZXJvX3Jlc3VsdHM9NTAmc2x1cm1fc2lnbmFsX2RlbGF5X3M9MCZtYXhfZmFpbGVkX2pvYnM9MCZleGNsdWRlPSZ1c2VybmFtZT0mZ2VuZXJhdGlvbl9zdHJhdGVneT0mcm9vdF92ZW52X2Rpcj0md29ya2Rpcj0mZG9udF9qaXRfY29tcGlsZT0wJmZpdF9vdXRfb2ZfZGVzaWduPTAmcmVmaXRfb25fY3Y9MCZzaG93X2dlbmVyYXRlX3RpbWVfdGFibGU9MCZkb250X3dhcm1fc3RhcnRfcmVmaXR0aW5nPTAmbWF4X2F0dGVtcHRzX2Zvcl9nZW5lcmF0aW9uPTIwJm51bV9yZXN0YXJ0cz0yMCZyYXdfc2FtcGxlcz0xMDI0Jm1heF9hYmFuZG9uZWRfcmV0cmlhbD0yMCZtYXhfbnVtX29mX3BhcmFsbGVsX3NydW5zPTE2JmZvcmNlX2Nob2ljZV9mb3JfcmFuZ2VzPTAmbm9fdHJhbnNmb3JtX2lucHV0cz0wJmZpdF9hYmFuZG9uZWQ9MCZub19ub3JtYWxpemVfeT0wJnZlcmJvc2U9MCZnZW5lcmF0ZV9hbGxfam9ic19hdF9vbmNlPTAmZmxhbWVfZ3JhcGg9MCZjaGVja291dF90b19sYXRlc3RfdGVzdGVkX3ZlcnNpb249MCZwYXJhbWV0ZXJfMF9uYW1lPWVwb2NocyZwYXJhbWV0ZXJfMF90eXBlPXJhbmdlJnBhcmFtZXRlcl8wX21pbj0xMCZwYXJhbWV0ZXJfMF9tYXg9MjAwJnBhcmFtZXRlcl8wX251bWJlcl90eXBlPWludCZwYXJhbWV0ZXJfMF9sb2dfc2NhbGU9ZmFsc2UmcGFyYW1ldGVyXzFfbmFtZT1sciZwYXJhbWV0ZXJfMV90eXBlPXJhbmdlJnBhcmFtZXRlcl8xX21pbj0wLjAwMDAxJnBhcmFtZXRlcl8xX21heD0wLjEmcGFyYW1ldGVyXzFfbnVtYmVyX3R5cGU9ZmxvYXQmcGFyYW1ldGVyXzFfbG9nX3NjYWxlPWZhbHNlJnBhcmFtZXRlcl8yX25hbWU9YmF0Y2hfc2l6ZSZwYXJhbWV0ZXJfMl90eXBlPXJhbmdlJnBhcmFtZXRlcl8yX21pbj04JnBhcmFtZXRlcl8yX21heD0yMDQ4JnBhcmFtZXRlcl8yX251bWJlcl90eXBlPWludCZwYXJhbWV0ZXJfMl9sb2dfc2NhbGU9ZmFsc2UmcGFyYW1ldGVyXzNfbmFtZT1oaWRkZW5fc2l6ZSZwYXJhbWV0ZXJfM190eXBlPXJhbmdlJnBhcmFtZXRlcl8zX21pbj04JnBhcmFtZXRlcl8zX21heD0yMDQ4JnBhcmFtZXRlcl8zX251bWJlcl90eXBlPWludCZwYXJhbWV0ZXJfM19sb2dfc2NhbGU9ZmFsc2UmcGFyYW1ldGVyXzRfbmFtZT1kcm9wb3V0JnBhcmFtZXRlcl80X3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzRfbWluPTAmcGFyYW1ldGVyXzRfbWF4PTAuNSZwYXJhbWV0ZXJfNF9udW1iZXJfdHlwZT1mbG9hdCZwYXJhbWV0ZXJfNF9sb2dfc2NhbGU9ZmFsc2UmcGFyYW1ldGVyXzVfbmFtZT1hY3RpdmF0aW9uJnBhcmFtZXRlcl81X3R5cGU9Zml4ZWQmcGFyYW1ldGVyXzVfdmFsdWU9bGVha3lfcmVsdSZwYXJhbWV0ZXJfNl9uYW1lPW51bV9kZW5zZV9sYXllcnMmcGFyYW1ldGVyXzZfdHlwZT1yYW5nZSZwYXJhbWV0ZXJfNl9taW49MSZwYXJhbWV0ZXJfNl9tYXg9NCZwYXJhbWV0ZXJfNl9udW1iZXJfdHlwZT1pbnQmcGFyYW1ldGVyXzZfbG9nX3NjYWxlPWZhbHNlJnBhcmFtZXRlcl83X25hbWU9aW5pdCZwYXJhbWV0ZXJfN190eXBlPWZpeGVkJnBhcmFtZXRlcl83X3ZhbHVlPW5vcm1hbCZwYXJhbWV0ZXJfOF9uYW1lPXdlaWdodF9kZWNheSZwYXJhbWV0ZXJfOF90eXBlPXJhbmdlJnBhcmFtZXRlcl84X21pbj0wJnBhcmFtZXRlcl84X21heD0xJnBhcmFtZXRlcl84X251bWJlcl90eXBlPWZsb2F0JnBhcmFtZXRlcl84X2xvZ19zY2FsZT1mYWxzZSZwYXJ0aXRpb249YWxwaGEmbnVtX3BhcmFtZXRlcnM9OQ==
⠋ Disabling logging...
⠋ Setting run folder...
⠋ Creating folder /data/cat/ws/pwinkler-mnist_tst/omniopt/runs/mnist_gpu_noall/1...
⠋ Writing revert_to_random_when_seemingly_exhausted file ...
⠋ Writing username state file...
⠋ Writing result names file...
⠋ Writing result min/max file...
⠋ Saving state files...
Run-folder: /data/cat/ws/pwinkler-mnist_tst/omniopt/runs/mnist_gpu_noall/1
⠋ Printing run info...
⠋ Initializing NVIDIA-Logs...
⠋ Writing ui_url file if it is present...
⠋ Writing live_share file if it is present...
⠋ Writing job_start_time file...
⠹ Writing git info file...
⠋ Checking max_eval...
⠋ Calculating number of steps...
⠋ Adding excluded nodes...
⠋ Handling random steps...
⠋ Initializing ax_client...
[WARNING 07-31 16:37:04] ax.service.ax_client: Selecting a GenerationStrategy when using BatchTrials is in beta. Double check the recommended strategy matches your expectations.
⠋ Setting orchestrator...
You have 1 CPUs available for the main process. Using CUDA device NVIDIA H100. Generation strategy: SOBOL for 20 steps and then BOTORCH_MODULAR for 480 steps.
Run-Program: python3 .tests/mnist/train --epochs %epochs --learning_rate %lr --batch_size %batch_size --hidden_size %hidden_size --dropout %dropout --activation %activation --num_dense_layers %num_dense_layers --init %init --weight_decay %weight_decay
                                  Experiment parameters                                   
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃ Name             ┃ Type  ┃ Lower bound ┃ Upper bound ┃ Values     ┃ Type  ┃ Log Scale? ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│ epochs           │ range │ 10          │ 200         │            │ int   │ No         │
│ lr               │ range │ 1e-05       │ 0.1         │            │ float │ No         │
│ batch_size       │ range │ 8           │ 2048        │            │ int   │ No         │
│ hidden_size      │ range │ 8           │ 2048        │            │ int   │ No         │
│ dropout          │ range │ 0           │ 0.5         │            │ float │ No         │
│ activation       │ fixed │             │             │ leaky_relu │       │            │
│ num_dense_layers │ range │ 1           │ 4           │            │ int   │ No         │
│ init             │ fixed │             │             │ normal     │       │            │
│ weight_decay     │ range │ 0           │ 1           │            │ float │ No         │
└──────────────────┴───────┴─────────────┴─────────────┴────────────┴───────┴────────────┘
        Result-Names         
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Result-Name ┃ Min or max? ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ VAL_ACC     │         max │
└─────────────┴─────────────┘
See https://imageseg.scads.de/omniax/share?user_id=pwinkler&experiment_name=mnist_gpu_noall&run_nr=8 for live-results.

█▀▀▀▀▀█  ▀▄▄████ ▄▀█ ▀██▄▀██▀ █▀▀▀▀▀█
█ ███ █ ▀██▄▀ ▀   ▄▀  █ ▀ ▄ █ █ ███ █
█ ▀▀▀ █  ▄▀   ██▄███▄▀▀█▄  █▄ █ ▀▀▀ █
▀▀▀▀▀▀▀ █▄█ █▄▀▄▀▄█ █▄█ █▄█▄▀ ▀▀▀▀▀▀▀
█▀▀ ██▀▀█ ██ █▄ ██ █▄▀  ▄▀  ███  ▄█ ▄
▀▀█▀▄▄▀ ▀▄ ▄ █▀ ▄▄▄▄█▄ ▄▀▄▀▄  ██ ▄  █
█▀ ▀▀ ▀▀█▀▀▄██▄▀█▀█ ▄█  ▄█ ▀███▀  ▄ █
▀▄▄ █ ▀▀█▀ ▀█▄▄ █▄▄ █▄▄▄▄▄█ █ ▀ ▄█ █▀
 █▄█▀▀▀ ▀  █ ▀ ▀█▀ ▀██▄▀ █▄▀███▄  ▄ █
▄▄█▀▀ ▀▀ ▄▀▄ █  ▄▄▄▄▄█▀▄██▀▄▄█▀▀ ▀ █▀
▀▀▄ ██▀▀▀▀ ▄█ ▄█▄▀  ██▀ ██ ▀█▀█  ▄▄▀█
 ▄▀ ▀ ▀ ▄▀█▀██▄▄▄█ ▄▄█ ▄ ▄▄ ▄ █▀▀▀▀█▀
▀▀▄▀█▄▀▀  ▄█ ▀  ▄▀ ▀██ ▀█▀█▀█▄█   ▄▀█
▀▄ ▀▄ ▀▄▄▄▄▄ █ ▄▀▄█▄ ▄▄ ▄█▄ ▄█▄█ ▀▀█▀
▀ ▀  ▀▀ ▄█ ▄██▄ ▄█▄▀█▀▄█▄█▄██▀▀▀█▄ ▄▄
█▀▀▀▀▀█ █▀▄▀██▀ ▄██ ▀▄▀ ▀▄▄██ ▀ █▀▀█▀
█ ███ █ ▀███▀▀█ ██▄ ▄█▄ ▄█▄ ▀▀▀▀█  ▄█
█ ▀▀▀ █ ███▄▀▄▄ ▄▄ ▄▄▄▀▄▄████  ▀▀▀ █▀
▀▀▀▀▀▀▀ ▀  ▀ ▀▀▀    ▀▀  ▀▀ ▀▀ ▀▀▀  ▀▀
Sobol, failed: 4 ('VAL_ACC: ' not found), completed/unknown 1/1∑2 (5%/20), started new job                        :   0%|░░░░░░░░░░| 0/500 [05:53failed: 5 ('VAL_ACC: ' not found), running 1∑1 (5%/20), getting new HP set                                 :   0%|░░░░░░░░░░| 0/500 [06:18

2025-07-31 16:37:09: SOBOL, Started OmniOpt2 run...
2025-07-31 16:37:18: Sobol, getting new HP set     
2025-07-31 16:37:39: Sobol, requested 1 jobs, got 1, 21.14 s/job
2025-07-31 16:37:45: Sobol, eval #1/1 start                     
2025-07-31 16:37:50: Sobol, starting new job                    
2025-07-31 16:37:56: Sobol, unknown 1∑1 (5%/20), started new job
2025-07-31 16:38:10: Sobol, pending 1∑1 (5%/20), getting new HP set
2025-07-31 16:38:18: Sobol, pending 1∑1 (5%/20), requested 1 jobs, got 1, 8.42 s/job
2025-07-31 16:38:23: Sobol, pending 1∑1 (0%/20), eval #1/1 start                    
2025-07-31 16:38:32: Sobol, completed 1∑1 (0%/20), starting new job                 
2025-07-31 16:38:38: Sobol, completed/unknown 1/1∑2 (5%/20), started new job        
2025-07-31 16:39:01: Sobol, completed 2∑2 (0%/20), job_failed                       
2025-07-31 16:39:01: Sobol, completed 2∑2 (0%/20), job_failed                       
2025-07-31 16:39:36: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), finishing jobs (_get_next_trials), finished 2 jobs
2025-07-31 16:39:41: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), getting new HP set                                
2025-07-31 16:39:50: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), requested 1 jobs, got 1, 9.45 s/job               
2025-07-31 16:39:55: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), eval #1/1 start                                   
2025-07-31 16:40:00: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), starting new job                                  
2025-07-31 16:40:06: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), unknown 1∑1 (5%/20), started new job              
2025-07-31 16:40:11: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), pending 1∑1 (5%/20), getting new HP set           
2025-07-31 16:40:20: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), running 1∑1 (5%/20), requested 1 jobs, got 1, 9.34 s/job
2025-07-31 16:40:25: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), running 1∑1 (0%/20), eval #1/1 start                    
2025-07-31 16:40:33: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), running 1∑1 (0%/20), starting new job                   
2025-07-31 16:40:40: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), completed/unknown 1/1∑2 (5%/20), started new job        
2025-07-31 16:40:45: Sobol, failed: 2 ('VAL_ACC: <FLOAT>' not found), completed/running 1/1∑2 (5%/20), job_failed             
2025-07-31 16:41:17: Sobol, failed: 3 ('VAL_ACC: <FLOAT>' not found), completed 1∑1 (0%/20), finishing jobs (_get_next_trials), finished 1 job
2025-07-31 16:41:22: Sobol, failed: 3 ('VAL_ACC: <FLOAT>' not found), completed 1∑1 (0%/20), getting new HP set                               
2025-07-31 16:41:31: Sobol, failed: 3 ('VAL_ACC: <FLOAT>' not found), completed 1∑1 (0%/20), requested 1 jobs, got 1, 9.31 s/job              
2025-07-31 16:41:36: Sobol, failed: 3 ('VAL_ACC: <FLOAT>' not found), completed 1∑1 (0%/20), eval #1/1 start                                  
2025-07-31 16:41:45: Sobol, failed: 3 ('VAL_ACC: <FLOAT>' not found), completed 1∑1 (0%/20), starting new job                                 
2025-07-31 16:41:58: Sobol, failed: 3 ('VAL_ACC: <FLOAT>' not found), completed/unknown 1/1∑2 (5%/20), started new job                        
2025-07-31 16:42:04: Sobol, failed: 3 ('VAL_ACC: <FLOAT>' not found), completed/pending 1/1∑2 (5%/20), job_failed                             
2025-07-31 16:42:22: Sobol, failed: 4 ('VAL_ACC: <FLOAT>' not found), running 1∑1 (5%/20), finishing jobs (_get_next_trials), finished 1 job  
2025-07-31 16:42:35: Sobol, failed: 4 ('VAL_ACC: <FLOAT>' not found), running 1∑1 (0%/20), getting new HP set                                 
2025-07-31 16:42:45: Sobol, failed: 4 ('VAL_ACC: <FLOAT>' not found), running 1∑1 (0%/20), requested 1 jobs, got 1, 15.94 s/job               
2025-07-31 16:42:51: Sobol, failed: 4 ('VAL_ACC: <FLOAT>' not found), completed 1∑1 (0%/20), eval #1/1 start                                  
2025-07-31 16:42:55: Sobol, failed: 4 ('VAL_ACC: <FLOAT>' not found), completed 1∑1 (0%/20), starting new job                                 
2025-07-31 16:43:02: Sobol, failed: 4 ('VAL_ACC: <FLOAT>' not found), completed/unknown 1/1∑2 (5%/20), started new job                        
2025-07-31 16:43:08: Sobol, failed: 4 ('VAL_ACC: <FLOAT>' not found), completed/pending 1/1∑2 (5%/20), job_failed                             
2025-07-31 16:43:22: Sobol, failed: 5 ('VAL_ACC: <FLOAT>' not found), running 1∑1 (5%/20), finishing jobs (_get_next_trials), finished 1 job  
2025-07-31 16:43:28: Sobol, failed: 5 ('VAL_ACC: <FLOAT>' not found), running 1∑1 (5%/20), getting new HP set

Arguments Overview

Key	Value
config_yaml	None
config_toml	None
config_json	None
num_random_steps	20
max_eval	500
run_program	[['cHl0aG9uMyAudGVzdHMvbW5pc3QvdHJhaW4gLS1lcG9jaHMgJWVwb2NocyAtLWxlYXJuaW5nX3JhdGUgJWxyIC0tYmF0Y2hfc2l6ZSAlYmF0Y2hfc2l6ZSAtLWhpZGRlbl9zaXplICVoaWRkZW5f…
experiment_name	mnist_gpu_noall
mem_gb	10
parameter	[['epochs', 'range', '10', '200', 'int', 'false'], ['lr', 'range', '0.00001', '0.1', 'float', 'false'], ['batch_size', 'range', '8', '2048', 'int',
	'false'], ['hidden_size', 'range', '8', '2048', 'int', 'false'], ['dropout', 'range', '0', '0.5', 'float', 'false'], ['activation', 'fixed',
	'leaky_relu'], ['num_dense_layers', 'range', '1', '4', 'int', 'false'], ['init', 'fixed', 'normal'], ['weight_decay', 'range', '0', '1', 'float',
	'false']]
continue_previous_job	None
experiment_constraints	None
run_dir	runs
seed	None
verbose_tqdm	False
model	BOTORCH_MODULAR
gridsearch	False
occ	False
show_sixel_scatter	False
show_sixel_general	False
show_sixel_trial_index_result	False
follow	True
send_anonymized_usage_stats	True
ui_url	aHR0cHM6Ly9pbWFnZXNlZy5zY2Fkcy5kZS9vbW5pYXgvZ3VpP3BhcnRpdGlvbj1hbHBoYSZleHBlcmltZW50X25hbWU9bW5pc3RfZ3B1X25vYWxsJnJlc2VydmF0aW9uPSZhY2NvdW50PSZtZW1fZ2I…
root_venv_dir	/home/pwinkler
exclude	None
main_process_gb	8
max_nr_of_zero_results	50
abbreviate_job_names	False
orchestrator_file	None
checkout_to_latest_tested_version	False
live_share	True
disable_tqdm	False
disable_previous_job_constraint	False
workdir
occ_type	euclid
result_names	['VAL_ACC=max']
minkowski_p	2
signed_weighted_euclidean_weights
generation_strategy	None
generate_all_jobs_at_once	False
revert_to_random_when_seemingly_exhausted	True
load_data_from_existing_jobs	[]
n_estimators_randomforest	100
max_attempts_for_generation	20
external_generator	None
username	None
max_failed_jobs	0
num_cpus_main_job	None
calculate_pareto_front_of_job	[]
show_generate_time_table	False
force_choice_for_ranges	False
max_abandoned_retrial	20
share_password	None
dryrun	False
db_url	None
run_program_once	None
dont_warm_start_refitting	False
refit_on_cv	False
fit_out_of_design	False
fit_abandoned	False
dont_jit_compile	False
num_restarts	20
raw_samples	1024
max_num_of_parallel_sruns	16
no_transform_inputs	False
no_normalize_y	False
transforms	[]
num_parallel_jobs	20
worker_timeout	120
slurm_use_srun	False
time	2880
partition	alpha
reservation	None
force_local_execution	False
slurm_signal_delay_s	0
nodes_per_job	1
cpus_per_task	1
account	None
gpus	1
run_mode	local
verbose	False
verbose_break_run_search_table	False
debug	False
flame_graph	False
no_sleep	False
tests	False
show_worker_percentage_table_at_end	False
auto_exclude_defective_hosts	False
run_tests_that_fail_on_taurus	False
raise_in_eval	False
show_ram_every_n_seconds	0
show_generation_and_submission_sixel	False
just_return_defaults	False
prettyprint	False

1753972629.7125926,20,0,0
1753972633.7014668,20,0,0
1753972633.821031,20,0,0

timestamp,ram_usage_mb,cpu_usage_percent
1753972624,710.2265625,12.8
1753972629,710.7265625,12.0
1753972633,710.7265625,11.8
1753972633,710.7265625,8.3
1753972633,710.7265625,14.8
1753972633,710.7265625,7.6
1753972633,710.7265625,16.7

submitit INFO (2025-07-31 16:38:11,848) - Starting with JobEnvironment(job_id=531140, hostname=c150, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2025-07-31 16:38:11,848) - Loading pickle: /data/cat/ws/pwinkler-mnist_tst/omniopt/runs/mnist_gpu_noall/1/single_runs/531140/531140_submitted.pkl
  File "/data/cat/ws/pwinkler-mnist_tst/omniopt/.tests/mnist/train", line 3
    function run_nvidia_smi {
             ^^^^^^^^^^^^^^
SyntaxError: invalid syntax

Parameters: {"epochs": 10, "lr": 0.06942201295375824, "batch_size": 318, "hidden_size": 1494, "dropout": 0.06775018572807312, "num_dense_layers": 1, "weight_decay": 0.611398458480835, "activation": "leaky_relu", "init": "normal"}
Debug-Infos: 
========
DEBUG INFOS START:
Program-Code: python3 .tests/mnist/train --epochs 10 --learning_rate 0.06942201295375824421 --batch_size 318 --hidden_size 1494 --dropout 0.06775018572807312012 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0.61139845848083496094
pwd: /data/cat/ws/pwinkler-mnist_tst/omniopt
File: .tests/mnist/train
UID: 2054851
GID: 200270
SLURM_JOB_ID: 531140
Status-Change-Time: 1753972544.0972786
Size: 203 Bytes
Permissions: -rwxr-xr-x
Owner: pwinkler
Last access: 1753972696.8644495
Last modification: 1753972526.2362525
Hostname: c150
========
DEBUG INFOS END

python3 .tests/mnist/train --epochs 10 --learning_rate 0.06942201295375824421 --batch_size 318 --hidden_size 1494 --dropout 0.06775018572807312012 --activation leaky_relu --num_dense_layers 1 --init normal --weight_decay 0.61139845848083496094
stdout was empty
stderr:
   File "/data/cat/ws/pwinkler-mnist_tst/omniopt/.tests/mnist/train", line 3
    function run_nvidia_smi {
             ^^^^^^^^^^^^^^
SyntaxError: invalid syntax

Result: {'VAL_ACC': None}
Final-results: {'VAL_ACC': None}
EXIT_CODE: 1
submitit INFO (2025-07-31 16:38:23,019) - Job completed successfully
submitit INFO (2025-07-31 16:38:23,022) - Exiting after successful completion