To cancel, press CTRL c, then run 'scancel 1822647'
⠋ Importing logging...
⠋ Importing warnings...
⠋ Importing argparse...
⠋ Importing datetime...
⠋ Importing dataclass...
⠋ Importing socket...
⠋ Importing stat...
⠋ Importing pwd...
⠋ Importing base64...
⠙ Importing json...
⠇ Importing yaml...
⠇ Importing toml...
⠹ Importing csv...
⠋ Importing ast...
⠋ Importing rich.table...
⠋ Importing rich print...
⠋ Importing rich.pretty...
⠙ Importing rich.prompt...
⠋ Importing types.FunctionType...
⠋ Importing typing...
⠹ Importing ThreadPoolExecutor...
⠹ Importing submitit.LocalExecutor...
⠋ Importing submitit.Job...
⠼ Importing importlib.util...
⠋ Importing platform...
⠋ Importing inspect frame info...
⠋ Importing pathlib.Path...
⠋ Importing uuid...
⠸ Importing cowsay...
⠋ Importing shutil...
⠋ Importing itertools.combinations...
⠋ Importing os.listdir...
⠋ Importing os.path...
⠙ Importing PIL.Image...
⠴ Importing sixel...
⠋ Importing subprocess...
⠦ Importing tqdm...
⠋ Importing beartype...
⠋ Importing statistics...
⠼ Trying to import pyfiglet...
⠼ Importing helpers...
⠋ Parsing arguments...
⠸ Importing torch...
⠋ Importing numpy...
[WARNING 08-15 02:20:52] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
⠏ Importing ax...
⠋ Importing ax.core.generator_run...
⠋ Importing Cont_X_trans and Y_trans from ax.adapter.registry...
⠋ Importing ax.core.arm...
⠋ Importing ax.core.objective...
⠋ Importing ax.core.Metric...
⠋ Importing ax.exceptions.core...
⠋ Importing ax.exceptions.generation_strategy...
⠋ Importing CORE_DECODER_REGISTRY...
⠋ Trying ax.generation_strategy.generation_node...
⠋ Importing GenerationStep, GenerationStrategy from generation_strategy...
⠋ Importing GenerationNode from generation_node...
⠋ Importing ExternalGenerationNode...
⠋ Importing MaxTrials...
⠋ Importing GeneratorSpec...
⠋ Importing Models from ax.generation_strategy.registry...
⠋ Importing get_pending_observation_features...
⠋ Importing load_experiment...
⠋ Importing save_experiment...
⠋ Importing save_experiment_to_db...
⠋ Importing TrialStatus...
⠋ Importing Data...
⠋ Importing Experiment...
⠋ Importing parameter types...
⠋ Importing TParameterization...
⠋ Importing pandas...
⠏ Importing AxClient and ObjectiveProperties...
⠋ Importing RandomForestRegressor...
⠋ Importing botorch...
⠋ Importing submitit...
⠋ Importing ax logger...
⠋ Importing SQL-Storage-Stuff...
Run-UUID: 7cbacd0d-277b-4d4d-9437-af024620efe2
_________________________________________________
/ \
| OmniOpt2 - Optimizing like it's the Matrix, but I |
| am the One. |
\ /
=================================================
\
\
\
\
. .
/ `. .' \
.---. < > < > .---.
| \ \ - ~ ~ - / / |
_____ ~-..-~ ~-..-~
| | \~~~\.' `./~~~/
--------- \__/ \__/
.' O \ / / \ "
(_____, `._.' | } \/~~~/
`----. / } | / \__/
`-. | / | / `. ,~~|
~-.__| /_ - ~ ^| /- _ `..-' f: f:
| / | / ~-. `-. _|| _||_
|_____| |_____| ~ - . _ _ _ _ __>
⠋ Writing worker creation log...
omniopt --partition=alpha --experiment_name=mnist_gpu --mem_gb=40 --time=2880 --worker_timeout=120 --max_eval=500 --num_parallel_jobs=20 --gpus=1 --num_random_steps=20 --live_share --send_anonymized_usage_stats --result_names VAL_ACC=max --run_program=cHl0aG9uMyAudGVzdHMvbW5pc3QvdHJhaW4gLS1lcG9jaHMgJWVwb2NocyAtLWxlYXJuaW5nX3JhdGUgJWxyIC0tYmF0Y2hfc2l6ZSAlYmF0Y2hfc2l6ZSAtLWhpZGRlbl9zaXplICVoaWRkZW5fc2l6ZSAtLWRyb3BvdXQgJWRyb3BvdXQgLS1hY3RpdmF0aW9uICVhY3RpdmF0aW9uIC0tbnVtX2RlbnNlX2xheWVycyAlbnVtX2RlbnNlX2xheWVycyAtLWluaXQgJWluaXQgLS13ZWlnaHRfZGVjYXkgJXdlaWdodF9kZWNheQ== --cpus_per_task=1 --nodes_per_job=1 --revert_to_random_when_seemingly_exhausted --model=BOTORCH_MODULAR --n_estimators_randomforest=100 --run_mode=local --occ_type=euclid --main_process_gb=8 --max_nr_of_zero_results=50 --slurm_signal_delay_s=0 --max_failed_jobs=0 --dependency=omniopt_singleton --max_attempts_for_generation=20 --num_restarts=20 --raw_samples=1024 --max_abandoned_retrial=20 --max_num_of_parallel_sruns=16 --number_of_generators=1 --generate_all_jobs_at_once --parameter epochs range 10 30 int false --parameter lr range 0.000001 0.001 float true --parameter batch_size range 8 1024 int false --parameter hidden_size range 8 4096 int false --parameter dropout range 0 0.5 float false --parameter activation fixed leaky_relu --parameter num_dense_layers range 1 4 int false --parameter init fixed normal --parameter weight_decay fixed 0 --ui_url aHR0cHM6Ly9pbWFnZXNlZy5zY2Fkcy5kZS9vbW5pYXgvZ3VpP3BhcnRpdGlvbj1hbHBoYSZleHBlcmltZW50X25hbWU9bW5pc3RfZ3B1JnJlc2VydmF0aW9uPSZhY2NvdW50PSZtZW1fZ2I9NDAmdGltZT0yODgwJndvcmtlcl90aW1lb3V0PTEyMCZtYXhfZXZhbD01MDAmbnVtX3BhcmFsbGVsX2pvYnM9MjAmZ3B1cz0xJm51bV9yYW5kb21fc3RlcHM9MjAmZm9sbG93PTAmbGl2ZV9zaGFyZT0xJnNlbmRfYW5vbnltaXplZF91c2FnZV9zdGF0cz0xJmNvbnN0cmFpbnRzPSZyZXN1bHRfbmFtZXM9VkFMX0FDQyUzRG1heCZydW5fcHJvZ3JhbT1weXRob24zJTIwLnRlc3RzJTJGbW5pc3QlMkZ0cmFpbiUyMC0tZXBvY2hzJTIwJTI1ZXBvY2hzJTIwLS1sZWFybmluZ19yYXRlJTIwJTI1bHIlMjAtLWJhdGNoX3NpemUlMjAlMjViYXRjaF9zaXplJTIwLS1oaWRkZW5fc2l6ZSUyMCUyNWhpZGRlbl9zaXplJTIwLS1kcm9wb3V0JTIwJTI1ZHJvcG91dCUyMC0tYWN0aXZhdGlvbiUyMCUyNWFjdGl2YXRpb24lMjAtLW51bV9kZW5zZV9sYXllcnMlMjAlMjVudW1fZGVuc2VfbGF5ZXJzJTIwLS1pbml0JTIwJTI1aW5pdCUyMC0td2VpZ2h0X2RlY2F5JTIwJTI1d2VpZ2h0X2RlY2F5JnJ1bl9wcm9ncmFtX29uY2U9JmNwdXNfcGVyX3Rhc2s9MSZub2Rlc19wZXJfam9iPTEmc2VlZD0mZHJ5cnVuPTAmZGVidWc9MCZyZXZlcnRfdG9fcmFuZG9tX3doZW5fc2VlbWluZ2x5X2V4aGF1c3RlZD0xJmdyaWRzZWFyY2g9MCZtb2RlbD1CT1RPUkNIX01PRFVMQVImZXh0ZXJuYWxfZ2VuZXJhdG9yPSZuX2VzdGltYXRvcnNfcmFuZG9tZm9yZXN0PTEwMCZpbnN0YWxsYXRpb25fbWV0aG9kPWNsb25lJnJ1bl9tb2RlPWxvY2FsJmRpc2FibGVfdHFkbT0wJnZlcmJvc2VfdHFkbT0wJmZvcmNlX2xvY2FsX2V4ZWN1dGlvbj0wJmF1dG9fZXhjbHVkZV9kZWZlY3RpdmVfaG9zdHM9MCZzaG93X3NpeGVsX2dlbmVyYWw9MCZzaG93X3NpeGVsX3RyaWFsX2luZGV4X3Jlc3VsdD0wJnNob3dfc2l4ZWxfc2NhdHRlcj0wJnNob3dfd29ya2VyX3BlcmNlbnRhZ2VfdGFibGVfYXRfZW5kPTAmb2NjPTAmb2NjX3R5cGU9ZXVjbGlkJm5vX3NsZWVwPTAmc2x1cm1fdXNlX3NydW49MCZ2ZXJib3NlX2JyZWFrX3J1bl9zZWFyY2hfdGFibGU9MCZhYmJyZXZpYXRlX2pvYl9uYW1lcz0wJm1haW5fcHJvY2Vzc19nYj04Jm1heF9ucl9vZl96ZXJvX3Jlc3VsdHM9NTAmc2x1cm1fc2lnbmFsX2RlbGF5X3M9MCZtYXhfZmFpbGVkX2pvYnM9MCZleGNsdWRlPSZ1c2VybmFtZT0mZ2VuZXJhdGlvbl9zdHJhdGVneT0mcm9vdF92ZW52X2Rpcj0mZGVwZW5kZW5jeT1vbW5pb3B0X3NpbmdsZXRvbiZ3b3JrZGlyPSZkb250X2ppdF9jb21waWxlPTAmZml0X291dF9vZl9kZXNpZ249MCZyZWZpdF9vbl9jdj0wJnNob3dfZ2VuZXJhdGVfdGltZV90YWJsZT0wJmRvbnRfd2FybV9zdGFydF9yZWZpdHRpbmc9MCZtYXhfYXR0ZW1wdHNfZm9yX2dlbmVyYXRpb249MjAmbnVtX3Jlc3RhcnRzPTIwJnJhd19zYW1wbGVzPTEwMjQmbWF4X2FiYW5kb25lZF9yZXRyaWFsPTIwJm1heF9udW1fb2ZfcGFyYWxsZWxfc3J1bnM9MTYmbnVtYmVyX29mX2dlbmVyYXRvcnM9MSZmb3JjZV9jaG9pY2VfZm9yX3Jhbmdlcz0wJm5vX3RyYW5zZm9ybV9pbnB1dHM9MCZmaXRfYWJhbmRvbmVkPTAmbm9fbm9ybWFsaXplX3k9MCZ2ZXJib3NlPTAmZ2VuZXJhdGVfYWxsX2pvYnNfYXRfb25jZT0xJnNhdmVfdG9fZGF0YWJhc2U9MCZmbGFtZV9ncmFwaD0wJmNoZWNrb3V0X3RvX2xhdGVzdF90ZXN0ZWRfdmVyc2lvbj0wJnBhcmFtZXRlcl8wX25hbWU9ZXBvY2hzJnBhcmFtZXRlcl8wX3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzBfbWluPTEwJnBhcmFtZXRlcl8wX21heD0zMCZwYXJhbWV0ZXJfMF9udW1iZXJfdHlwZT1pbnQmcGFyYW1ldGVyXzBfbG9nX3NjYWxlPWZhbHNlJnBhcmFtZXRlcl8xX25hbWU9bHImcGFyYW1ldGVyXzFfdHlwZT1yYW5nZSZwYXJhbWV0ZXJfMV9taW49MC4wMDAwMDEmcGFyYW1ldGVyXzFfbWF4PTAuMDAxJnBhcmFtZXRlcl8xX251bWJlcl90eXBlPWZsb2F0JnBhcmFtZXRlcl8xX2xvZ19zY2FsZT10cnVlJnBhcmFtZXRlcl8yX25hbWU9YmF0Y2hfc2l6ZSZwYXJhbWV0ZXJfMl90eXBlPXJhbmdlJnBhcmFtZXRlcl8yX21pbj04JnBhcmFtZXRlcl8yX21heD0xMDI0JnBhcmFtZXRlcl8yX251bWJlcl90eXBlPWludCZwYXJhbWV0ZXJfMl9sb2dfc2NhbGU9ZmFsc2UmcGFyYW1ldGVyXzNfbmFtZT1oaWRkZW5fc2l6ZSZwYXJhbWV0ZXJfM190eXBlPXJhbmdlJnBhcmFtZXRlcl8zX21pbj04JnBhcmFtZXRlcl8zX21heD00MDk2JnBhcmFtZXRlcl8zX251bWJlcl90eXBlPWludCZwYXJhbWV0ZXJfM19sb2dfc2NhbGU9ZmFsc2UmcGFyYW1ldGVyXzRfbmFtZT1kcm9wb3V0JnBhcmFtZXRlcl80X3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzRfbWluPTAmcGFyYW1ldGVyXzRfbWF4PTAuNSZwYXJhbWV0ZXJfNF9udW1iZXJfdHlwZT1mbG9hdCZwYXJhbWV0ZXJfNF9sb2dfc2NhbGU9ZmFsc2UmcGFyYW1ldGVyXzVfbmFtZT1hY3RpdmF0aW9uJnBhcmFtZXRlcl81X3R5cGU9Zml4ZWQmcGFyYW1ldGVyXzVfdmFsdWU9bGVha3lfcmVsdSZwYXJhbWV0ZXJfNl9uYW1lPW51bV9kZW5zZV9sYXllcnMmcGFyYW1ldGVyXzZfdHlwZT1yYW5nZSZwYXJhbWV0ZXJfNl9taW49MSZwYXJhbWV0ZXJfNl9tYXg9NCZwYXJhbWV0ZXJfNl9udW1iZXJfdHlwZT1pbnQmcGFyYW1ldGVyXzZfbG9nX3NjYWxlPWZhbHNlJnBhcmFtZXRlcl83X25hbWU9aW5pdCZwYXJhbWV0ZXJfN190eXBlPWZpeGVkJnBhcmFtZXRlcl83X3ZhbHVlPW5vcm1hbCZwYXJhbWV0ZXJfOF9uYW1lPXdlaWdodF9kZWNheSZwYXJhbWV0ZXJfOF90eXBlPWZpeGVkJnBhcmFtZXRlcl84X3ZhbHVlPTAmcGFydGl0aW9uPWFscGhhJm51bV9wYXJhbWV0ZXJzPTk=
⠋ Disabling logging...
⠋ Setting run folder...
⠋ Creating folder /data/horse/ws/s3811141-omniopt_mnist_test_call/omniopt/runs/mnist_gpu/0...
⠋ Writing revert_to_random_when_seemingly_exhausted file ...
⠋ Writing username state file...
⠋ Writing result names file...
⠋ Writing result min/max file...
⠋ Saving state files...
Run-folder: /data/horse/ws/s3811141-omniopt_mnist_test_call/omniopt/runs/mnist_gpu/0
⠋ Writing live_share file if it is present...
⠋ Writing job_start_time file...
⠼ Writing git information
⠋ Checking max_eval...
⠋ Calculating number of steps...
⠋ Adding excluded nodes...
⠋ Setting global generation strategy
⠋ Initializing ax_client...
⠋ Setting orchestrator...
⠧ Initializing live share...