To cancel, press CTRL c, then run 'scancel 1784996'
โ Importing logging...
โ Importing warnings...
โ Importing argparse...
โ Importing datetime...
โ Importing dataclass...
โ Importing hashlib...
โ Importing socket...
โ Importing stat...
โ Importing pwd...
โ Importing signal...
โ Importing base64...
โ Importing json...
โ ง Importing yaml...
โ Importing toml...
โ Importing csv...
โ Importing ast...
โ Importing rich.table...
โ Importing rich print...
โ Importing rich.pretty...
โ Importing rich.prompt...
โ Importing types.FunctionType...
โ Importing typing...
โ Importing ThreadPoolExecutor...
โ ด Importing submitit.LocalExecutor...
โ Importing submitit.Job...
โ Importing importlib.util...
โ Importing inspect...
โ Importing platform...
โ Importing inspect frame info...
โ Importing pathlib.Path...
โ Importing uuid...
โ Importing traceback...
โ Importing cowsay...
โ Importing psutil...
โ Importing shutil...
โ Importing itertools.combinations...
โ Importing os.listdir...
โ Importing os.path...
โ ผ Importing PIL.Image...
โ Importing sixel...
โ Importing subprocess...
โ Importing tqdm...
โ ด Importing beartype...
โ Importing statistics...
โ Trying to import pyfiglet...
โ Importing helpers...
โ Parsing arguments...
โ Importing torch...
โ Importing numpy...
โ Importing collections...
โ น Importing ax...
โ Importing ax.core.generator_run...
โ Importing Cont_X_trans and Y_trans from ax.modelbridge.registry...
โ Importing ax.core.arm...
โ Importing ax.core.objective...
โ Importing ax.core.Metric...
โ Importing ax.exceptions.core...
โ Importing ax.exceptions.generation_strategy...
โ Importing CORE_DECODER_REGISTRY...
โ Trying ax.generation_strategy.generation_node...
โ Importing GenerationStep, GenerationStrategy from generation_strategy...
โ Importing GenerationNode from generation_node...
โ Importing ExternalGenerationNode...
โ Importing MaxTrials...
โ Importing GeneratorSpec...
โ Importing Models from ax.modelbridge.registry...
โ Importing get_pending_observation_features...
โ Importing load_experiment...
โ Importing save_experiment...
โ Importing save_experiment_to_db...
โ Importing TrialStatus...
โ Importing Data...
โ Importing Experiment...
โ Importing parameter types...
โ Importing TParameterization...
โ Importing pandas...
โ Importing AxClient and ObjectiveProperties...
โ Importing RandomForestRegressor...
โ Importing botorch...
โ Importing submitit...
โ Importing ax logger...
โ Importing SQL-Storage-Stuff...
Run-UUID: 1674a734-0919-4158-93a6-c65137e8d987
_________________________________________________
/ \
| OmniOpt2 - Channeling my inner Gandalf: โYou shal |
| l not pass... without fine-tuning!โ |
\ /
=================================================
\
\
\
\
,.
(_|,.
,' /, )_______ _
__j o``-' `.'-)'
(") \'
`-j |
`-._( /
|_\ |--^. /
/_]'|_| /_)_/
/_]' /_]'
โ Writing worker creation log...
omniopt --partition is_ignored_here --experiment_name=slurm_in_docker_test --mem_gb=1 --time=60 --worker_timeout=1 --max_eval=1000 --num_parallel_jobs=20 --run_program ZWNobyAiUkVTVUxUOiAlKGludF9wYXJhbSklKGludF9wYXJhbV90d28pJShmbG9hdF9wYXJhbSki --parameter int_param range -100 10 int --parameter float_param range -100 10 float --parameter int_param_two range -100 10 int --follow --num_random_steps 2000 --model BOTORCH_MODULAR --auto_exclude_defective_hosts --max_nr_of_zero 3 --show_generate_time_table --show_generation_and_submission_sixel --no_sleep --number_of_generators 5 --live_share --gpus=1
โ Disabling logging...
โ Setting run folder...
โ Creating folder /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1...
โ Writing revert_to_random_when_seemingly_exhausted file ...
โ Writing username state file...
โ Writing result names file...
โ Writing result min/max file...
โ Saving state files...
Run-folder: /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1
โ Printing run info...
โ Initializing NVIDIA-Logs...
โ Writing ui_url file if it is present...
โ Writing live_share file if it is present...
โ Writing job_start_time file...
โ Writing git info file...
โ Checking max_eval...
You have less --max_eval 1000 than --num_random_steps 2000. Switched both.
โ ธ Calculating number of steps...
โ Adding excluded nodes...
โ Handling random steps...
โ Initializing ax_client...
[WARNING 08-12 16:48:22] ax.service.ax_client: Selecting a GenerationStrategy when using BatchTrials is in beta. Double check the recommended strategy matches your expectations.
โ Setting orchestrator...
You have 2 CPUs available for the main process. Using CUDA device NVIDIA A100-SXM4-40GB. Generation strategy: SOBOL for 1000 steps.
Run-Program: echo "RESULT: %(int_param)%(int_param_two)%(float_param)"
Experiment parameters
โโโโโโโโโโโโโโโโโณโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโณโโโโโโโโโโโโโ
โ Name โ Type โ Lower bound โ Upper bound โ Type โ Log Scale? โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ int_param โ range โ -100 โ 10 โ int โ No โ
โ float_param โ range โ -100 โ 10 โ float โ No โ
โ int_param_two โ range โ -100 โ 10 โ int โ No โ
โโโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโโโโ
Result-Names
โโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โ Result-Name โ Min or max? โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ RESULT โ min โ
โโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
Started worker 1 via sbatch: Submitted batch job 1785002
Started worker 2 via sbatch: Submitted batch job 1785003
Started worker 3 via sbatch: Submitted batch job 1785004
Started worker 4 via sbatch: Submitted batch job 1785005
Importing rich.table...
โ Importing rich.table...
โ Importing rich print...
โ Importing rich print...
โ Importing rich.pretty...
โ Importing rich.pretty...
โ Importing rich.prompt...
โ Importing rich.prompt...
โ Importing types.FunctionType...
โ Importing types.FunctionType...
โ Importing typing...
โ Importing typing...
โ Importing ThreadPoolExecutor...
โ Importing ThreadPoolExecutor...
โ Importing submitit.LocalExecutor...
โ Importing submitit.LocalExecutor...
โ Importing submitit.Job...
โ Importing submitit.Job...
โ Importing importlib.util...
โ Importing importlib.util...
โ Importing inspect...
โ Importing inspect...
โ Importing platform...
โ Importing platform...
โ Importing inspect frame info...
โ Importing inspect frame info...
โ Importing pathlib.Path...
โ Importing pathlib.Path...
โ Importing uuid...
โ Importing uuid...
โ Importing traceback...
โ Importing traceback...
โ Importing cowsay...
โ Importing cowsay...
โ Importing psutil...
โ Importing psutil...
โ Importing shutil...
โ Importing shutil...
โ Importing itertools.combinations...
โ Importing itertools.combinations...
โ Importing os.listdir...
โ Importing os.listdir...
โ Importing os.path...
โ Importing os.path...
โ Importing PIL.Image...
โ Importing PIL.Image...
โ Importing sixel...
โ Importing sixel...
โ Importing subprocess...
โ Importing subprocess...
โ Importing tqdm...
โ Importing tqdm...
โ Importing beartype...
โ Importing beartype...
โ Importing statistics...
โ Importing statistics...
โ Trying to import pyfiglet...
โ Trying to import pyfiglet...
โ ฆ Importing helpers...
โ ฆ Importing helpers...
โ Parsing arguments...
โ Parsing arguments...
โ ผ Importing torch...To cancel, press CTRL c, then run 'scancel 1785003'
โ Importing logging...
โ Importing warnings...
โ Importing argparse...
โ Importing datetime...
โ Importing dataclass...
โ Importing hashlib...
โ Importing socket...
โ Importing stat...
โ Importing pwd...
โ Importing signal...
โ Importing base64...
โ Importing json...
โ Importing yaml...
โ Importing toml...
โ Importing csv...
โ Importing ast...
โ Importing rich.table...
โ Importing rich print...
โ Importing rich.pretty...
โ Importing rich.prompt...
โ Importing types.FunctionType...
โ Importing typing...
โ Importing ThreadPoolExecutor...
โ Importing submitit.LocalExecutor...
โ Importing submitit.Job...
โ Importing importlib.util...
โ Importing inspect...
โ Importing platform...
โ Importing inspect frame info...
โ Importing pathlib.Path...
โ Importing uuid...
โ Importing traceback...
โ Importing cowsay...
โ Importing psutil...
โ Importing shutil...
โ Importing itertools.combinations...
โ Importing os.listdir...
โ Importing os.path...
โ Importing PIL.Image...
โ Importing sixel...
โ Importing subprocess...
โ Importing tqdm...
โ ImpoiOpt2 - Tuning hyperparameters with the finess |
| e of a cat trying not to knock over the vase. |
\ /
=================================================
\
\
\
\
. _ .
|\_|/__/|
/ / \/ \ \
/__|O||O|__ \
|/_ \_/\_/ _\ |
| | (____) | ||
\/\___/\__/ //
(_/ ||
| ||
| ||\
\ //_/
\______//
__ || __||
(____(____)
โ Writing worker creation log...
omniopt --partition is_ignored_here --experiment_name=slurm_in_docker_test --mem_gb=1 --time=60 --worker_timeout=1 --max_eval=1000 --num_parallel_jobs=20 --run_program ZWNobyAiUkVTVUxUOiAlKGludF9wYXJhbSklKGludF9wYXJhbV90d28pJShmbG9hdF9wYXJhbSki --parameter int_param range -100 10 int --parameter float_param range -100 10 float --parameter int_param_two range -100 10 int --follow --num_random_steps 2000 --model BOTORCH_MODULAR --auto_exclude_defective_hosts --max_nr_of_zero 3 --show_generate_time_table --show_generation_and_submission_sixel --no_sleep --number_of_generators 5 --live_share --gpus=1 --worker_generator_path /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1
โ Disabling logging...
set_run_folder: Using /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1 as worker-generation path, will append additional worker to it
โ Setting run folder...
โ Creating folder /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1...
โ Writing revert_to_random_when_seemingly_exhausted file ...
โ Writing username state file...
โ Writing result names file...
โ Writing result min/max file...
โ Saving state files...
Run-folder: /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1
โ Printing run info...
โ Initializing NVIDIA-Logs...
โ Writing ui_url file if it is present...
โ Writing live_share file if it is present...
โ Writing job_start_time file...
โ น Importing torch...
โ Importing numpy...
โ Importing collections...
โ ผ Importing ax...
โ Importing ax.core.generator_run...
โ ผ Importing ax...
โ Importing Cont_X_trans and Y_trans from ax.modelbridge.registry...
โ Importing ax.core.arm...
โ Importing ax.core.generator_run...
โ Importing Cont_X_trans and Y_trans from ax.modelbridge.registry...
โ Importing ax.core.objective...
โ Importing ax.core.arm...
โ Importing ax.core.Metric...
โ Importing ax.core.objective...
โ Importing ax.exceptions.core...
โ Importing ax.core.Metric...
โ Importing ax.exceptions.generation_strategy...
โ Importing ax.exceptions.core...
โ Importing CORE_DECODER_REGISTRY...
โ Importing ax.exceptions.generation_strategy...
โ Trying ax.generation_strategy.generation_node...
โ Importing CORE_DECODER_REGISTRY...
โ Importing GenerationStep, GenerationStrategy from generation_strategy...
โ Trying ax.generation_strategy.generation_node...
โ Importing GenerationNode from generation_node...
โ Importing GenerationStep, GenerationStrategy from generation_strategy...
โ Importing ExternalGenerationNode...
โ Importing GenerationNode from generation_node...
โ Importing MaxTrials...
โ Importing ExternalGenerationNode...
โ Importing GeneratorSpec...
โ Importing MaxTrials...
โ Importing Models from ax.modelbridge.registry...
โ Importing GeneratorSpec...
โ Importing get_pending_observation_features...
โ Importing Models from ax.modelbridge.registry...
โ Importing load_experiment...
โ Importing save_experiment...
โ Importing get_pending_observation_features...
โ Importing save_experiment_to_db...
โ Importing load_experiment...
โ Importing TrialStatus...
โ Importing save_experiment...
โ Importing Data...
โ Importing save_experiment_to_db...
โ Importing Experiment...
โ Importing TrialStatus...
โ Importing parameter types...
โ Importing Data...
โ Importing TParameterization...
โ Importing Experiment...
โ Importing pandas...
โ Importing parameter types...
โ Importing TParameterization...
โ Importing pandas...
โ ผ Importing ax...โ Importing AxClient and ObjectiveProperties...
โ Importing RandomForestRegressor...
โ Importing AxClient and ObjectiveProperties...
โ Importing botorch...
โ Importing RandomForestRegressor...
โ Importing submitit...
โ Importing ax logger...
โ Importing botorch...
โ Importing SQL-Storage-Stuff...
โ Importing submitit...
โ Importing ax logger...
โ Importing SQL-Storage-Stuff...
โ ง Importing ax...
โ Importing ax.core.generator_run...
โ Importing Cont_X_trans and Y_trans from ax.modelbridge.registry...
โ Importing ax.core.arm...
โ Importing ax.core.objective...
โ Importing ax.core.Metric...
โ Importing ax.exceptions.core...
โ Importing ax.exceptions.generation_strategy...
โ Importing CORE_DECODER_REGISTRY...
โ Trying ax.generation_strategy.generation_node...
โ Importing GenerationStep, GenerationStrategy from generation_strategy...
โ Importing GenerationNode from generation_node...
โ Importing ExternalGenerationNode...
โ Importing MaxTrials...
โ Importing GeneratorSpec...
โ Importing Models from ax.modelbridge.registry...
โ Importing get_pending_observation_features...
โ Importing load_experiment...
โ Importing save_experiment...
โ Importing save_experiment_to_db...
โ Importing TrialStatus...
โ Importing Data...
โ Importing Experiment...
โ Importing parameter types...
โ Importing TParameterization...
โ Importing pandas...
โ Importing AxClient and ObjectiveProperties...
โ Importing RandomForestRegressor...
โ Importing botorch...
โ Importing submitit...
โ Importing ax logger...
โ Importing SQL-Storage-Stuff...
Run-UUID: 1674a734-0919-4158-93a6-c65137e8d987
Run-UUID: 1674a734-0919-4158-93a6-c65137e8d987
_____ _ _____ _ _____
| _ | (_) _ | | | / __ \
| | | |_ __ ___ _ __ _| | | |_ __ | |_`' / /'
| | | | '_ ` _ \| '_ \| | | | | '_ \| __| / /
\ \_/ / | | | | | | | | \ \_/ / |_) | |_./ /___
\___/|_| |_| |_|_| |_|_|\___/| .__/ \__\_____/
| |
|_|
?25l _____ _ _____ ______
/ ___ \ (_)/ ___ \ _ (_____ \
| | | |____ ____ _| | | |____ | |_ ____) )
| | | | \| _ \| | | | | _ \| _) /_____/
| |___| | | | | | | | | |___| | | | | |__ _______
\_____/|_|_|_|_| |_|_|\_____/| ||_/ \___|_______)
|_|
โ Writing worker creation log...
โ Writing worker creation log...
omniopt --partition is_ignored_here --experiment_name=slurm_in_docker_test --mem_gb=1 --time=60 --worker_timeout=1 --max_eval=1000 --num_parallel_jobs=20 --run_program ZWNobyAiUkVTVUxUOiAlKGludF9wYXJhbSklKGludF9wYXJhbV90d28pJShmbG9hdF9wYXJhbSki --parameter int_param range -100 10 int --parameter float_param range -100 10 float --parameter int_param_two range -100 10 int --follow --num_random_steps 2000 --model BOTORCH_MODULAR --auto_exclude_defective_hosts --max_nr_of_zero 3 --show_generate_time_table --show_generation_and_submission_sixel --no_sleep --number_of_generators 5 --live_share --gpus=1 --worker_generator_path /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1
niopt --partition is_ignored_here --experiment_name=slurm_in_docker_test --mem_gb=1 --time=60 --worker_timeout=1 --max_eval=1000 --num_parallel_jobs=20 --run_program ZWNobyAiUkVTVUxUOiAlKGludF9wYXJhbSklKGludF9wYXJhbV90d28pJShmbG9hdF9wYXJhbSki --parameter int_param range -100 10 int --parameter float_param range -100 10 float --parameter int_param_two range -100 10 int --follow --num_random_steps 2000 --model BOTORCH_MODULAR --auto_exclude_defective_hosts --max_nr_of_zero 3 --show_generate_time_table --show_generation_and_submission_sixel --no_sleep --number_of_generators 5 --live_share --gpus=1 --worker_generator_path /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1
โ Disabling logging...
set_run_folder: Using /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1 as worker-generation path, will append additional worker to it
โ Setting run folder...โ Disabling logging...
set_run_folder: Using /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1 as worker-generation path, will append additional worker to it
โ Setting run folder...
?25lRun-UUID: 1674a734-0919-4158-93a6-c65137e8d987
_________________________________________________
/ \
| OmniOpt2 - This is not madness... this is hyperpa |
| rameter tuning! |
\ /
=================================================
\
\
\
.--.
|o_o |
|:_/ |
// \ \
(| | )
/'\_ _/`\
\___)=(___/
โ Writing worker creation log...
โ Setting run folder...
โ Writing git info file...
โ Checking max_eval...
You have less --max_eval 1000 than --num_random_steps 2000. Switched both.
โ Calculating number of steps...
โ Adding excluded nodes...
โ Handling random steps...
โ Initializing ax_client...
Checkpoint file found after 0 seconds
2mWriting job_start_time file...
โ ผ Creating folder /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1...
โ Writing revert_to_random_when_seemingly_exhausted file ...
โ Writing username state file...
โ Writing result names file...
โ Writing result min/max file...
โ ด Writing git info file...โ Saving state files...
Run-folder: /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/slurm_in_docker_test/1
โ Printing run info...
โ Initializing NVIDIA-Logs...
โ Writing ui_url file if it is present...
โ Writing live_share file if it is present...
โ Writing job_start_time file...
โ ง Writing git info file...
โ Checking max_eval...
You have less --max_eval 1000 than --num_random_steps 2000. Switched both.
โ ผ Writing git info file...
โ Checking max_eval...
You have less --max_eval 1000 than --num_random_steps 2000. Switched both.
โ ผ Writing git info file...
โ Checking max_eval...
You have less --max_eval 1000 than --num_random_steps 2000. Switched both.
โ Calculating number of steps...
โ Calculating number of steps...
โ Calculating number of steps...
โ Adding excluded nodes...
โ Handling random steps...
โ Adding excluded nodes...
โ Handling random steps...
โ Adding excluded nodes...
โ Handling random steps...
โ Initializing ax_client...
Checkpoint file found after 0 seconds
โ Initializing ax_client...
Checkpoint file found after 0 seconds
โ Initializing ax_client...
Checkpoint file found after 0 seconds
timestamp,ram_usage_mb,cpu_usage_percent
1755010109,678.4921875,9.5
1755010205,678.73046875,9.2