OmniOpt2

Experiment overview

Setting	Value
Model for non-random steps	BOTORCH_MODULAR
Max. nr. evaluations	1000
Number random steps	20
Nr. of workers (parameter)	20
Main process memory (GB)	8
Worker memory (GB)	32

Experiment parameters

Name	Type	Lower bound	Upper bound	Values	Type	Log Scale?
batch_size	range	16	512		int	No
ffn	range	2	512		int	No
max_turnstiles	range	5	64		int	No
epochs	range	10	500		int	No
n_layers	range	1	20		int	No
d_model	range	2	8		int	No
max_sequence_length	range	5	64		int	No
val_batches	fixed			128
lr	range	1e-07	0.1		float	No
n_heads	range	1	128		int	No
batches_per_epoch	range	1	20		int	No

Result names and types

name	min/max
VAL_LOSS	min
PARAMS	min

Git-Version

Commit: 997d0a84c9f3676ab83077d5b431c1fe7626cc5c (9290)

To cancel, press CTRL c, then run 'scancel 3493595'
⠋ Importing logging...
⠋ Importing warnings...
⠋ Importing argparse...
⠋ Importing datetime...
⠋ Importing dataclass...
⠋ Importing socket...
⠋ Importing stat...
⠋ Importing pwd...
⠋ Importing base64...
⠋ Importing json...
⠋ Importing yaml...
⠋ Importing toml...
⠋ Importing csv...
⠋ Importing ast...
⠋ Importing rich.table...
⠋ Importing rich print...
⠋ Importing rich.pretty...
⠋ Importing pformat...
⠋ Importing rich.prompt...
⠋ Importing types.FunctionType...
⠋ Importing typing...
⠋ Importing ThreadPoolExecutor...
⠋ Importing submitit.LocalExecutor...
⠋ Importing submitit.Job...
⠋ Importing importlib.util...
⠋ Importing platform...
⠋ Importing inspect frame info...
⠋ Importing pathlib.Path...
⠋ Importing uuid...
⠋ Importing cowsay...
⠋ Importing shutil...
⠋ Importing itertools.combinations...
⠋ Importing os.listdir...
⠋ Importing os.path...
⠋ Importing PIL.Image...
⠋ Importing sixel...
⠋ Importing subprocess...
⠋ Importing tqdm...
⠋ Importing beartype...
[WARNING 05-06 07:41:20] ax.storage.sqa_store.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
⠋ Importing rendering stuff...
⠋ Importing statistics...
⠋ Trying to import pyfiglet...
⠋ Importing helpers...
⠋ Importing pareto...
⠋ Parsing arguments...
⠋ Importing torch...
⠋ Importing numpy...
⠋ Importing ax...
⠋ Importing ax.core.generator_run...
⠋ Importing Cont_X_trans and Y_trans from ax.adapter.registry...
⠋ Importing ax.core.arm...
⠋ Importing ax.core.objective...
⠋ Importing ax.core.Metric...
⠋ Importing ax.exceptions.core...
⠋ Importing ax.exceptions.generation_strategy...
⠋ Importing CORE_DECODER_REGISTRY...
⠋ Trying ax.generation_strategy.generation_node...
⠋ Importing GenerationStep, GenerationStrategy from generation_strategy...
⠋ Importing GenerationNode from generation_node...
⠋ Importing ExternalGenerationNode...
⠋ Importing MinTrials...
⠋ Importing GeneratorSpec...
⠋ Importing Generators from ax.generation_strategy.registry...
⠋ Importing load_experiment...
⠋ Importing save_experiment...
⠋ Importing save_experiment_to_db...
⠋ Importing TrialStatus...
⠋ Importing Data...
⠋ Importing Experiment...
⠋ Importing parameter types...
⠋ Importing TParameterization...
⠋ Importing pandas...
⠋ Importing AxClient and ObjectiveProperties...
⠋ Importing RandomForestRegressor...
⠋ Importing botorch...
⠋ Importing submitit...
⠋ Importing ax logger...
⠋ Importing SQL-Storage-Stuff...
Run-UUID: 4c793f8d-23b6-4c91-8f04-fe45f20f5d4c
  _________________________________________________
 /                                                 \
| OmniOpt2 - The fractal of hyperparameters: The de |
| eper you go, the more you see.                    |
 \                                                 /
  =================================================
                                                        \
                                                         \
                                                          \
                                                           \
                                                              .-=-==--==--.
                                                        ..-=="  ,'o`)      `.
                                                      ,'         `"'         \
                                                     :  (                     `.__...._
                                                     |                  )    /         `-=-.
                                                     :       ,vv.-._   /    /               `---==-._
                                                      \/\/\/VV ^ d88`;'    /                         `.
                                                          ``  ^/d88P!'    /             ,              `._
                                                             ^/    !'   ,.      ,      /                  "-,,__,,--'""""-.
                                                            ^/    !'  ,'  \ . .(      (         _           )  ) ) ) ))_,-.\
                                                           ^(__ ,!',"'   ;:+.:%:a.     \:.. . ,'          )  )  ) ) ,"'    '
                                                           ',,,'','     /o:::":%:%a.    \:.:.:         .    )  ) _,'
                                                            """'       ;':::'' `+%%%a._  \%:%|         ;.). _,-""
                                                                   ,-='_.-'      ``:%::)  )%:|        /:._,"
                                                                  (/(/"           ," ,'_,'%%%:       (_,'
                                                                                 (  (//(`.___;        \
                                                                                  \     \    `         `
                                                                                   `.    `.   `.        :
                                                                                     \. . .\    : . . . :
                                                                                      \. . .:    `.. . .:
                                                                                       `..:.:\     \:...\
                                                                                        ;:.:.;      ::...:
                                                                                        ):%::       :::::;
                                                                                    __,::%:(        :::::
                                                                                 ,;:%%%%%%%:        ;:%::
                                                                                   ;,--""-.`\  ,=--':%:%:\
                                                                                  /"       "| /-".:%%%%%%%\
                                                                                                  ;,-"'`)%%)
                                                                                                 /"      "|
⠋ Writing worker creation log...
omniopt --partition=alpha --experiment_name=tinyLogicLLM2 --mem_gb=32 --time=7200 --worker_timeout=720 --max_eval=1000 --num_parallel_jobs=20 --gpus=1 --num_random_steps=20 --follow --live_share --send_anonymized_usage_stats --result_names 'VAL_LOSS=min PARAMS=min' --run_program='cHl0aG9uMyAvZGF0YS9ob3JzZS93cy9zMzgxMTE0MS1ncm9rMi90cmFpbi5weSAtLWJhdGNoLXNpemU9JShiYXRjaF9zaXplKSAtLXRvcG8gLS10b3BvLWV2ZXJ5IDEgLS1rZWxwLWV2ZXJ5PTEgLS1uLWxheWVycz0lKG5fbGF5ZXJzKSAtLWQtbW9kZWw9JShkX21vZGVsKSAtLW4taGVhZHM9JShuX2hlYWRzKSAtLXRva2VuaXplcl9pbml0aWFsX25yPTEwMDAgLS1scj0lKGxyKSAtLWJhdGNoZXMtcGVyLWVwb2NoPSUoYmF0Y2hlc19wZXJfZXBvY2gpIC0tdmFsLWJhdGNoZXM9JSh2YWxfYmF0Y2hlcykgLS1lcG9jaHM9JShlcG9jaHMpIC0tdGFzaz10dXJuc3RpbGUgLS1tYXgtc2VxLWxlbiAlKG1heF9zZXF1ZW5jZV9sZW5ndGgpIC0tbWF4LXR1cm5zdGlsZXMgJShtYXhfdHVybnN0aWxlcykgLS1zY2hlZHVsZXI9Y29zaW5lIC0tZmZuPSUoZmZuKQo=' --cpus_per_task=1 --nodes_per_job=1 --revert_to_random_when_seemingly_exhausted --model=BOTORCH_MODULAR --n_estimators_randomforest=100 --run_mode=local --show_sixel_general --show_sixel_trial_index_result --show_sixel_scatter --occ_type=euclid --main_process_gb=8 --nr_evals_per_arm=1 --max_nr_of_zero_results=50 --slurm_signal_delay_s=0 --max_failed_jobs=0 --max_attempts_for_generation=20 --num_restarts=20 --raw_samples=1024 --max_abandoned_retrial=20 --max_num_of_parallel_sruns=16 --number_of_generators=1 --generate_all_jobs_at_once --parameter batch_size range 16 512 int false --parameter ffn range 2 512 int false --parameter max_turnstiles range 5 64 int false --parameter epochs range 10 500 int false --parameter n_layers range 1 20 int false --parameter d_model range 2 8 int false --parameter max_sequence_length range 5 64 int false --parameter val_batches fixed 128 --parameter lr range 0.0000001 0.1 float false --parameter n_heads range 1 128 int false --parameter batches_per_epoch range 1 20 int false
⠋ Disabling logging...
⠋ Setting run folder...
⠋ Creating folder /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0...
⠋ Writing revert_to_random_when_seemingly_exhausted file ...
⠋ Writing username state file...
⠋ Writing result names file...
⠋ Writing result min/max file...
⠋ Saving state files...
Run-folder: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0
⠋ Writing live_share file if it is present...
⠋ Writing job_start_time file...
⠙ Writing git information
⠋ Checking max_eval...
⠋ Calculating number of steps...
⠋ Adding excluded nodes...
⠋ Initializing ax_client...
⠋ Setting orchestrator...
See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=tinyLogicLLM2&run_nr=0 for live-results.

You have 1 CPUs available for the main process. Using CUDA device NVIDIA H100.
Generation strategy: SOBOL for 20 steps and then BOTORCH_MODULAR for 980 steps.
Run-Program: python3 /data/horse/ws/s3811141-grok2/train.py --batch-size=%(batch_size) --topo --topo-every 1 --kelp-every=1 --n-layers=%(n_layers) --d-model=%(d_model) --n-heads=%(n_heads) --tokenizer_initial_nr=1000 --lr=%(lr) --batches-per-epoch=%(batches_per_epoch) --val-batches=%(val_batches) --epochs=%(epochs) --task=turnstile --max-seq-len %(max_sequence_length) --max-turnstiles %(max_turnstiles) --scheduler=cosine --ffn=%(ffn)

Experiment parameters
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃ Name                ┃ Type  ┃ Lower bound ┃ Upper bound ┃ Values ┃ Type  ┃ Log Scale? ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│ batch_size          │ range │ 16          │ 512         │        │ int   │ No         │
│ ffn                 │ range │ 2           │ 512         │        │ int   │ No         │
│ max_turnstiles      │ range │ 5           │ 64          │        │ int   │ No         │
│ epochs              │ range │ 10          │ 500         │        │ int   │ No         │
│ n_layers            │ range │ 1           │ 20          │        │ int   │ No         │
│ d_model             │ range │ 2           │ 8           │        │ int   │ No         │
│ max_sequence_length │ range │ 5           │ 64          │        │ int   │ No         │
│ val_batches         │ fixed │             │             │ 128    │       │            │
│ lr                  │ range │ 1e-07       │ 0.1         │        │ float │ No         │
│ n_heads             │ range │ 1           │ 128         │        │ int   │ No         │
│ batches_per_epoch   │ range │ 1           │ 20          │        │ int   │ No         │
└─────────────────────┴───────┴─────────────┴─────────────┴────────┴───────┴────────────┘
Result-Names
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Result-Name ┃ Min or max? ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ VAL_LOSS    │         min │
│ PARAMS      │         min │
└─────────────┴─────────────┘
⠋ Write files and show overview

Arguments Overview

Key	Value
config_yaml	None
config_toml	None
config_json	None
num_random_steps	20
max_eval	1000
run_program	[["'cHl0aG9uMyAvZGF0YS9ob3JzZS93cy9zMzgxMTE0MS1ncm9rMi90cmFpbi5weSAtLWJhdGNoLXNpemU9JShiYXRjaF9zaXplKSAtLXRvcG8gLS10b3BvLWV2ZXJ5IDEgLS1rZWxwLWV2ZXJ5PTE…
experiment_name	tinyLogicLLM2
mem_gb	32
parameter	[['batch_size', 'range', '16', '512', 'int', 'false'], ['ffn', 'range', '2', '512', 'int', 'false'], ['max_turnstiles', 'range', '5', '64', 'int',
	'false'], ['epochs', 'range', '10', '500', 'int', 'false'], ['n_layers', 'range', '1', '20', 'int', 'false'], ['d_model', 'range', '2', '8', 'int',
	'false'], ['max_sequence_length', 'range', '5', '64', 'int', 'false'], ['val_batches', 'fixed', '128'], ['lr', 'range', '0.0000001', '0.1', 'float',
	'false'], ['n_heads', 'range', '1', '128', 'int', 'false'], ['batches_per_epoch', 'range', '1', '20', 'int', 'false']]
continue_previous_job	None
experiment_constraints	None
run_dir	runs
seed	None
verbose_tqdm	False
model	BOTORCH_MODULAR
gridsearch	False
occ	False
show_sixel_scatter	True
show_sixel_general	True
show_sixel_trial_index_result	True
follow	True
send_anonymized_usage_stats	True
ui_url	None
root_venv_dir	/home/s3811141
exclude	None
main_process_gb	8
max_nr_of_zero_results	50
abbreviate_job_names	False
orchestrator_file	None
checkout_to_latest_tested_version	False
live_share	True
disable_tqdm	False
disable_previous_job_constraint	False
workdir
occ_type	euclid
result_names	["'VAL_LOSS=min", "PARAMS=min'"]
minkowski_p	2
signed_weighted_euclidean_weights
generation_strategy	None
generate_all_jobs_at_once	True
revert_to_random_when_seemingly_exhausted	True
load_data_from_existing_jobs	[]
n_estimators_randomforest	100
max_attempts_for_generation	20
external_generator	None
username	None
max_failed_jobs	0
num_cpus_main_job	None
calculate_pareto_front_of_job	[]
show_generate_time_table	False
force_choice_for_ranges	False
max_abandoned_retrial	20
share_password	None
dryrun	False
db_url	None
run_program_once	None
worker_generator_path	None
save_to_database	False
range_max_difference	1000000
skip_search	False
nr_evals_per_arm	1
dont_warm_start_refitting	False
refit_on_cv	False
fit_out_of_design	False
fit_abandoned	False
dont_jit_compile	False
num_restarts	20
raw_samples	1024
max_num_of_parallel_sruns	16
no_transform_inputs	False
no_normalize_y	False
transforms	[]
number_of_generators	1
num_parallel_jobs	20
worker_timeout	720
slurm_use_srun	False
time	7200
partition	alpha
reservation	None
force_local_execution	False
slurm_signal_delay_s	0
nodes_per_job	1
cpus_per_task	1
account	None
gpus	1
dependency	None
run_mode	local
verbose	False
verbose_break_run_search_table	False
debug	False
flame_graph	False
memray	False
no_sleep	False
tests	False
show_worker_percentage_table_at_end	False
auto_exclude_defective_hosts	False
run_tests_that_fail_on_taurus	False
raise_in_eval	False
show_ram_every_n_seconds	0
show_generation_and_submission_sixel	False
just_return_defaults	False
prettyprint	False
runtime_debug	False
debug_stack_regex
debug_stack_trace_regex	None
show_func_name	False
beartype	False

1778046082.911,20,0,0
1778046084.0601,20,0,0

This logs the CPU and RAM usage of the main worker process.

timestamp,ram_usage_mb,cpu_usage_percent
1778046082,793.1484375,10.2