⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Job 3493635 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493635/3493635_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493635/3493635_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,379) - Starting with JobEnvironment(job_id=3493635, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,381) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493635/3493635_submitted.pkl
Trial-Index: 18
slurmstepd: error: *** JOB 3493635 ON c144 CANCELLED AT 2026-05-06T19:59:40 DUE TO TIME LIMIT ***
⚠ Job 3493630 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493630/3493630_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493630/3493630_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,381) - Starting with JobEnvironment(job_id=3493630, hostname=c148, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,383) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493630/3493630_submitted.pkl
Trial-Index: 14
slurmstepd: error: *** JOB 3493630 ON c148 CANCELLED AT 2026-05-06T19:59:41 DUE TO TIME LIMIT ***
⚠ Job 3493634 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493634/3493634_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493634/3493634_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,379) - Starting with JobEnvironment(job_id=3493634, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,381) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493634/3493634_submitted.pkl
Trial-Index: 16
slurmstepd: error: *** JOB 3493634 ON c144 CANCELLED AT 2026-05-06T19:59:40 DUE TO TIME LIMIT ***
⚠ Job 3493645 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493645/3493645_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493645/3493645_0_log.out
----------------------------------------
submitit INFO (2026-05-06 08:00:08,561) - Starting with JobEnvironment(job_id=3493645, hostname=c124, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 08:00:08,563) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493645/3493645_submitted.pkl
Trial-Index: 9
slurmstepd: error: *** JOB 3493645 ON c124 CANCELLED AT 2026-05-06T20:00:10 DUE TO TIME LIMIT ***
⚠ Job 3493629 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493629/3493629_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493629/3493629_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,381) - Starting with JobEnvironment(job_id=3493629, hostname=c148, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,383) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493629/3493629_submitted.pkl
Trial-Index: 2
slurmstepd: error: *** JOB 3493629 ON c148 CANCELLED AT 2026-05-06T19:59:40 DUE TO TIME LIMIT ***
⚠ Job 3493643 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493643/3493643_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493643/3493643_0_log.out
----------------------------------------
submitit INFO (2026-05-06 08:00:08,561) - Starting with JobEnvironment(job_id=3493643, hostname=c124, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 08:00:08,563) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493643/3493643_submitted.pkl
Trial-Index: 13
slurmstepd: error: *** JOB 3493643 ON c124 CANCELLED AT 2026-05-06T20:00:10 DUE TO TIME LIMIT ***
⚠ Job 3493636 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493636/3493636_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493636/3493636_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,440) - Starting with JobEnvironment(job_id=3493636, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,442) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493636/3493636_submitted.pkl
Trial-Index: 7
slurmstepd: error: *** JOB 3493636 ON c140 CANCELLED AT 2026-05-06T19:59:40 DUE TO TIME LIMIT ***
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Job 3497627 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497627/3497627_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497627/3497627_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:14:15,093) - Starting with JobEnvironment(job_id=3497627, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:14:15,095) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497627/3497627_submitted.pkl
Trial-Index: 26
slurmstepd: error: *** JOB 3497627 ON c142 CANCELLED AT 2026-05-08T10:14:39 DUE TO TIME LIMIT ***
⚠ Job 3497629 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497629/3497629_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497629/3497629_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:14:15,009) - Starting with JobEnvironment(job_id=3497629, hostname=c130, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:14:15,011) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497629/3497629_submitted.pkl
Trial-Index: 34
slurmstepd: error: *** JOB 3497629 ON c130 CANCELLED AT 2026-05-08T10:14:38 DUE TO TIME LIMIT ***
⚠ Job 3497633 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497633/3497633_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497633/3497633_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:14:45,100) - Starting with JobEnvironment(job_id=3497633, hostname=c92, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:14:45,102) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497633/3497633_submitted.pkl
Trial-Index: 30
slurmstepd: error: *** JOB 3497633 ON c92 CANCELLED AT 2026-05-08T10:15:08 DUE TO TIME LIMIT ***
⚠ Job 3497643 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497643/3497643_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497643/3497643_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:15:45,783) - Starting with JobEnvironment(job_id=3497643, hostname=c145, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:15:45,785) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497643/3497643_submitted.pkl
Trial-Index: 38
slurmstepd: error: *** JOB 3497643 ON c145 CANCELLED AT 2026-05-08T10:16:08 DUE TO TIME LIMIT ***
⚠ Job 3497624 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497624/3497624_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497624/3497624_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:13:44,773) - Starting with JobEnvironment(job_id=3497624, hostname=c103, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:13:44,775) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497624/3497624_submitted.pkl
Trial-Index: 21
slurmstepd: error: *** JOB 3497624 ON c103 CANCELLED AT 2026-05-08T10:14:09 DUE TO TIME LIMIT ***
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Job 3504529 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504529/3504529_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504529/3504529_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:23:05,381) - Starting with JobEnvironment(job_id=3504529, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:23:05,384) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504529/3504529_submitted.pkl
Trial-Index: 49
slurmstepd: error: *** JOB 3504529 ON c27 CANCELLED AT 2026-05-08T22:23:21 DUE TO TIME LIMIT ***
⚠ Job 3504535 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504535/3504535_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504535/3504535_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:25:38,255) - Starting with JobEnvironment(job_id=3504535, hostname=c143, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:25:38,258) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504535/3504535_submitted.pkl
Trial-Index: 54
slurmstepd: error: *** JOB 3504535 ON c143 CANCELLED AT 2026-05-08T22:25:52 DUE TO TIME LIMIT ***
⚠ Job 3504540 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504540/3504540_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504540/3504540_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:29:13,048) - Starting with JobEnvironment(job_id=3504540, hostname=c111, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:29:13,051) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504540/3504540_submitted.pkl
Trial-Index: 57
slurmstepd: error: *** JOB 3504540 ON c111 CANCELLED AT 2026-05-08T22:29:22 DUE TO TIME LIMIT ***
⚠ Job 3504526 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504526/3504526_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504526/3504526_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:23:05,629) - Starting with JobEnvironment(job_id=3504526, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:23:05,632) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504526/3504526_submitted.pkl
Trial-Index: 53
slurmstepd: error: *** JOB 3504526 ON c142 CANCELLED AT 2026-05-08T22:23:21 DUE TO TIME LIMIT ***
⚠ Job 3504534 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504534/3504534_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504534/3504534_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:25:07,789) - Starting with JobEnvironment(job_id=3504534, hostname=c82, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:25:07,792) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504534/3504534_submitted.pkl
Trial-Index: 51
slurmstepd: error: *** JOB 3504534 ON c82 CANCELLED AT 2026-05-08T22:25:22 DUE TO TIME LIMIT ***
⚠ Job 3504527 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504527/3504527_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504527/3504527_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:23:06,051) - Starting with JobEnvironment(job_id=3504527, hostname=c102, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:23:06,054) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504527/3504527_submitted.pkl
Trial-Index: 40
slurmstepd: error: *** JOB 3504527 ON c102 CANCELLED AT 2026-05-08T22:23:21 DUE TO TIME LIMIT ***
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
To cancel, press CTRLc, then run 'scancel 3493626'
⠋Importing logging...⠋Importing warnings...⠋Importing argparse...⠋Importing datetime...⠋Importing dataclass...⠋Importing socket...⠋Importing stat...⠋Importing pwd...⠋Importing base64...⠋Importing json...⠋Importing yaml...⠋Importing toml...⠋Importing csv...⠋Importing ast...⠋Importing rich.table...⠋Importing rich print...⠋Importing rich.pretty...⠋Importing pformat...⠋Importing rich.prompt...⠋Importing types.FunctionType...⠋Importing typing...⠋Importing ThreadPoolExecutor...⠋Importing submitit.LocalExecutor...⠋Importing submitit.Job...⠋Importing importlib.util...⠋Importing platform...⠋Importing inspect frame info...⠋Importing pathlib.Path...⠋Importing uuid...⠋Importing cowsay...⠋Importing shutil...⠋Importing itertools.combinations...⠋Importing os.listdir...⠋Importing os.path...⠋Importing PIL.Image...⠋Importing sixel...⠋Importing subprocess...⠋Importing tqdm...⠋Importing beartype...
[WARNING 05-06 07:58:54] ax.storage.sqa_store.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
⠏Importing rendering stuff...⠋Importing statistics...⠋Trying to import pyfiglet...⠋Importing helpers...⠋Importing pareto...⠋Parsing arguments...⠋Importing torch...⠋Importing numpy...⠋Importing ax...⠋Importing ax.core.generator_run...⠋Importing Cont_X_trans and Y_trans from ax.adapter.registry...⠋Importing ax.core.arm...⠋Importing ax.core.objective...⠋Importing ax.core.Metric...⠋Importing ax.exceptions.core...⠋Importing ax.exceptions.generation_strategy...⠋Importing CORE_DECODER_REGISTRY...⠋Trying ax.generation_strategy.generation_node...⠋Importing GenerationStep, GenerationStrategy from generation_strategy...⠋Importing GenerationNode from generation_node...⠋Importing ExternalGenerationNode...⠋Importing MinTrials...⠋Importing GeneratorSpec...⠋Importing Generators from ax.generation_strategy.registry...⠋Importing load_experiment...⠋Importing save_experiment...⠋Importing save_experiment_to_db...⠋Importing TrialStatus...⠋Importing Data...⠋Importing Experiment...⠋Importing parameter types...⠋Importing TParameterization...⠋Importing pandas...⠋Importing AxClient and ObjectiveProperties...⠋Importing RandomForestRegressor...⠋Importing botorch...⠋Importing submitit...⠋Importing ax logger...⠋Importing SQL-Storage-Stuff...
Run-UUID: 6ec0b16b-8362-479a-b057-e9e64e252084
____ _ ____ _ ___
/ __ \ (_)/ __ \ | | |__ \
| | | |_ __ ___ _ __ _| | | |_ __ | |_ ) |
| | | | '_ ` _ \| '_ \| | | | | '_ \| __| / /
| |__| | | | | | | | | | | |__| | |_) | |_ / /_
\____/|_| |_| |_|_| |_|_|\____/| .__/ \__|____|
| |
|_|
⠋Writing worker creation log...
omniopt --partition=alpha --experiment_name=tinyLogicLLM2 --mem_gb=32 --time=7200 --worker_timeout=720 --max_eval=1000 --num_parallel_jobs=20 --gpus=1 --num_random_steps=20 --follow --live_share --send_anonymized_usage_stats --result_names 'VAL_LOSS=min PARAMS=min' --run_program='cHl0aG9uMyAvZGF0YS9ob3JzZS93cy9zMzgxMTE0MS1ncm9rMi90cmFpbi5weSAtLWJhdGNoLXNpemU9JShiYXRjaF9zaXplKSAtLXRvcG8gLS10b3BvLWV2ZXJ5IDEgLS1rZWxwLWV2ZXJ5PTEgLS1uLWxheWVycz0lKG5fbGF5ZXJzKSAtLWQtbW9kZWw9JShkX21vZGVsKSAtLW4taGVhZHM9JShuX2hlYWRzKSAtLXRva2VuaXplcl9pbml0aWFsX25yPTEwMDAgLS1scj0lKGxyKSAtLWJhdGNoZXMtcGVyLWVwb2NoPSUoYmF0Y2hlc19wZXJfZXBvY2gpIC0tdmFsLWJhdGNoZXM9JSh2YWxfYmF0Y2hlcykgLS1lcG9jaHM9JShlcG9jaHMpIC0tdGFzaz10dXJuc3RpbGUgLS1tYXgtc2VxLWxlbiAlKG1heF9zZXF1ZW5jZV9sZW5ndGgpIC0tbWF4LXR1cm5zdGlsZXMgJShtYXhfdHVybnN0aWxlcykgLS1zY2hlZHVsZXI9Y29zaW5lIC0tZmZuPSUoZmZuKQo=' --run_program_once='cHl0aG9uMyAvZGF0YS9ob3JzZS93cy9zMzgxMTE0MS1ncm9rMi90cmFpbi5weSAtLWV4aXQtYWZ0ZXItaW5zdGFsbA==' --cpus_per_task=1 --nodes_per_job=1 --revert_to_random_when_seemingly_exhausted --model=BOTORCH_MODULAR --n_estimators_randomforest=100 --run_mode=local --show_sixel_general --show_sixel_trial_index_result --show_sixel_scatter --occ_type=euclid --main_process_gb=8 --nr_evals_per_arm=1 --max_nr_of_zero_results=50 --slurm_signal_delay_s=0 --max_failed_jobs=0 --max_attempts_for_generation=20 --num_restarts=20 --raw_samples=1024 --max_abandoned_retrial=20 --max_num_of_parallel_sruns=16 --number_of_generators=1 --generate_all_jobs_at_once --parameter batch_size range 16 512 int false --parameter ffn range 2 512 int false --parameter max_turnstiles range 5 64 int false --parameter epochs range 10 500 int false --parameter n_layers range 1 20 int false --parameter d_model range 2 8 int false --parameter max_sequence_length range 5 64 int false --parameter val_batches fixed 128 --parameter lr range 0.0000001 0.1 float false --parameter n_heads range 1 128 int false --parameter batches_per_epoch range 1 20 int false
⠋Disabling logging...⠋Setting run folder...⠋Creating folder /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0...⠋Writing revert_to_random_when_seemingly_exhausted file ...⠋Writing username state file...⠋Writing result names file...⠋Writing result min/max file...
Exiting after installation
Executing command: [cyan]python3 /data/horse/ws/s3811141-grok2/train.py --exit-after-install[/cyan]
Setup script completed successfully ✅
⠋Saving state files...Run-folder: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0⠋Writing live_share file if it is present...⠋Writing job_start_time file...⠙Writing git information⠋Checking max_eval...⠋Calculating number of steps...⠋Adding excluded nodes...⠋Initializing ax_client...⠋Setting orchestrator...See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=tinyLogicLLM2&run_nr=1 for live-results.You have 1 CPUs available for the main process.Using CUDA device NVIDIA H100.Generation strategy: SOBOL for 20 steps and then BOTORCH_MODULAR for 980 steps.
Run-Program: python3 /data/horse/ws/s3811141-grok2/train.py --batch-size=%(batch_size) --topo --topo-every 1 --kelp-every=1 --n-layers=%(n_layers) --d-model=%(d_model) --n-heads=%(n_heads) --tokenizer_initial_nr=1000 --lr=%(lr) --batches-per-epoch=%(batches_per_epoch) --val-batches=%(val_batches) --epochs=%(epochs) --task=turnstile --max-seq-len %(max_sequence_length) --max-turnstiles %(max_turnstiles) --scheduler=cosine --ffn=%(ffn)
Experiment parameters
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃Name ┃Type ┃Lower bound┃Upper bound┃Values┃Type ┃Log Scale?┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│batch_size │range│16 │512 ││int │No │
│ffn │range│2 │512 ││int │No │
│max_turnstiles │range│5 │64 ││int │No │
│epochs │range│10 │500 ││int │No │
│n_layers │range│1 │20 ││int │No │
│d_model │range│2 │8 ││int │No │
│max_sequence_length│range│5 │64 ││int │No │
│val_batches │fixed│││128 │││
│lr │range│1e-07 │0.1 ││float│No │
│n_heads │range│1 │128 ││int │No │
│batches_per_epoch │range│1 │20 ││int │No │
└─────────────────────┴───────┴─────────────┴─────────────┴────────┴───────┴────────────┘
Result-Names
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃Result-Name┃Min or max?┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│VAL_LOSS │ min│
│PARAMS │ min│
└─────────────┴─────────────┘
⠋Write files and show overview
SOBOL, running/pending 10/10 = ∑20/20, waiting for 20 jobs: 0%|░░░░░░░░░░| 0/1000 [00:28, ?it/s]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, running 20 = ∑20/20, job_failed : 0%|░░░░░░░░░░| 0/1000 [01:00, ?it/s]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 4 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), running 16 = ∑16/20, waiting for 16 jobs : 0%|░░░░░░░░░░| 0/1000 [01:01, ?it/s]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 6 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), running 14 = ∑14/20, waiting for 14 jobs : 0%|░░░░░░░░░░| 0/1000 [01:18, ?it/s]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 6 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), running 14 = ∑14/20, job_failed : 0%|░░░░░░░░░░| 0/1000 [01:28, ?it/s]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 11 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, timeout 7 = ∑7/20, waiting for 7 jobs : 0%|░░░░░░░░░░| 2/1000 [12:02:15<3502:56:41, 12635.87s/it]
⚠ Job 3493629 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493629/3493629_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493629/3493629_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,381) - Starting with JobEnvironment(job_id=3493629, hostname=c148, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,383) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493629/3493629_submitted.pkl
Trial-Index: 2
slurmstepd: error: *** JOB 3493629 ON c148 CANCELLED AT 2026-05-06T19:59:40 DUE TO TIME LIMIT ***
⚠ Job 3493635 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493635/3493635_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493635/3493635_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,379) - Starting with JobEnvironment(job_id=3493635, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,381) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493635/3493635_submitted.pkl
Trial-Index: 18
slurmstepd: error: *** JOB 3493635 ON c144 CANCELLED AT 2026-05-06T19:59:40 DUE TO TIME LIMIT ***
⚠ Job 3493643 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493643/3493643_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493643/3493643_0_log.out
----------------------------------------
submitit INFO (2026-05-06 08:00:08,561) - Starting with JobEnvironment(job_id=3493643, hostname=c124, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 08:00:08,563) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493643/3493643_submitted.pkl
Trial-Index: 13
slurmstepd: error: *** JOB 3493643 ON c124 CANCELLED AT 2026-05-06T20:00:10 DUE TO TIME LIMIT ***
⚠ Job 3493645 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493645/3493645_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493645/3493645_0_log.out
----------------------------------------
submitit INFO (2026-05-06 08:00:08,561) - Starting with JobEnvironment(job_id=3493645, hostname=c124, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 08:00:08,563) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493645/3493645_submitted.pkl
Trial-Index: 9
slurmstepd: error: *** JOB 3493645 ON c124 CANCELLED AT 2026-05-06T20:00:10 DUE TO TIME LIMIT ***
⚠ Job 3493634 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493634/3493634_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493634/3493634_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,379) - Starting with JobEnvironment(job_id=3493634, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,381) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493634/3493634_submitted.pkl
Trial-Index: 16
slurmstepd: error: *** JOB 3493634 ON c144 CANCELLED AT 2026-05-06T19:59:40 DUE TO TIME LIMIT ***
⚠ Job 3493630 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493630/3493630_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493630/3493630_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,381) - Starting with JobEnvironment(job_id=3493630, hostname=c148, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,383) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493630/3493630_submitted.pkl
Trial-Index: 14
slurmstepd: error: *** JOB 3493630 ON c148 CANCELLED AT 2026-05-06T19:59:41 DUE TO TIME LIMIT ***
⚠ Job 3493636 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493636/3493636_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493636/3493636_0_log.out
----------------------------------------
submitit INFO (2026-05-06 07:59:38,440) - Starting with JobEnvironment(job_id=3493636, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-06 07:59:38,442) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3493636/3493636_submitted.pkl
Trial-Index: 7
slurmstepd: error: *** JOB 3493636 ON c140 CANCELLED AT 2026-05-06T19:59:40 DUE TO TIME LIMIT ***
SOBOL, failed: 18 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, pending 20 = ∑20/20, waiting for 20 jobs : 0%|░░░░░░░░░░| 2/1000 [12:03:49<3502:56:41, 12635.87s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 19 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, pending 19 = ∑19/20, waiting for 20 jobs, finished 1 job : 0%|░░░░░░░░░░| 2/1000 [38:15:16<3502:56:41, 12635.87s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 20 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, pending 18 = ∑18/20, waiting for 18 jobs : 0%|░░░░░░░░░░| 2/1000 [38:15:22<3502:56:41, 12635.87s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 21 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, pending 17 = ∑17/20, waiting for 17 jobs : 0%|░░░░░░░░░░| 2/1000 [38:15:47<3502:56:41, 12635.87s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 22 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, pending 16 = ∑16/20, waiting for 17 jobs, finished 1 job : 0%|░░░░░░░░░░| 2/1000 [38:15:52<3502:56:41, 12635.87s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 23 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, pending 15 = ∑15/20, waiting for 15 jobs : 0%|░░░░░░░░░░| 2/1000 [38:15:58<3502:56:41, 12635.87s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 24 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, pending 14 = ∑14/20, waiting for 14 jobs : 0%|░░░░░░░░░░| 2/1000 [38:16:13<3502:56:41, 12635.87s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 26 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, pending 12 = ∑12/20, waiting for 12 jobs : 0%|░░░░░░░░░░| 2/1000 [38:16:25<3502:56:41, 12635.87s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 27 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 2 done, pending 11 = ∑11/20, waiting for 11 jobs : 0%|░░░░░░░░░░| 2/1000 [38:16:31<3502:56:41, 12635.87s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 28 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, running 5 = ∑5/20, waiting for 5 jobs : 1%|░░░░░░░░░░| 7/1000 [45:07:22<4663:37:40, 16907.41s/it]
⚠ Job 3497627 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497627/3497627_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497627/3497627_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:14:15,093) - Starting with JobEnvironment(job_id=3497627, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:14:15,095) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497627/3497627_submitted.pkl
Trial-Index: 26
slurmstepd: error: *** JOB 3497627 ON c142 CANCELLED AT 2026-05-08T10:14:39 DUE TO TIME LIMIT ***
⚠ Job 3497633 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497633/3497633_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497633/3497633_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:14:45,100) - Starting with JobEnvironment(job_id=3497633, hostname=c92, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:14:45,102) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497633/3497633_submitted.pkl
Trial-Index: 30
slurmstepd: error: *** JOB 3497633 ON c92 CANCELLED AT 2026-05-08T10:15:08 DUE TO TIME LIMIT ***
⚠ Job 3497629 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497629/3497629_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497629/3497629_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:14:15,009) - Starting with JobEnvironment(job_id=3497629, hostname=c130, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:14:15,011) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497629/3497629_submitted.pkl
Trial-Index: 34
slurmstepd: error: *** JOB 3497629 ON c130 CANCELLED AT 2026-05-08T10:14:38 DUE TO TIME LIMIT ***
⚠ Job 3497643 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497643/3497643_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497643/3497643_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:15:45,783) - Starting with JobEnvironment(job_id=3497643, hostname=c145, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:15:45,785) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497643/3497643_submitted.pkl
Trial-Index: 38
slurmstepd: error: *** JOB 3497643 ON c145 CANCELLED AT 2026-05-08T10:16:08 DUE TO TIME LIMIT ***
⚠ Job 3497624 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497624/3497624_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497624/3497624_0_log.out
----------------------------------------
submitit INFO (2026-05-07 22:13:44,773) - Starting with JobEnvironment(job_id=3497624, hostname=c103, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-07 22:13:44,775) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3497624/3497624_submitted.pkl
Trial-Index: 21
slurmstepd: error: *** JOB 3497624 ON c103 CANCELLED AT 2026-05-08T10:14:09 DUE TO TIME LIMIT ***
SOBOL, failed: 33 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, pending/running 18/2 = ∑20/20, waiting for 20 jobs : 1%|░░░░░░░░░░| 7/1000 [50:22:21<4663:37:40, 16907.41s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 34 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, running/pending 8/11 = ∑19/20, waiting for 19 jobs : 1%|░░░░░░░░░░| 7/1000 [50:24:39<4663:37:40, 16907.41s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 36 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, running/pending 6/11 = ∑17/20, waiting for 17 jobs : 1%|░░░░░░░░░░| 7/1000 [50:24:58<4663:37:40, 16907.41s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 37 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, running/pending 5/11 = ∑16/20, waiting for 16 jobs : 1%|░░░░░░░░░░| 7/1000 [50:25:40<4663:37:40, 16907.41s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 39 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, running/pending 5/9 = ∑14/20, waiting for 14 jobs : 1%|░░░░░░░░░░| 7/1000 [50:25:55<4663:37:40, 16907.41s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 40 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, running/pending 4/9 = ∑13/20, waiting for 13 jobs : 1%|░░░░░░░░░░| 7/1000 [50:26:12<4663:37:40, 16907.41s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 41 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, running/pending 4/8 = ∑12/20, waiting for 12 jobs : 1%|░░░░░░░░░░| 7/1000 [50:26:23<4663:37:40, 16907.41s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 42 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, running/pending 8/3 = ∑11/20, waiting for 11 jobs : 1%|░░░░░░░░░░| 7/1000 [50:27:18<4663:37:40, 16907.41s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 43 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 7 done, running/pending 8/2 = ∑10/20, waiting for 10 jobs : 1%|░░░░░░░░░░| 7/1000 [50:30:37<4663:37:40, 16907.41s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 44 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, timeout 6 = ∑6/20, waiting for 6 jobs : 1%|░░░░░░░░░░| 10/1000 [62:33:10<4112:05:18, 14953.05s/it]
⚠ Job 3504534 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504534/3504534_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504534/3504534_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:25:07,789) - Starting with JobEnvironment(job_id=3504534, hostname=c82, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:25:07,792) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504534/3504534_submitted.pkl
Trial-Index: 51
slurmstepd: error: *** JOB 3504534 ON c82 CANCELLED AT 2026-05-08T22:25:22 DUE TO TIME LIMIT ***
⚠ Job 3504529 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504529/3504529_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504529/3504529_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:23:05,381) - Starting with JobEnvironment(job_id=3504529, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:23:05,384) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504529/3504529_submitted.pkl
Trial-Index: 49
slurmstepd: error: *** JOB 3504529 ON c27 CANCELLED AT 2026-05-08T22:23:21 DUE TO TIME LIMIT ***
⚠ Job 3504526 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504526/3504526_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504526/3504526_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:23:05,629) - Starting with JobEnvironment(job_id=3504526, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:23:05,632) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504526/3504526_submitted.pkl
Trial-Index: 53
slurmstepd: error: *** JOB 3504526 ON c142 CANCELLED AT 2026-05-08T22:23:21 DUE TO TIME LIMIT ***
⚠ Job 3504540 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504540/3504540_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504540/3504540_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:29:13,048) - Starting with JobEnvironment(job_id=3504540, hostname=c111, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:29:13,051) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504540/3504540_submitted.pkl
Trial-Index: 57
slurmstepd: error: *** JOB 3504540 ON c111 CANCELLED AT 2026-05-08T22:29:22 DUE TO TIME LIMIT ***
⚠ Job 3504535 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504535/3504535_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504535/3504535_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:25:38,255) - Starting with JobEnvironment(job_id=3504535, hostname=c143, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:25:38,258) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504535/3504535_submitted.pkl
Trial-Index: 54
slurmstepd: error: *** JOB 3504535 ON c143 CANCELLED AT 2026-05-08T22:25:52 DUE TO TIME LIMIT ***
⚠ Job 3504527 (task: 0) with path /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504527/3504527_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504527/3504527_0_log.out
----------------------------------------
submitit INFO (2026-05-08 10:23:06,051) - Starting with JobEnvironment(job_id=3504527, hostname=c102, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-08 10:23:06,054) - Loading pickle: /data/horse/ws/s3811141-grok2/omniopt/runs/tinyLogicLLM2/0/single_runs/3504527/3504527_submitted.pkl
Trial-Index: 40
slurmstepd: error: *** JOB 3504527 ON c102 CANCELLED AT 2026-05-08T22:23:21 DUE TO TIME LIMIT ***
SOBOL, failed: 50 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, running/pending 1/19 = ∑20/20, waiting for 20 jobs : 1%|░░░░░░░░░░| 10/1000 [62:37:24<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 51 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, running/pending 1/18 = ∑19/20, waiting for 19 jobs : 1%|░░░░░░░░░░| 10/1000 [62:38:35<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 52 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, pending 18 = ∑18/20, waiting for 18 jobs : 1%|░░░░░░░░░░| 10/1000 [62:39:01<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 53 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, running/pending 2/15 = ∑17/20, waiting for 17 jobs : 1%|░░░░░░░░░░| 10/1000 [62:40:57<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 54 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, running/pending 2/14 = ∑16/20, waiting for 16 jobs : 1%|░░░░░░░░░░| 10/1000 [62:41:58<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 55 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, running/pending 2/13 = ∑15/20, waiting for 15 jobs : 1%|░░░░░░░░░░| 10/1000 [62:42:28<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 56 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, running/pending 2/12 = ∑14/20, waiting for 14 jobs : 1%|░░░░░░░░░░| 10/1000 [62:42:42<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 57 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, running/pending 2/11 = ∑13/20, waiting for 13 jobs : 1%|░░░░░░░░░░| 10/1000 [62:43:12<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 58 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, running/pending 2/10 = ∑12/20, waiting for 12 jobs : 1%|░░░░░░░░░░| 10/1000 [62:43:42<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 59 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 10 done, running/pending 2/9 = ∑11/20, waiting for 11 jobs : 1%|░░░░░░░░░░| 10/1000 [62:44:00<4112:05:18, 14953.05s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 60 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 13 done, running 7 = ∑7/20, new result: VAL_LOSS: 0.083276, PARAMS: 2271332.000000 : 1%|░░░░░░░░░░| 14/1000 [72:48:05<3942:42:14, 14395.27s/it]
2026-05-06 07:59:13 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, Started OmniOpt2 run...
2026-05-06 07:59:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #1/20
2026-05-06 07:59:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #2/20
2026-05-06 07:59:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #3/20
2026-05-06 07:59:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #4/20
2026-05-06 07:59:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #5/20
2026-05-06 07:59:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #6/20
2026-05-06 07:59:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #7/20
2026-05-06 07:59:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #8/20
2026-05-06 07:59:15 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #9/20
2026-05-06 07:59:15 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #10/20
2026-05-06 07:59:16 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #11/20
2026-05-06 07:59:16 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #12/20
2026-05-06 07:59:16 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #13/20
2026-05-06 07:59:16 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #14/20
2026-05-06 07:59:17 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #15/20
2026-05-06 07:59:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #16/20
2026-05-06 07:59:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #17/20
2026-05-06 07:59:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #18/20
2026-05-06 07:59:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #19/20
2026-05-06 07:59:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, getting new HP set #20/20
2026-05-06 07:59:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, requested 20 jobs, got 20, 0.25 s/job
2026-05-06 07:59:19 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #1/20 start
2026-05-06 07:59:19 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #2/20 start
2026-05-06 07:59:19 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #3/20 start
2026-05-06 07:59:19 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #4/20 start
2026-05-06 07:59:21 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #5/20 start
2026-05-06 07:59:21 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #6/20 start
2026-05-06 07:59:21 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #7/20 start
2026-05-06 07:59:21 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #8/20 start
2026-05-06 07:59:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #9/20 start
2026-05-06 07:59:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #10/20 start
2026-05-06 07:59:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #11/20 start
2026-05-06 07:59:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #12/20 start
2026-05-06 07:59:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #13/20 start
2026-05-06 07:59:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #14/20 start
2026-05-06 07:59:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #15/20 start
2026-05-06 07:59:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #16/20 start
2026-05-06 07:59:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #17/20 start
2026-05-06 07:59:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #18/20 start
2026-05-06 07:59:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #19/20 start
2026-05-06 07:59:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, eval #20/20 start
2026-05-06 07:59:25 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, starting new job
2026-05-06 07:59:26 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 1 = ∑1/20, started new job
2026-05-06 07:59:26 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 2 = ∑2/20, started new job
2026-05-06 07:59:26 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 3 = ∑3/20, started new job
2026-05-06 07:59:26 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 2 = ∑2/20, starting new job
2026-05-06 07:59:26 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 3 = ∑3/20, starting new job
2026-05-06 07:59:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 4 = ∑4/20, started new job
2026-05-06 07:59:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 4 = ∑4/20, starting new job
2026-05-06 07:59:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 5 = ∑5/20, started new job
2026-05-06 07:59:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 4 = ∑4/20, starting new job
2026-05-06 07:59:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, unknown 6 = ∑6/20, started new job
2026-05-06 07:59:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/1 = ∑7/20, started new job
2026-05-06 07:59:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/2 = ∑8/20, started new job
2026-05-06 07:59:28 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/3 = ∑9/20, started new job
2026-05-06 07:59:28 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/4 = ∑10/20, started new job
2026-05-06 07:59:28 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/5 = ∑11/20, started new job
2026-05-06 07:59:28 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/6 = ∑12/20, started new job
2026-05-06 07:59:28 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/7 = ∑13/20, started new job
2026-05-06 07:59:29 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/8 = ∑14/20, started new job
2026-05-06 07:59:29 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/9 = ∑15/20, started new job
2026-05-06 07:59:29 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/10 = ∑16/20, started new job
2026-05-06 07:59:29 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/11 = ∑17/20, started new job
2026-05-06 07:59:29 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/12 = ∑18/20, started new job
2026-05-06 07:59:30 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/13 = ∑19/20, started new job
2026-05-06 07:59:30 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/14 = ∑20/20, started new job
2026-05-06 07:59:31 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending/unknown 6/14 = ∑20/20, waiting for 20 jobs
2026-05-06 07:59:35 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-06 07:59:40 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, running/pending 10/10 = ∑20/20, waiting for 20 jobs
2026-05-06 08:00:12 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, running 20 = ∑20/20, job_failed
2026-05-06 08:00:12 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 4 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 16 = ∑16/20, waiting for 20 jobs, finished 4 jobs
2026-05-06 08:00:12 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 4 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 16 = ∑16/20, waiting for 16 jobs
2026-05-06 08:00:30 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 4 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 16 = ∑16/20, job_failed
2026-05-06 08:00:30 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 6 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 14 = ∑14/20, waiting for 16 jobs, finished 2 jobs
2026-05-06 08:00:30 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 6 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 14 = ∑14/20, waiting for 14 jobs
2026-05-06 08:00:40 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 6 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 14 = ∑14/20, job_failed
2026-05-06 08:00:41 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 7 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 13 = ∑13/20, waiting for 14 jobs, finished 1 job
2026-05-06 08:00:41 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 7 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 13 = ∑13/20, waiting for 13 jobs
2026-05-06 08:00:42 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 7 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 13 = ∑13/20, job_failed
2026-05-06 08:00:42 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 9 = ∑9/20, waiting for 13 jobs, finished 4 jobs
2026-05-06 08:00:42 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 9 = ∑9/20, waiting for 9 jobs
2026-05-06 10:39:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 9 = ∑9/20, new result: VAL_LOSS: 0.000008, PARAMS: 446420.000000
2026-05-06 10:39:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 1 done, running 8 = ∑8/20, waiting for 9 jobs, finished 1 job
2026-05-06 10:39:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 1 done, running 8 = ∑8/20, waiting for 8 jobs
2026-05-06 14:45:15 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 1 done, running 8 = ∑8/20, new result: VAL_LOSS: 0.000311, PARAMS: 880836.000000
2026-05-06 14:45:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, running 7 = ∑7/20, waiting for 8 jobs, finished 1 job
2026-05-06 14:45:28 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, running 7 = ∑7/20, waiting for 7 jobs
2026-05-06 20:01:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, timeout 7 = ∑7/20, waiting for 7 jobs
2026-05-06 20:01:45 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 11 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, timeout 7 = ∑7/20, job_failed
2026-05-06 20:01:49 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, waiting for 7 jobs, finished 7 jobs
2026-05-06 20:01:51 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #1/20
2026-05-06 20:01:52 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #2/20
2026-05-06 20:01:54 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #3/20
2026-05-06 20:01:54 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #4/20
2026-05-06 20:01:55 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #5/20
2026-05-06 20:01:55 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #6/20
2026-05-06 20:01:55 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #7/20
2026-05-06 20:01:56 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #8/20
2026-05-06 20:01:56 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #9/20
2026-05-06 20:01:57 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #10/20
2026-05-06 20:01:57 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #11/20
2026-05-06 20:01:59 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #12/20
2026-05-06 20:01:59 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #13/20
2026-05-06 20:02:00 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #14/20
2026-05-06 20:02:00 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #15/20
2026-05-06 20:02:01 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #16/20
2026-05-06 20:02:01 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #17/20
2026-05-06 20:02:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #18/20
2026-05-06 20:02:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #19/20
2026-05-06 20:02:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, getting new HP set #20/20
2026-05-06 20:02:04 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, requested 20 jobs, got 20, 0.63 s/job
2026-05-06 20:02:06 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #1/20 start
2026-05-06 20:02:07 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #2/20 start
2026-05-06 20:02:08 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #3/20 start
2026-05-06 20:02:10 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #4/20 start
2026-05-06 20:02:11 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #5/20 start
2026-05-06 20:02:12 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #6/20 start
2026-05-06 20:02:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #7/20 start
2026-05-06 20:02:15 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #8/20 start
2026-05-06 20:02:16 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #9/20 start
2026-05-06 20:02:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #10/20 start
2026-05-06 20:02:19 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #11/20 start
2026-05-06 20:02:20 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #12/20 start
2026-05-06 20:02:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #13/20 start
2026-05-06 20:02:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #14/20 start
2026-05-06 20:02:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #15/20 start
2026-05-06 20:02:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #16/20 start
2026-05-06 20:02:28 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #17/20 start
2026-05-06 20:02:30 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #18/20 start
2026-05-06 20:02:32 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #19/20 start
2026-05-06 20:02:34 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, eval #20/20 start
2026-05-06 20:02:43 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, starting new job
2026-05-06 20:02:49 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending/unknown 7/2 = ∑9/20, started new job
2026-05-06 20:02:50 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending/unknown 7/3 = ∑10/20, started new job
2026-05-06 20:02:50 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending/unknown 7/6 = ∑13/20, started new job
2026-05-06 20:02:51 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending/unknown 7/8 = ∑15/20, started new job
2026-05-06 20:02:52 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending/unknown 7/9 = ∑16/20, started new job
2026-05-06 20:02:53 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 16 = ∑16/20, started new job
2026-05-06 20:02:54 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 16 = ∑16/20, starting new job
2026-05-06 20:02:56 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending/unknown 18/1 = ∑19/20, started new job
2026-05-06 20:02:56 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending/unknown 18/2 = ∑20/20, started new job
2026-05-06 20:02:58 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending/unknown 18/2 = ∑20/20, waiting for 20 jobs
2026-05-06 20:03:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-07 22:14:26 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 18 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 20 = ∑20/20, job_failed
2026-05-07 22:14:28 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 19 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-07 22:14:29 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 19 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 19 = ∑19/20, waiting for 19 jobs
2026-05-07 22:14:31 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 19 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 19 = ∑19/20, job_failed
2026-05-07 22:14:33 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 20 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-07 22:14:33 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 20 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 18 = ∑18/20, waiting for 18 jobs
2026-05-07 22:14:55 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 20 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 18 = ∑18/20, job_failed
2026-05-07 22:14:57 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 21 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 17 = ∑17/20, waiting for 18 jobs, finished 1 job
2026-05-07 22:14:58 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 21 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 17 = ∑17/20, waiting for 17 jobs
2026-05-07 22:15:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 21 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 17 = ∑17/20, job_failed
2026-05-07 22:15:04 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 22 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 16 = ∑16/20, waiting for 17 jobs, finished 1 job
2026-05-07 22:15:05 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 22 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 16 = ∑16/20, waiting for 16 jobs
2026-05-07 22:15:07 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 22 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 16 = ∑16/20, job_failed
2026-05-07 22:15:09 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 23 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 15 = ∑15/20, waiting for 16 jobs, finished 1 job
2026-05-07 22:15:09 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 23 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 15 = ∑15/20, waiting for 15 jobs
2026-05-07 22:15:21 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 23 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 15 = ∑15/20, job_failed
2026-05-07 22:15:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 24 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 14 = ∑14/20, waiting for 15 jobs, finished 1 job
2026-05-07 22:15:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 24 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 14 = ∑14/20, waiting for 14 jobs
2026-05-07 22:15:33 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 24 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 14 = ∑14/20, job_failed
2026-05-07 22:15:36 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 26 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 12 = ∑12/20, waiting for 14 jobs, finished 2 jobs
2026-05-07 22:15:36 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 26 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 12 = ∑12/20, waiting for 12 jobs
2026-05-07 22:15:39 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 26 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 12 = ∑12/20, job_failed
2026-05-07 22:15:41 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 27 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 11 = ∑11/20, waiting for 12 jobs, finished 1 job
2026-05-07 22:15:42 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 27 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 11 = ∑11/20, waiting for 11 jobs
2026-05-07 22:15:56 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 27 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 11 = ∑11/20, job_failed
2026-05-07 22:15:58 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 10 = ∑10/20, waiting for 11 jobs, finished 1 job
2026-05-07 22:15:59 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, pending 10 = ∑10/20, waiting for 10 jobs
2026-05-07 22:18:04 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, running 10 = ∑10/20, waiting for 10 jobs
2026-05-07 22:38:17 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, running 10 = ∑10/20, new result: VAL_LOSS: 0.000017, PARAMS: 66963.000000
2026-05-07 22:38:38 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 3 done, running 9 = ∑9/20, waiting for 10 jobs, finished 1 job
2026-05-07 22:38:39 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 3 done, running 9 = ∑9/20, waiting for 9 jobs
2026-05-07 22:51:21 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 3 done, running 9 = ∑9/20, new result: VAL_LOSS: 0.621535, PARAMS: 186084.000000
2026-05-07 22:51:40 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 4 done, running 8 = ∑8/20, waiting for 9 jobs, finished 1 job
2026-05-07 22:51:40 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 4 done, running 8 = ∑8/20, waiting for 8 jobs
2026-05-07 23:31:32 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 4 done, running 8 = ∑8/20, new result: VAL_LOSS: 0.000019, PARAMS: 168732.000000
2026-05-07 23:31:55 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 5 done, running 7 = ∑7/20, waiting for 8 jobs, finished 1 job
2026-05-07 23:31:56 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 5 done, running 7 = ∑7/20, waiting for 7 jobs
2026-05-08 00:25:47 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 5 done, running 7 = ∑7/20, new result: VAL_LOSS: 0.693210, PARAMS: 3331322.000000
2026-05-08 00:26:09 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 6 done, running 6 = ∑6/20, waiting for 7 jobs, finished 1 job
2026-05-08 00:26:10 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 6 done, running 6 = ∑6/20, waiting for 6 jobs
2026-05-08 05:06:05 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 6 done, running 6 = ∑6/20, new result: VAL_LOSS: 0.000000, PARAMS: 135242.000000
2026-05-08 05:06:33 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running 5 = ∑5/20, waiting for 6 jobs, finished 1 job
2026-05-08 05:06:34 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running 5 = ∑5/20, waiting for 5 jobs
2026-05-08 10:18:57 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 28 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, timeout 5 = ∑5/20, job_failed
2026-05-08 10:19:04 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, waiting for 5 jobs, finished 5 jobs
2026-05-08 10:19:07 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #1/20
2026-05-08 10:19:08 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #2/20
2026-05-08 10:19:09 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #3/20
2026-05-08 10:19:11 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #4/20
2026-05-08 10:19:12 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #5/20
2026-05-08 10:19:13 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #6/20
2026-05-08 10:19:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #7/20
2026-05-08 10:19:15 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #8/20
2026-05-08 10:19:15 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #9/20
2026-05-08 10:19:16 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #10/20
2026-05-08 10:19:17 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #11/20
2026-05-08 10:19:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #12/20
2026-05-08 10:19:19 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #13/20
2026-05-08 10:19:20 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #14/20
2026-05-08 10:19:20 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #15/20
2026-05-08 10:19:21 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #16/20
2026-05-08 10:19:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #17/20
2026-05-08 10:19:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #18/20
2026-05-08 10:19:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #19/20
2026-05-08 10:19:25 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, getting new HP set #20/20
2026-05-08 10:19:26 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, requested 20 jobs, got 20, 0.94 s/job
2026-05-08 10:19:29 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #1/20 start
2026-05-08 10:19:32 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #2/20 start
2026-05-08 10:19:34 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #3/20 start
2026-05-08 10:19:37 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #4/20 start
2026-05-08 10:19:39 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #5/20 start
2026-05-08 10:19:42 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #6/20 start
2026-05-08 10:19:45 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #7/20 start
2026-05-08 10:19:48 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #8/20 start
2026-05-08 10:19:50 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #9/20 start
2026-05-08 10:19:53 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #10/20 start
2026-05-08 10:19:55 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #11/20 start
2026-05-08 10:19:58 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #12/20 start
2026-05-08 10:20:00 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #13/20 start
2026-05-08 10:20:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #14/20 start
2026-05-08 10:20:05 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #15/20 start
2026-05-08 10:20:07 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #16/20 start
2026-05-08 10:20:10 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #17/20 start
2026-05-08 10:20:13 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #18/20 start
2026-05-08 10:20:15 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #19/20 start
2026-05-08 10:20:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, eval #20/20 start
2026-05-08 10:20:41 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, starting new job
2026-05-08 10:21:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, pending 16 = ∑16/20, started new job
2026-05-08 10:21:06 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, pending/running 15/1 = ∑16/20, started new job
2026-05-08 10:21:12 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, pending/running 15/1 = ∑16/20, starting new job
2026-05-08 10:21:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, pending/running 19/1 = ∑20/20, started new job
2026-05-08 10:21:21 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, pending/running 19/1 = ∑20/20, waiting for 20 jobs
2026-05-08 10:21:33 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, pending/running 18/2 = ∑20/20, waiting for 20 jobs
2026-05-08 10:22:39 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 33 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, pending/running 18/2 = ∑20/20, job_failed
2026-05-08 10:22:42 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 34 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, pending/running 18/1 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-08 10:22:43 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 34 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, pending/running 18/1 = ∑19/20, waiting for 19 jobs
2026-05-08 10:23:52 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 34 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 8/11 = ∑19/20, waiting for 19 jobs
2026-05-08 10:24:05 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 34 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 8/11 = ∑19/20, job_failed
2026-05-08 10:24:09 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 36 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 6/11 = ∑17/20, waiting for 19 jobs, finished 2 jobs
2026-05-08 10:24:10 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 36 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 6/11 = ∑17/20, waiting for 17 jobs
2026-05-08 10:24:48 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 36 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 6/11 = ∑17/20, job_failed
2026-05-08 10:24:51 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 37 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 5/11 = ∑16/20, waiting for 17 jobs, finished 1 job
2026-05-08 10:24:51 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 37 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 5/11 = ∑16/20, waiting for 16 jobs
2026-05-08 10:25:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 37 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 5/11 = ∑16/20, job_failed
2026-05-08 10:25:06 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 39 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 5/9 = ∑14/20, waiting for 16 jobs, finished 2 jobs
2026-05-08 10:25:08 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 39 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 5/9 = ∑14/20, waiting for 14 jobs
2026-05-08 10:25:20 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 39 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 5/9 = ∑14/20, job_failed
2026-05-08 10:25:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 40 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 4/9 = ∑13/20, waiting for 14 jobs, finished 1 job
2026-05-08 10:25:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 40 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 4/9 = ∑13/20, waiting for 13 jobs
2026-05-08 10:25:31 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 40 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 4/9 = ∑13/20, job_failed
2026-05-08 10:25:33 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 41 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 4/8 = ∑12/20, waiting for 13 jobs, finished 1 job
2026-05-08 10:25:34 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 41 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 4/8 = ∑12/20, waiting for 12 jobs
2026-05-08 10:26:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 41 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 4/8 = ∑12/20, job_failed
2026-05-08 10:26:04 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 42 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 4/7 = ∑11/20, waiting for 12 jobs, finished 1 job
2026-05-08 10:26:05 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 42 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 4/7 = ∑11/20, waiting for 11 jobs
2026-05-08 10:26:31 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 42 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 8/3 = ∑11/20, waiting for 11 jobs
2026-05-08 10:29:44 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 42 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 8/3 = ∑11/20, job_failed
2026-05-08 10:29:47 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 43 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 8/2 = ∑10/20, waiting for 11 jobs, finished 1 job
2026-05-08 10:29:48 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 43 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 8/2 = ∑10/20, waiting for 10 jobs
2026-05-08 10:30:43 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 43 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 8/2 = ∑10/20, job_failed
2026-05-08 10:30:46 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 8/1 = ∑9/20, waiting for 10 jobs, finished 1 job
2026-05-08 10:30:47 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running/pending 8/1 = ∑9/20, waiting for 9 jobs
2026-05-08 10:31:48 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running 9 = ∑9/20, waiting for 9 jobs
2026-05-08 12:31:17 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running 9 = ∑9/20, new result: VAL_LOSS: 0.000003, PARAMS: 948268.000000
2026-05-08 12:31:52 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running 8 = ∑8/20, waiting for 9 jobs, finished 1 job
2026-05-08 12:31:53 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running 8 = ∑8/20, waiting for 8 jobs
2026-05-08 13:56:54 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running 8 = ∑8/20, new result: VAL_LOSS: 0.000128, PARAMS: 739020.000000
2026-05-08 13:57:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 9 done, running 7 = ∑7/20, waiting for 8 jobs, finished 1 job
2026-05-08 13:57:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 9 done, running 7 = ∑7/20, waiting for 7 jobs
2026-05-08 17:50:26 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 9 done, running 7 = ∑7/20, new result: VAL_LOSS: 0.679808, PARAMS: 5835188.000000
2026-05-08 17:51:00 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running 6 = ∑6/20, waiting for 7 jobs, finished 1 job
2026-05-08 17:51:01 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running 6 = ∑6/20, waiting for 6 jobs
2026-05-08 22:32:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, timeout 6 = ∑6/20, waiting for 6 jobs
2026-05-08 22:32:43 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 44 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, timeout 6 = ∑6/20, job_failed
2026-05-08 22:32:53 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, waiting for 6 jobs, finished 6 jobs
2026-05-08 22:32:56 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #1/20
2026-05-08 22:32:58 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #2/20
2026-05-08 22:32:59 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #3/20
2026-05-08 22:33:00 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #4/20
2026-05-08 22:33:01 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #5/20
2026-05-08 22:33:02 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #6/20
2026-05-08 22:33:03 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #7/20
2026-05-08 22:33:04 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #8/20
2026-05-08 22:33:05 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #9/20
2026-05-08 22:33:06 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #10/20
2026-05-08 22:33:07 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #11/20
2026-05-08 22:33:08 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #12/20
2026-05-08 22:33:09 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #13/20
2026-05-08 22:33:10 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #14/20
2026-05-08 22:33:11 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #15/20
2026-05-08 22:33:12 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #16/20
2026-05-08 22:33:13 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #17/20
2026-05-08 22:33:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #18/20
2026-05-08 22:33:15 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #19/20
2026-05-08 22:33:16 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, getting new HP set #20/20
2026-05-08 22:33:17 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, requested 20 jobs, got 20, 1.07 s/job
2026-05-08 22:33:21 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #1/20 start
2026-05-08 22:33:24 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #2/20 start
2026-05-08 22:33:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #3/20 start
2026-05-08 22:33:30 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #4/20 start
2026-05-08 22:33:33 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #5/20 start
2026-05-08 22:33:37 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #6/20 start
2026-05-08 22:33:40 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #7/20 start
2026-05-08 22:33:43 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #8/20 start
2026-05-08 22:33:46 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #9/20 start
2026-05-08 22:33:49 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #10/20 start
2026-05-08 22:33:52 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #11/20 start
2026-05-08 22:33:54 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #12/20 start
2026-05-08 22:33:57 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #13/20 start
2026-05-08 22:34:00 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #14/20 start
2026-05-08 22:34:04 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #15/20 start
2026-05-08 22:34:07 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #16/20 start
2026-05-08 22:34:11 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #17/20 start
2026-05-08 22:34:14 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #18/20 start
2026-05-08 22:34:17 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #19/20 start
2026-05-08 22:34:20 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, eval #20/20 start
2026-05-08 22:34:45 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, starting new job
2026-05-08 22:34:59 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 4 = ∑4/20, starting new job
2026-05-08 22:35:10 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 14 = ∑14/20, started new job
2026-05-08 22:35:10 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 15 = ∑15/20, started new job
2026-05-08 22:35:15 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 16 = ∑16/20, started new job
2026-05-08 22:35:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 16 = ∑16/20, starting new job
2026-05-08 22:35:29 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending/unknown 19/1 = ∑20/20, started new job
2026-05-08 22:35:33 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-08 22:36:37 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 1/19 = ∑20/20, waiting for 20 jobs
2026-05-08 22:37:39 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 50 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 1/19 = ∑20/20, job_failed
2026-05-08 22:37:42 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 51 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-08 22:37:43 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 51 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 19 = ∑19/20, waiting for 19 jobs
2026-05-08 22:37:48 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 51 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 1/18 = ∑19/20, waiting for 19 jobs
2026-05-08 22:38:08 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 51 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 1/18 = ∑19/20, job_failed
2026-05-08 22:38:11 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 52 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-08 22:38:12 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 52 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 18 = ∑18/20, waiting for 18 jobs
2026-05-08 22:40:05 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 52 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 18 = ∑18/20, job_failed
2026-05-08 22:40:08 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 53 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, pending 17 = ∑17/20, waiting for 18 jobs, finished 1 job
2026-05-08 22:40:09 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 53 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/15 = ∑17/20, waiting for 17 jobs
2026-05-08 22:41:05 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 53 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/15 = ∑17/20, job_failed
2026-05-08 22:41:08 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 54 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/14 = ∑16/20, waiting for 17 jobs, finished 1 job
2026-05-08 22:41:09 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 54 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/14 = ∑16/20, waiting for 16 jobs
2026-05-08 22:41:35 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 54 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/14 = ∑16/20, job_failed
2026-05-08 22:41:38 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 55 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/13 = ∑15/20, waiting for 16 jobs, finished 1 job
2026-05-08 22:41:39 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 55 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/13 = ∑15/20, waiting for 15 jobs
2026-05-08 22:41:48 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 55 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/13 = ∑15/20, job_failed
2026-05-08 22:41:52 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 56 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/12 = ∑14/20, waiting for 15 jobs, finished 1 job
2026-05-08 22:41:53 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 56 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/12 = ∑14/20, waiting for 14 jobs
2026-05-08 22:42:19 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 56 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/12 = ∑14/20, job_failed
2026-05-08 22:42:22 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 57 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/11 = ∑13/20, waiting for 14 jobs, finished 1 job
2026-05-08 22:42:23 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 57 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/11 = ∑13/20, waiting for 13 jobs
2026-05-08 22:42:49 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 57 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/11 = ∑13/20, job_failed
2026-05-08 22:42:52 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 58 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/10 = ∑12/20, waiting for 13 jobs, finished 1 job
2026-05-08 22:42:53 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 58 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/10 = ∑12/20, waiting for 12 jobs
2026-05-08 22:43:07 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 58 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/10 = ∑12/20, job_failed
2026-05-08 22:43:10 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 59 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/9 = ∑11/20, waiting for 12 jobs, finished 1 job
2026-05-08 22:43:11 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 59 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/9 = ∑11/20, waiting for 11 jobs
2026-05-08 22:43:36 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 59 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/9 = ∑11/20, job_failed
2026-05-08 22:43:40 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/8 = ∑10/20, waiting for 11 jobs, finished 1 job
2026-05-08 22:43:40 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running/pending 2/8 = ∑10/20, waiting for 10 jobs
2026-05-08 22:44:54 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running 10 = ∑10/20, waiting for 10 jobs
2026-05-08 22:58:52 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 10 done, running 10 = ∑10/20, new result: VAL_LOSS: 0.375029, PARAMS: 1852760.000000
2026-05-08 22:59:26 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 11 done, running 9 = ∑9/20, waiting for 10 jobs, finished 1 job
2026-05-08 22:59:27 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 11 done, running 9 = ∑9/20, waiting for 9 jobs
2026-05-09 00:15:32 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 11 done, running 9 = ∑9/20, new result: VAL_LOSS: 0.693464, PARAMS: 688000.000000
2026-05-09 00:16:05 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 12 done, running 8 = ∑8/20, waiting for 9 jobs, finished 1 job
2026-05-09 00:16:06 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 12 done, running 8 = ∑8/20, waiting for 8 jobs
2026-05-09 03:17:36 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 12 done, running 8 = ∑8/20, new result: VAL_LOSS: 0.000005, PARAMS: 3384127.000000
2026-05-09 03:18:18 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 13 done, running 7 = ∑7/20, waiting for 8 jobs, finished 1 job
2026-05-09 03:18:19 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 13 done, running 7 = ∑7/20, waiting for 7 jobs
2026-05-09 08:47:17 (9f5e8dc9-b4c8-415d-9f90-aa840d25c57b): SOBOL, failed: 60 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 13 done, running 7 = ∑7/20, new result: VAL_LOSS: 0.083276, PARAMS: 2271332.000000