⚠ Job 3531618 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531618/3531618_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531618/3531618_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:17:13,900) - Starting with JobEnvironment(job_id=3531618, hostname=c56, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:17:13,903) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531618/3531618_submitted.pkl
Trial-Index: 10
slurmstepd: error: *** JOB 3531618 ON c56 CANCELLED AT 2026-05-11T10:17:15 DUE TO TIME LIMIT ***
⚠ Job 3531621 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531621/3531621_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531621/3531621_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:27:16,820) - Starting with JobEnvironment(job_id=3531621, hostname=c26, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:27:16,822) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531621/3531621_submitted.pkl
Trial-Index: 3
slurmstepd: error: *** JOB 3531621 ON c26 CANCELLED AT 2026-05-11T10:27:15 DUE TO TIME LIMIT ***
⚠ Job 3531623 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531623/3531623_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531623/3531623_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:27:46,208) - Starting with JobEnvironment(job_id=3531623, hostname=c23, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:27:46,210) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531623/3531623_submitted.pkl
Trial-Index: 5
slurmstepd: error: *** JOB 3531623 ON c23 CANCELLED AT 2026-05-11T10:27:45 DUE TO TIME LIMIT ***
⚠ Job 3531619 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531619/3531619_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531619/3531619_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:25:16,070) - Starting with JobEnvironment(job_id=3531619, hostname=c19, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:25:16,072) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531619/3531619_submitted.pkl
Trial-Index: 11
slurmstepd: error: *** JOB 3531619 ON c19 CANCELLED AT 2026-05-11T10:25:15 DUE TO TIME LIMIT ***
⚠ Job 3531620 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531620/3531620_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531620/3531620_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:26:16,606) - Starting with JobEnvironment(job_id=3531620, hostname=c23, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:26:16,608) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531620/3531620_submitted.pkl
Trial-Index: 4
slurmstepd: error: *** JOB 3531620 ON c23 CANCELLED AT 2026-05-11T10:26:15 DUE TO TIME LIMIT ***
⚠ Job 3531622 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531622/3531622_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531622/3531622_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:27:46,088) - Starting with JobEnvironment(job_id=3531622, hostname=c36, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:27:46,090) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531622/3531622_submitted.pkl
Trial-Index: 6
slurmstepd: error: *** JOB 3531622 ON c36 CANCELLED AT 2026-05-11T10:27:45 DUE TO TIME LIMIT ***
⚠ Job 3531626 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531626/3531626_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531626/3531626_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:37:48,597) - Starting with JobEnvironment(job_id=3531626, hostname=c131, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:37:48,599) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531626/3531626_submitted.pkl
Trial-Index: 7
slurmstepd: error: *** JOB 3531626 ON c131 CANCELLED AT 2026-05-11T10:38:15 DUE TO TIME LIMIT ***
⚠ Job 3531625 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531625/3531625_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531625/3531625_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:37:18,489) - Starting with JobEnvironment(job_id=3531625, hostname=c26, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:37:18,491) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531625/3531625_submitted.pkl
Trial-Index: 8
slurmstepd: error: *** JOB 3531625 ON c26 CANCELLED AT 2026-05-11T10:37:45 DUE TO TIME LIMIT ***
⚠ Job 3531624 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531624/3531624_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531624/3531624_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:37:18,490) - Starting with JobEnvironment(job_id=3531624, hostname=c26, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:37:18,491) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531624/3531624_submitted.pkl
Trial-Index: 14
slurmstepd: error: *** JOB 3531624 ON c26 CANCELLED AT 2026-05-11T10:37:45 DUE TO TIME LIMIT ***
⚠ Job 3531628 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531628/3531628_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531628/3531628_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:43:19,896) - Starting with JobEnvironment(job_id=3531628, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:43:19,898) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531628/3531628_submitted.pkl
Trial-Index: 13
slurmstepd: error: *** JOB 3531628 ON c27 CANCELLED AT 2026-05-11T10:43:45 DUE TO TIME LIMIT ***
⚠ Job 3531629 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531629/3531629_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531629/3531629_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:43:50,860) - Starting with JobEnvironment(job_id=3531629, hostname=c149, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:43:50,862) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531629/3531629_submitted.pkl
Trial-Index: 2
slurmstepd: error: *** JOB 3531629 ON c149 CANCELLED AT 2026-05-11T10:44:15 DUE TO TIME LIMIT ***
⚠ Job 3531631 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531631/3531631_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531631/3531631_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:44:50,620) - Starting with JobEnvironment(job_id=3531631, hostname=c59, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:44:50,622) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531631/3531631_submitted.pkl
Trial-Index: 9
slurmstepd: error: *** JOB 3531631 ON c59 CANCELLED AT 2026-05-11T10:45:15 DUE TO TIME LIMIT ***
⚠ Job 3531632 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531632/3531632_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531632/3531632_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:44:50,620) - Starting with JobEnvironment(job_id=3531632, hostname=c59, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:44:50,622) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531632/3531632_submitted.pkl
Trial-Index: 1
slurmstepd: error: *** JOB 3531632 ON c59 CANCELLED AT 2026-05-11T10:45:15 DUE TO TIME LIMIT ***
⚠ Job 3531635 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531635/3531635_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531635/3531635_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:46:50,491) - Starting with JobEnvironment(job_id=3531635, hostname=c89, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:46:50,493) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531635/3531635_submitted.pkl
Trial-Index: 18
slurmstepd: error: *** JOB 3531635 ON c89 CANCELLED AT 2026-05-11T10:47:15 DUE TO TIME LIMIT ***
⚠ Job 3531634 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531634/3531634_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531634/3531634_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:46:20,052) - Starting with JobEnvironment(job_id=3531634, hostname=c85, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:46:20,054) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531634/3531634_submitted.pkl
Trial-Index: 17
slurmstepd: error: *** JOB 3531634 ON c85 CANCELLED AT 2026-05-11T10:46:45 DUE TO TIME LIMIT ***
⚠ Job 3531636 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531636/3531636_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531636/3531636_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:47:50,921) - Starting with JobEnvironment(job_id=3531636, hostname=c147, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:47:50,927) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531636/3531636_submitted.pkl
Trial-Index: 19
slurmstepd: error: *** JOB 3531636 ON c147 CANCELLED AT 2026-05-11T10:48:15 DUE TO TIME LIMIT ***
⚠ Job 3531633 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531633/3531633_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531633/3531633_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:45:49,795) - Starting with JobEnvironment(job_id=3531633, hostname=c96, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:45:49,797) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531633/3531633_submitted.pkl
Trial-Index: 12
slurmstepd: error: *** JOB 3531633 ON c96 CANCELLED AT 2026-05-11T10:46:15 DUE TO TIME LIMIT ***
⚠ Job 3533156 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533156/3533156_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533156/3533156_0_log.out
----------------------------------------
submitit INFO (2026-05-12 00:32:43,388) - Starting with JobEnvironment(job_id=3533156, hostname=c109, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 00:32:43,390) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533156/3533156_submitted.pkl
Trial-Index: 20
slurmstepd: error: *** JOB 3533156 ON c109 CANCELLED AT 2026-05-12T12:33:05 DUE TO TIME LIMIT ***
⚠ Job 3533160 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533160/3533160_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533160/3533160_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:36:12,532) - Starting with JobEnvironment(job_id=3533160, hostname=c106, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:36:12,534) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533160/3533160_submitted.pkl
Trial-Index: 34
slurmstepd: error: *** JOB 3533160 ON c106 CANCELLED AT 2026-05-12T15:36:11 DUE TO TIME LIMIT ***
⚠ Job 3533161 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533161/3533161_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533161/3533161_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:36:42,457) - Starting with JobEnvironment(job_id=3533161, hostname=c148, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:36:42,458) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533161/3533161_submitted.pkl
Trial-Index: 21
slurmstepd: error: *** JOB 3533161 ON c148 CANCELLED AT 2026-05-12T15:36:41 DUE TO TIME LIMIT ***
⚠ Job 3533159 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533159/3533159_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533159/3533159_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:36:12,532) - Starting with JobEnvironment(job_id=3533159, hostname=c106, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:36:12,534) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533159/3533159_submitted.pkl
Trial-Index: 29
slurmstepd: error: *** JOB 3533159 ON c106 CANCELLED AT 2026-05-12T15:36:11 DUE TO TIME LIMIT ***
⚠ Job 3533157 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533157/3533157_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533157/3533157_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:33:11,102) - Starting with JobEnvironment(job_id=3533157, hostname=c94, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:33:11,104) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533157/3533157_submitted.pkl
Trial-Index: 30
slurmstepd: error: *** JOB 3533157 ON c94 CANCELLED AT 2026-05-12T15:33:11 DUE TO TIME LIMIT ***
⚠ Job 3533164 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533164/3533164_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533164/3533164_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:41:45,326) - Starting with JobEnvironment(job_id=3533164, hostname=c100, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:41:45,329) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533164/3533164_submitted.pkl
Trial-Index: 25
slurmstepd: error: *** JOB 3533164 ON c100 CANCELLED AT 2026-05-12T15:42:11 DUE TO TIME LIMIT ***
⚠ Job 3533171 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533171/3533171_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533171/3533171_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:45:47,758) - Starting with JobEnvironment(job_id=3533171, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:45:47,764) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533171/3533171_submitted.pkl
Trial-Index: 28
slurmstepd: error: *** JOB 3533171 ON c108 CANCELLED AT 2026-05-12T15:46:11 DUE TO TIME LIMIT ***
⚠ Job 3533166 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533166/3533166_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533166/3533166_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:44:17,191) - Starting with JobEnvironment(job_id=3533166, hostname=c65, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:44:17,197) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533166/3533166_submitted.pkl
Trial-Index: 26
slurmstepd: error: *** JOB 3533166 ON c65 CANCELLED AT 2026-05-12T15:44:41 DUE TO TIME LIMIT ***
⚠ Job 3533167 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533167/3533167_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533167/3533167_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:44:47,385) - Starting with JobEnvironment(job_id=3533167, hostname=c86, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:44:47,388) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533167/3533167_submitted.pkl
Trial-Index: 35
slurmstepd: error: *** JOB 3533167 ON c86 CANCELLED AT 2026-05-12T15:45:11 DUE TO TIME LIMIT ***
⚠ Job 3533173 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533173/3533173_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533173/3533173_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:46:18,457) - Starting with JobEnvironment(job_id=3533173, hostname=c143, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:46:18,463) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533173/3533173_submitted.pkl
Trial-Index: 38
slurmstepd: error: *** JOB 3533173 ON c143 CANCELLED AT 2026-05-12T15:46:41 DUE TO TIME LIMIT ***
⚠ Job 3533162 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533162/3533162_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533162/3533162_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:39:43,984) - Starting with JobEnvironment(job_id=3533162, hostname=c104, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:39:43,986) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533162/3533162_submitted.pkl
Trial-Index: 33
slurmstepd: error: *** JOB 3533162 ON c104 CANCELLED AT 2026-05-12T15:40:11 DUE TO TIME LIMIT ***
⚠ Job 3533165 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533165/3533165_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533165/3533165_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:43:46,391) - Starting with JobEnvironment(job_id=3533165, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:43:46,395) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533165/3533165_submitted.pkl
Trial-Index: 22
slurmstepd: error: *** JOB 3533165 ON c99 CANCELLED AT 2026-05-12T15:44:11 DUE TO TIME LIMIT ***
⚠ Job 3533168 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533168/3533168_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533168/3533168_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:45:18,025) - Starting with JobEnvironment(job_id=3533168, hostname=c117, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:45:18,029) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533168/3533168_submitted.pkl
Trial-Index: 32
slurmstepd: error: *** JOB 3533168 ON c117 CANCELLED AT 2026-05-12T15:45:41 DUE TO TIME LIMIT ***
⚠ Job 3533174 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533174/3533174_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533174/3533174_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:46:18,355) - Starting with JobEnvironment(job_id=3533174, hostname=c117, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:46:18,361) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533174/3533174_submitted.pkl
Trial-Index: 39
slurmstepd: error: *** JOB 3533174 ON c117 CANCELLED AT 2026-05-12T15:46:41 DUE TO TIME LIMIT ***
⚠ Job 3533172 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533172/3533172_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533172/3533172_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:45:48,445) - Starting with JobEnvironment(job_id=3533172, hostname=c34, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:45:48,451) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533172/3533172_submitted.pkl
Trial-Index: 37
slurmstepd: error: *** JOB 3533172 ON c34 CANCELLED AT 2026-05-12T15:46:11 DUE TO TIME LIMIT ***
⚠ Job 3533163 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533163/3533163_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533163/3533163_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:42:15,524) - Starting with JobEnvironment(job_id=3533163, hostname=c150, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:42:15,526) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533163/3533163_submitted.pkl
Trial-Index: 24
slurmstepd: error: *** JOB 3533163 ON c150 CANCELLED AT 2026-05-12T15:42:41 DUE TO TIME LIMIT ***
⚠ Job 3533169 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533169/3533169_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533169/3533169_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:45:17,900) - Starting with JobEnvironment(job_id=3533169, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:45:17,904) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533169/3533169_submitted.pkl
Trial-Index: 31
slurmstepd: error: *** JOB 3533169 ON c108 CANCELLED AT 2026-05-12T15:45:41 DUE TO TIME LIMIT ***
⚠ Job 3539660 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539660/3539660_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539660/3539660_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:50,816) - Starting with JobEnvironment(job_id=3539660, hostname=c59, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:50,819) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539660/3539660_submitted.pkl
Trial-Index: 41
slurmstepd: error: *** JOB 3539660 ON c59 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
⚠ Job 3539659 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539659/3539659_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539659/3539659_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:50,958) - Starting with JobEnvironment(job_id=3539659, hostname=c70, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:50,960) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539659/3539659_submitted.pkl
Trial-Index: 53
slurmstepd: error: *** JOB 3539659 ON c70 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
⚠ Job 3539658 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539658/3539658_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539658/3539658_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:51,974) - Starting with JobEnvironment(job_id=3539658, hostname=c71, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:51,976) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539658/3539658_submitted.pkl
Trial-Index: 49
slurmstepd: error: *** JOB 3539658 ON c71 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
⚠ Job 3539656 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539656/3539656_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539656/3539656_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:50,796) - Starting with JobEnvironment(job_id=3539656, hostname=c93, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:50,799) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539656/3539656_submitted.pkl
Trial-Index: 46
slurmstepd: error: *** JOB 3539656 ON c93 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
⚠ Job 3539655 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539655/3539655_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539655/3539655_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:51,836) - Starting with JobEnvironment(job_id=3539655, hostname=c96, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:51,839) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539655/3539655_submitted.pkl
Trial-Index: 43
slurmstepd: error: *** JOB 3539655 ON c96 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
⚠ Job 3539663 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539663/3539663_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539663/3539663_0_log.out
----------------------------------------
submitit INFO (2026-05-13 01:26:05,361) - Starting with JobEnvironment(job_id=3539663, hostname=c57, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 01:26:05,363) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539663/3539663_submitted.pkl
Trial-Index: 50
slurmstepd: error: *** JOB 3539663 ON c57 CANCELLED AT 2026-05-13T13:26:14 DUE TO TIME LIMIT ***
⚠ Job 3539661 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539661/3539661_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539661/3539661_0_log.out
----------------------------------------
submitit INFO (2026-05-13 01:26:05,630) - Starting with JobEnvironment(job_id=3539661, hostname=c60, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 01:26:05,633) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539661/3539661_submitted.pkl
Trial-Index: 45
slurmstepd: error: *** JOB 3539661 ON c60 CANCELLED AT 2026-05-13T13:26:14 DUE TO TIME LIMIT ***
⚠ Job 3539664 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539664/3539664_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539664/3539664_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:08:21,457) - Starting with JobEnvironment(job_id=3539664, hostname=c34, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:08:21,461) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539664/3539664_submitted.pkl
Trial-Index: 42
slurmstepd: error: *** JOB 3539664 ON c34 CANCELLED AT 2026-05-13T17:08:28 DUE TO TIME LIMIT ***
⚠ Job 3539665 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539665/3539665_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539665/3539665_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:18:32,340) - Starting with JobEnvironment(job_id=3539665, hostname=c147, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:18:32,342) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539665/3539665_submitted.pkl
Trial-Index: 44
slurmstepd: error: *** JOB 3539665 ON c147 CANCELLED AT 2026-05-13T17:19:00 DUE TO TIME LIMIT ***
⚠ Job 3539667 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539667/3539667_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539667/3539667_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:23:37,852) - Starting with JobEnvironment(job_id=3539667, hostname=c46, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:23:37,854) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539667/3539667_submitted.pkl
Trial-Index: 52
slurmstepd: error: *** JOB 3539667 ON c46 CANCELLED AT 2026-05-13T17:24:00 DUE TO TIME LIMIT ***
⚠ Job 3539666 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539666/3539666_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539666/3539666_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:23:37,225) - Starting with JobEnvironment(job_id=3539666, hostname=c111, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:23:37,227) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539666/3539666_submitted.pkl
Trial-Index: 48
slurmstepd: error: *** JOB 3539666 ON c111 CANCELLED AT 2026-05-13T17:24:00 DUE TO TIME LIMIT ***
⚠ Job 3539668 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539668/3539668_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539668/3539668_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:23:37,852) - Starting with JobEnvironment(job_id=3539668, hostname=c46, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:23:37,854) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539668/3539668_submitted.pkl
Trial-Index: 54
slurmstepd: error: *** JOB 3539668 ON c46 CANCELLED AT 2026-05-13T17:24:00 DUE TO TIME LIMIT ***
⚠ Job 3539669 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539669/3539669_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539669/3539669_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:34:48,508) - Starting with JobEnvironment(job_id=3539669, hostname=c55, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:34:48,511) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539669/3539669_submitted.pkl
Trial-Index: 55
slurmstepd: error: *** JOB 3539669 ON c55 CANCELLED AT 2026-05-13T17:35:00 DUE TO TIME LIMIT ***
⚠ Job 3539677 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539677/3539677_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539677/3539677_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:38:52,569) - Starting with JobEnvironment(job_id=3539677, hostname=c73, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:38:52,571) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539677/3539677_submitted.pkl
Trial-Index: 57
slurmstepd: error: *** JOB 3539677 ON c73 CANCELLED AT 2026-05-13T17:39:01 DUE TO TIME LIMIT ***
⚠ Job 3539678 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539678/3539678_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539678/3539678_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:41:25,346) - Starting with JobEnvironment(job_id=3539678, hostname=c107, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:41:25,348) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539678/3539678_submitted.pkl
Trial-Index: 59
slurmstepd: error: *** JOB 3539678 ON c107 CANCELLED AT 2026-05-13T17:41:31 DUE TO TIME LIMIT ***
⚠ Job 3539679 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539679/3539679_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539679/3539679_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:43:26,886) - Starting with JobEnvironment(job_id=3539679, hostname=c15, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:43:26,889) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539679/3539679_submitted.pkl
Trial-Index: 58
slurmstepd: error: *** JOB 3539679 ON c15 CANCELLED AT 2026-05-13T17:43:31 DUE TO TIME LIMIT ***
⚠ Job 3543791 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543791/3543791_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543791/3543791_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:57:27,796) - Starting with JobEnvironment(job_id=3543791, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:57:27,799) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543791/3543791_submitted.pkl
Trial-Index: 64
slurmstepd: error: *** JOB 3543791 ON c99 CANCELLED AT 2026-05-14T05:57:28 DUE TO TIME LIMIT ***
⚠ Job 3543777 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543777/3543777_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543777/3543777_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:55:25,401) - Starting with JobEnvironment(job_id=3543777, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:55:25,403) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543777/3543777_submitted.pkl
Trial-Index: 69
slurmstepd: error: *** JOB 3543777 ON c142 CANCELLED AT 2026-05-14T05:55:28 DUE TO TIME LIMIT ***
⚠ Job 3543778 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543778/3543778_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543778/3543778_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:55:25,378) - Starting with JobEnvironment(job_id=3543778, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:55:25,380) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543778/3543778_submitted.pkl
Trial-Index: 75
slurmstepd: error: *** JOB 3543778 ON c108 CANCELLED AT 2026-05-14T05:55:28 DUE TO TIME LIMIT ***
⚠ Job 3543775 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543775/3543775_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543775/3543775_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:54:55,674) - Starting with JobEnvironment(job_id=3543775, hostname=c42, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:54:55,676) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543775/3543775_submitted.pkl
Trial-Index: 65
slurmstepd: error: *** JOB 3543775 ON c42 CANCELLED AT 2026-05-14T05:54:58 DUE TO TIME LIMIT ***
⚠ Job 3543779 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543779/3543779_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543779/3543779_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:55:56,582) - Starting with JobEnvironment(job_id=3543779, hostname=c145, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:55:56,584) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543779/3543779_submitted.pkl
Trial-Index: 73
slurmstepd: error: *** JOB 3543779 ON c145 CANCELLED AT 2026-05-14T05:55:58 DUE TO TIME LIMIT ***
⚠ Job 3543776 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543776/3543776_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543776/3543776_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:54:55,674) - Starting with JobEnvironment(job_id=3543776, hostname=c40, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:54:55,676) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543776/3543776_submitted.pkl
Trial-Index: 60
slurmstepd: error: *** JOB 3543776 ON c40 CANCELLED AT 2026-05-14T05:54:58 DUE TO TIME LIMIT ***
⚠ Job 3543793 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543793/3543793_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543793/3543793_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:57:59,080) - Starting with JobEnvironment(job_id=3543793, hostname=c90, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:57:59,083) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543793/3543793_submitted.pkl
Trial-Index: 74
slurmstepd: error: *** JOB 3543793 ON c90 CANCELLED AT 2026-05-14T05:57:58 DUE TO TIME LIMIT ***
⚠ Job 3543773 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543773/3543773_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543773/3543773_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:54:55,086) - Starting with JobEnvironment(job_id=3543773, hostname=c117, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:54:55,088) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543773/3543773_submitted.pkl
Trial-Index: 66
slurmstepd: error: *** JOB 3543773 ON c117 CANCELLED AT 2026-05-14T05:54:58 DUE TO TIME LIMIT ***
⚠ Job 3543794 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543794/3543794_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543794/3543794_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:58:29,223) - Starting with JobEnvironment(job_id=3543794, hostname=c71, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:58:29,225) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543794/3543794_submitted.pkl
Trial-Index: 68
slurmstepd: error: *** JOB 3543794 ON c71 CANCELLED AT 2026-05-14T05:58:28 DUE TO TIME LIMIT ***
⚠ Job 3543789 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543789/3543789_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543789/3543789_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:56:27,018) - Starting with JobEnvironment(job_id=3543789, hostname=c89, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:56:27,019) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543789/3543789_submitted.pkl
Trial-Index: 62
slurmstepd: error: *** JOB 3543789 ON c89 CANCELLED AT 2026-05-14T05:56:28 DUE TO TIME LIMIT ***
⚠ Job 3543780 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543780/3543780_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543780/3543780_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:55:56,192) - Starting with JobEnvironment(job_id=3543780, hostname=c111, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:55:56,194) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543780/3543780_submitted.pkl
Trial-Index: 71
slurmstepd: error: *** JOB 3543780 ON c111 CANCELLED AT 2026-05-14T05:55:58 DUE TO TIME LIMIT ***
⚠ Job 3543799 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543799/3543799_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543799/3543799_0_log.out
----------------------------------------
submitit INFO (2026-05-13 18:00:00,947) - Starting with JobEnvironment(job_id=3543799, hostname=c94, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 18:00:00,950) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543799/3543799_submitted.pkl
Trial-Index: 77
slurmstepd: error: *** JOB 3543799 ON c94 CANCELLED AT 2026-05-14T06:00:28 DUE TO TIME LIMIT ***
⚠ Job 3543800 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543800/3543800_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543800/3543800_0_log.out
----------------------------------------
submitit INFO (2026-05-13 18:00:00,927) - Starting with JobEnvironment(job_id=3543800, hostname=c71, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 18:00:00,931) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543800/3543800_submitted.pkl
Trial-Index: 76
slurmstepd: error: *** JOB 3543800 ON c71 CANCELLED AT 2026-05-14T06:00:28 DUE TO TIME LIMIT ***
⚠ Job 3543802 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543802/3543802_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543802/3543802_0_log.out
----------------------------------------
submitit INFO (2026-05-13 18:03:03,040) - Starting with JobEnvironment(job_id=3543802, hostname=c114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 18:03:03,042) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543802/3543802_submitted.pkl
Trial-Index: 78
slurmstepd: error: *** JOB 3543802 ON c114 CANCELLED AT 2026-05-14T06:03:28 DUE TO TIME LIMIT ***
⚠ Job 3545415 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545415/3545415_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545415/3545415_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:13:10,352) - Starting with JobEnvironment(job_id=3545415, hostname=c114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:13:10,354) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545415/3545415_submitted.pkl
Trial-Index: 82
slurmstepd: error: *** JOB 3545415 ON c114 CANCELLED AT 2026-05-14T18:13:09 DUE TO TIME LIMIT ***
⚠ Job 3545418 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545418/3545418_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545418/3545418_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:17:44,523) - Starting with JobEnvironment(job_id=3545418, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:17:44,525) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545418/3545418_submitted.pkl
Trial-Index: 93
slurmstepd: error: *** JOB 3545418 ON c136 CANCELLED AT 2026-05-14T18:18:09 DUE TO TIME LIMIT ***
⚠ Job 3545416 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545416/3545416_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545416/3545416_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:13:40,778) - Starting with JobEnvironment(job_id=3545416, hostname=c118, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:13:40,780) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545416/3545416_submitted.pkl
Trial-Index: 92
slurmstepd: error: *** JOB 3545416 ON c118 CANCELLED AT 2026-05-14T18:13:40 DUE TO TIME LIMIT ***
⚠ Job 3545420 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545420/3545420_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545420/3545420_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:21:16,042) - Starting with JobEnvironment(job_id=3545420, hostname=c120, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:21:16,044) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545420/3545420_submitted.pkl
Trial-Index: 85
slurmstepd: error: *** JOB 3545420 ON c120 CANCELLED AT 2026-05-14T18:21:39 DUE TO TIME LIMIT ***
⚠ Job 3545422 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545422/3545422_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545422/3545422_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:23:18,445) - Starting with JobEnvironment(job_id=3545422, hostname=c120, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:23:18,447) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545422/3545422_submitted.pkl
Trial-Index: 89
slurmstepd: error: *** JOB 3545422 ON c120 CANCELLED AT 2026-05-14T18:23:39 DUE TO TIME LIMIT ***
⚠ Job 3545423 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545423/3545423_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545423/3545423_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:25:20,472) - Starting with JobEnvironment(job_id=3545423, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:25:20,475) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545423/3545423_submitted.pkl
Trial-Index: 87
slurmstepd: error: *** JOB 3545423 ON c27 CANCELLED AT 2026-05-14T18:25:39 DUE TO TIME LIMIT ***
⚠ Job 3545424 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545424/3545424_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545424/3545424_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:26:52,112) - Starting with JobEnvironment(job_id=3545424, hostname=c121, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:26:52,114) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545424/3545424_submitted.pkl
Trial-Index: 83
slurmstepd: error: *** JOB 3545424 ON c121 CANCELLED AT 2026-05-14T18:27:09 DUE TO TIME LIMIT ***
⚠ Job 3545421 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545421/3545421_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545421/3545421_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:21:16,304) - Starting with JobEnvironment(job_id=3545421, hostname=c106, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:21:16,306) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545421/3545421_submitted.pkl
Trial-Index: 94
slurmstepd: error: *** JOB 3545421 ON c106 CANCELLED AT 2026-05-14T18:21:39 DUE TO TIME LIMIT ***
⚠ Job 3545426 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545426/3545426_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545426/3545426_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:32:58,049) - Starting with JobEnvironment(job_id=3545426, hostname=c9, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:32:58,052) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545426/3545426_submitted.pkl
Trial-Index: 80
slurmstepd: error: *** JOB 3545426 ON c9 CANCELLED AT 2026-05-14T18:33:09 DUE TO TIME LIMIT ***
⚠ Job 3545428 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545428/3545428_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545428/3545428_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:40:34,210) - Starting with JobEnvironment(job_id=3545428, hostname=c133, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:40:34,212) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545428/3545428_submitted.pkl
Trial-Index: 86
slurmstepd: error: *** JOB 3545428 ON c133 CANCELLED AT 2026-05-14T18:40:39 DUE TO TIME LIMIT ***
⚠ Job 3545429 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545429/3545429_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545429/3545429_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:42:35,700) - Starting with JobEnvironment(job_id=3545429, hostname=c54, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:42:35,702) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545429/3545429_submitted.pkl
Trial-Index: 91
slurmstepd: error: *** JOB 3545429 ON c54 CANCELLED AT 2026-05-14T18:42:39 DUE TO TIME LIMIT ***
⚠ Job 3545430 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545430/3545430_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545430/3545430_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:43:05,647) - Starting with JobEnvironment(job_id=3545430, hostname=c123, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:43:05,649) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545430/3545430_submitted.pkl
Trial-Index: 81
slurmstepd: error: *** JOB 3545430 ON c123 CANCELLED AT 2026-05-14T18:43:09 DUE TO TIME LIMIT ***
⚠ Job 3545432 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545432/3545432_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545432/3545432_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:48:08,922) - Starting with JobEnvironment(job_id=3545432, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:48:08,923) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545432/3545432_submitted.pkl
Trial-Index: 99
slurmstepd: error: *** JOB 3545432 ON c108 CANCELLED AT 2026-05-14T18:48:09 DUE TO TIME LIMIT ***
⚠ Job 3545433 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545433/3545433_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545433/3545433_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:48:39,488) - Starting with JobEnvironment(job_id=3545433, hostname=c7, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:48:39,490) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545433/3545433_submitted.pkl
Trial-Index: 98
slurmstepd: error: *** JOB 3545433 ON c7 CANCELLED AT 2026-05-14T18:48:39 DUE TO TIME LIMIT ***
⚠ Job 3545434 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545434/3545434_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545434/3545434_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:50:11,616) - Starting with JobEnvironment(job_id=3545434, hostname=c49, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:50:11,619) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545434/3545434_submitted.pkl
Trial-Index: 97
slurmstepd: error: *** JOB 3545434 ON c49 CANCELLED AT 2026-05-14T18:50:39 DUE TO TIME LIMIT ***
get_ax_client_trial: trial_index 116 failed
execute_evaluation: _trial was not in execute_evaluation for params [116, {'batch_size': 16, 'ffn': 2, 'epochs': 212, 'n_layers': 2, 'd_model': 3, 'max_sequence_length': 128, 'lr': 1e-07, 'n_heads': 3, 'val_batches': '256', 'batches_per_epoch': '30'}, 4, 'systematic']
To cancel, press CTRLc, then run 'scancel 3531603'
⠋Importing logging...⠋Importing warnings...⠋Importing argparse...⠋Importing datetime...⠋Importing dataclass...⠋Importing socket...⠋Importing stat...⠋Importing pwd...⠋Importing base64...⠋Importing json...⠋Importing yaml...⠋Importing toml...⠋Importing csv...⠋Importing ast...⠋Importing rich.table...⠋Importing rich print...⠋Importing rich.pretty...⠋Importing pformat...⠋Importing rich.prompt...⠋Importing types.FunctionType...⠋Importing typing...⠋Importing ThreadPoolExecutor...⠋Importing submitit.LocalExecutor...⠋Importing submitit.Job...⠋Importing importlib.util...⠋Importing platform...⠋Importing inspect frame info...⠋Importing pathlib.Path...⠋Importing uuid...⠋Importing cowsay...⠋Importing shutil...⠋Importing itertools.combinations...⠋Importing os.listdir...⠋Importing os.path...⠋Importing PIL.Image...⠋Importing sixel...⠋Importing subprocess...⠋Importing tqdm...⠹Importing beartype...
[WARNING 05-10 21:52:30] ax.storage.sqa_store.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
⠙Importing rendering stuff...⠋Importing statistics...⠋Importing notifier...⠋Trying to import pyfiglet...⠋Importing helpers...⠋Importing pareto...⠋Parsing arguments...⠋Importing torch...⠋Importing numpy...⠋Importing ax...⠋Importing ax.core.generator_run...⠋Importing Cont_X_trans and Y_trans from ax.adapter.registry...⠋Importing ax.core.arm...⠋Importing ax.core.objective...⠋Importing ax.core.Metric...⠋Importing ax.exceptions.core...⠋Importing ax.exceptions.generation_strategy...⠋Importing CORE_DECODER_REGISTRY...⠋Trying ax.generation_strategy.generation_node...⠋Importing GenerationStep, GenerationStrategy from generation_strategy...⠋Importing GenerationNode from generation_node...⠋Importing ExternalGenerationNode...⠋Importing MinTrials...⠋Importing GeneratorSpec...⠋Importing Generators from ax.generation_strategy.registry...⠋Importing load_experiment...⠋Importing save_experiment...⠋Importing save_experiment_to_db...⠋Importing TrialStatus...⠋Importing Data...⠋Importing Experiment...⠋Importing parameter types...⠋Importing TParameterization...⠋Importing pandas...⠋Importing AxClient and ObjectiveProperties...⠋Importing RandomForestRegressor...⠋Importing botorch...⠋Importing submitit...⠋Importing ax logger...⠋Importing SQL-Storage-Stuff...
Run-UUID: 5ec7987b-7706-457a-8d5f-c1f197822434
/home/s3811141/.omniax_venvs/Python_3.11.3/x86_64/lib/python3.11/site-packages/plyer/platforms/linux/notification.py:103: UserWarning: The Python dbus package is not installed.
Try installing it with your distribution's package manager, it is usually called python-dbus or python3-dbus, but you might have to try dbus-python instead, e.g. when using pip.
warnings.warn(msg)
.d88888b. d8b .d88888b. 888 .d8888b.
d88P" "Y88b Y8Pd88P" "Y88b 888 d88P Y88b
888 888 888 888 888 888
888 88888888b.d88b. 88888b. 888888 88888888b. 888888 .d88P
888 888888 "888 "88b888 "88b888888 888888 "88b888 .od888P"
888 888888 888 888888 888888888 888888 888888 d88P"
Y88b. .d88P888 888 888888 888888Y88b. .d88P888 d88PY88b. 888"
"Y88888P" 888 888 888888 888888 "Y88888P" 88888P" "Y888888888888
888
888
888
⠋Writing worker creation log...
omniopt --partition=alpha --experiment_name=additionLLM0 --mem_gb=32 --time=7200 --worker_timeout=720 --max_eval=1000 --num_parallel_jobs=20 --gpus=1 --num_random_steps=20 --follow --live_share --send_anonymized_usage_stats --result_names 'VAL_LOSS=min PARAMS=min' --run_program='cHl0aG9uMyAvZGF0YS9ob3JzZS93cy9zMzgxMTE0MS1ncm9rMy90cmFpbi5weSAtLWJhdGNoLXNpemU9JShiYXRjaF9zaXplKSAtLXRvcG8gLS10b3BvLWV2ZXJ5IDEgLS1rZWxwLWV2ZXJ5PTEgLS1uLWxheWVycz0lKG5fbGF5ZXJzKSAtLWQtbW9kZWw9JShkX21vZGVsKSAtLW4taGVhZHM9JShuX2hlYWRzKSAtLXRva2VuaXplcl9pbml0aWFsX25yPTEwMDAgLS1scj0lKGxyKSAtLWJhdGNoZXMtcGVyLWVwb2NoPSUoYmF0Y2hlc19wZXJfZXBvY2gpIC0tdmFsLWJhdGNoZXM9JSh2YWxfYmF0Y2hlcykgLS1lcG9jaHM9JShlcG9jaHMpIC0tbWF4LXNlcS1sZW4gJShtYXhfc2VxdWVuY2VfbGVuZ3RoKSAtLXNjaGVkdWxlcj1jb3NpbmUgLS1mZm49JShmZm4pCg==' --run_program_once='cHl0aG9uMyAvZGF0YS9ob3JzZS93cy9zMzgxMTE0MS1ncm9rMi90cmFpbi5weSAtLWV4aXQtYWZ0ZXItaW5zdGFsbA==' --cpus_per_task=1 --nodes_per_job=1 --revert_to_random_when_seemingly_exhausted --model=BOTORCH_MODULAR --n_estimators_randomforest=100 --run_mode=local --show_sixel_general --show_sixel_trial_index_result --show_sixel_scatter --occ_type=euclid --main_process_gb=8 --nr_evals_per_arm=1 --max_nr_of_zero_results=50 --slurm_signal_delay_s=0 --max_failed_jobs=0 --max_attempts_for_generation=20 --num_restarts=20 --raw_samples=1024 --max_abandoned_retrial=20 --max_num_of_parallel_sruns=16 --number_of_generators=1 --generate_all_jobs_at_once --parameter batch_size range 16 512 int false --parameter ffn range 2 64 int false --parameter epochs range 10 500 int false --parameter n_layers range 1 4 int false --parameter d_model range 2 3 int false --parameter max_sequence_length range 32 128 int false --parameter val_batches fixed 256 --parameter lr range 0.0000001 0.1 float false --parameter n_heads range 1 32 int false --parameter batches_per_epoch fixed 30
⠋Disabling logging...⠋Setting run folder...⠋Creating folder /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0...⠋Writing revert_to_random_when_seemingly_exhausted file ...⠋Writing username state file...⠋Writing result names file...⠋Writing result min/max file...
Exiting after installation
Executing command: python3 /data/horse/ws/s3811141-grok2/train.py --exit-after-install
Setup script completed successfully ✅
⠋Saving state files...Run-folder: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0⠋Writing live_share file if it is present...⠋Writing job_start_time file...⠙Writing git information⠋Checking max_eval...⠋Calculating number of steps...⠋Adding excluded nodes...⠋Initializing ax_client...⠋Setting orchestrator...See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=additionLLM0&run_nr=0 for live-results.You have 1 CPUs available for the main process.Using CUDA device NVIDIA H100.Generation strategy: SOBOL for 20 steps and then BOTORCH_MODULAR for 980 steps.
Run-Program: python3 /data/horse/ws/s3811141-grok3/train.py --batch-size=%(batch_size) --topo --topo-every 1 --kelp-every=1 --n-layers=%(n_layers) --d-model=%(d_model) --n-heads=%(n_heads) --tokenizer_initial_nr=1000 --lr=%(lr) --batches-per-epoch=%(batches_per_epoch) --val-batches=%(val_batches) --epochs=%(epochs) --max-seq-len %(max_sequence_length) --scheduler=cosine --ffn=%(ffn)
Experiment parameters
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃Name ┃Type ┃Lower bound┃Upper bound┃Values┃Type ┃Log Scale?┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│batch_size │range│16 │512 ││int │No │
│ffn │range│2 │64 ││int │No │
│epochs │range│10 │500 ││int │No │
│n_layers │range│1 │4 ││int │No │
│d_model │range│2 │3 ││int │No │
│max_sequence_length│range│32 │128 ││int │No │
│val_batches │fixed│││256 │││
│lr │range│1e-07 │0.1 ││float│No │
│n_heads │range│1 │32 ││int │No │
│batches_per_epoch │fixed│││30 │││
└─────────────────────┴───────┴─────────────┴─────────────┴────────┴───────┴────────────┘
Result-Names
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃Result-Name┃Min or max?┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│VAL_LOSS │ min│
│PARAMS │ min│
└─────────────┴─────────────┘
⠋Write files and show overview
SOBOL, running 20 = ∑20/20, new result: VAL_LOSS: 0.022503, PARAMS: 70022.000000: 0%|░░░░░░░░░░| 0/1000 [1:47:10, ?it/s]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, 1 done, running 19 = ∑19/20, waiting for 19 jobs : 0%|░░░░░░░░░░| 1/1000 [1:47:14<1784:29:28, 6430.60s/it]
Exit-Code 3, this means: Invalid exit code detected
SOBOL, 1 done, running 19 = ∑19/20, new result: VAL_LOSS: 0.007289, PARAMS: 30922.000000: 0%|░░░░░░░░░░| 1/1000 [8:48:08<1784:29:28, 6430.60s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, 2 done, ▁█, running 18 = ∑18/20, new result: VAL_LOSS: 0.006747, PARAMS: 34121.000000: 0%|░░░░░░░░░░| 2/1000 [10:13:26<4852:54:00, 17505.45s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, 3 done, ▁▇█, running 17 = ∑17/20, waiting for 17 jobs : 0%|░░░░░░░░░░| 3/1000 [10:13:38<3281:36:42, 11849.35s/it]
⚠ Job 3531618 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531618/3531618_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531618/3531618_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:17:13,900) - Starting with JobEnvironment(job_id=3531618, hostname=c56, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:17:13,903) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531618/3531618_submitted.pkl
Trial-Index: 10
slurmstepd: error: *** JOB 3531618 ON c56 CANCELLED AT 2026-05-11T10:17:15 DUE TO TIME LIMIT ***
SOBOL, failed: 1, 3 done, ▁▇█, timeout/running 5/11 = ∑16/20, waiting for 16 jobs : 0%|░░░░░░░░░░| 3/1000 [12:39:10<3281:36:42, 11849.35s/it]
⚠ Job 3531619 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531619/3531619_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531619/3531619_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:25:16,070) - Starting with JobEnvironment(job_id=3531619, hostname=c19, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:25:16,072) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531619/3531619_submitted.pkl
Trial-Index: 11
slurmstepd: error: *** JOB 3531619 ON c19 CANCELLED AT 2026-05-11T10:25:15 DUE TO TIME LIMIT ***
⚠ Job 3531621 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531621/3531621_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531621/3531621_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:27:16,820) - Starting with JobEnvironment(job_id=3531621, hostname=c26, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:27:16,822) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531621/3531621_submitted.pkl
Trial-Index: 3
slurmstepd: error: *** JOB 3531621 ON c26 CANCELLED AT 2026-05-11T10:27:15 DUE TO TIME LIMIT ***
⚠ Job 3531623 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531623/3531623_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531623/3531623_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:27:46,208) - Starting with JobEnvironment(job_id=3531623, hostname=c23, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:27:46,210) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531623/3531623_submitted.pkl
Trial-Index: 5
slurmstepd: error: *** JOB 3531623 ON c23 CANCELLED AT 2026-05-11T10:27:45 DUE TO TIME LIMIT ***
⚠ Job 3531620 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531620/3531620_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531620/3531620_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:26:16,606) - Starting with JobEnvironment(job_id=3531620, hostname=c23, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:26:16,608) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531620/3531620_submitted.pkl
Trial-Index: 4
slurmstepd: error: *** JOB 3531620 ON c23 CANCELLED AT 2026-05-11T10:26:15 DUE TO TIME LIMIT ***
⚠ Job 3531622 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531622/3531622_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531622/3531622_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:27:46,088) - Starting with JobEnvironment(job_id=3531622, hostname=c36, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:27:46,090) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531622/3531622_submitted.pkl
Trial-Index: 6
slurmstepd: error: *** JOB 3531622 ON c36 CANCELLED AT 2026-05-11T10:27:45 DUE TO TIME LIMIT ***
SOBOL, failed: 6, 3 done, ▁▇█, timeout/running 3/8 = ∑11/20, waiting for 11 jobs : 0%|░░░░░░░░░░| 3/1000 [12:49:11<3281:36:42, 11849.35s/it]
⚠ Job 3531625 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531625/3531625_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531625/3531625_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:37:18,489) - Starting with JobEnvironment(job_id=3531625, hostname=c26, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:37:18,491) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531625/3531625_submitted.pkl
Trial-Index: 8
slurmstepd: error: *** JOB 3531625 ON c26 CANCELLED AT 2026-05-11T10:37:45 DUE TO TIME LIMIT ***
⚠ Job 3531626 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531626/3531626_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531626/3531626_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:37:48,597) - Starting with JobEnvironment(job_id=3531626, hostname=c131, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:37:48,599) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531626/3531626_submitted.pkl
Trial-Index: 7
slurmstepd: error: *** JOB 3531626 ON c131 CANCELLED AT 2026-05-11T10:38:15 DUE TO TIME LIMIT ***
⚠ Job 3531624 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531624/3531624_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531624/3531624_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:37:18,490) - Starting with JobEnvironment(job_id=3531624, hostname=c26, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:37:18,491) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531624/3531624_submitted.pkl
Trial-Index: 14
slurmstepd: error: *** JOB 3531624 ON c26 CANCELLED AT 2026-05-11T10:37:45 DUE TO TIME LIMIT ***
SOBOL, failed: 9, 3 done, ▁▇█, running 8 = ∑8/20, waiting for 8 jobs : 0%|░░░░░░░░░░| 3/1000 [12:49:30<3281:36:42, 11849.35s/it]
⚠ Job 3531628 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531628/3531628_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531628/3531628_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:43:19,896) - Starting with JobEnvironment(job_id=3531628, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:43:19,898) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531628/3531628_submitted.pkl
Trial-Index: 13
slurmstepd: error: *** JOB 3531628 ON c27 CANCELLED AT 2026-05-11T10:43:45 DUE TO TIME LIMIT ***
⚠ Job 3531629 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531629/3531629_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531629/3531629_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:43:50,860) - Starting with JobEnvironment(job_id=3531629, hostname=c149, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:43:50,862) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531629/3531629_submitted.pkl
Trial-Index: 2
slurmstepd: error: *** JOB 3531629 ON c149 CANCELLED AT 2026-05-11T10:44:15 DUE TO TIME LIMIT ***
⚠ Job 3531631 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531631/3531631_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531631/3531631_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:44:50,620) - Starting with JobEnvironment(job_id=3531631, hostname=c59, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:44:50,622) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531631/3531631_submitted.pkl
Trial-Index: 9
slurmstepd: error: *** JOB 3531631 ON c59 CANCELLED AT 2026-05-11T10:45:15 DUE TO TIME LIMIT ***
⚠ Job 3531632 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531632/3531632_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531632/3531632_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:44:50,620) - Starting with JobEnvironment(job_id=3531632, hostname=c59, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:44:50,622) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531632/3531632_submitted.pkl
Trial-Index: 1
slurmstepd: error: *** JOB 3531632 ON c59 CANCELLED AT 2026-05-11T10:45:15 DUE TO TIME LIMIT ***
⚠ Job 3531634 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531634/3531634_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531634/3531634_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:46:20,052) - Starting with JobEnvironment(job_id=3531634, hostname=c85, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:46:20,054) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531634/3531634_submitted.pkl
Trial-Index: 17
slurmstepd: error: *** JOB 3531634 ON c85 CANCELLED AT 2026-05-11T10:46:45 DUE TO TIME LIMIT ***
⚠ Job 3531635 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531635/3531635_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531635/3531635_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:46:50,491) - Starting with JobEnvironment(job_id=3531635, hostname=c89, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:46:50,493) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531635/3531635_submitted.pkl
Trial-Index: 18
slurmstepd: error: *** JOB 3531635 ON c89 CANCELLED AT 2026-05-11T10:47:15 DUE TO TIME LIMIT ***
⚠ Job 3531636 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531636/3531636_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531636/3531636_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:47:50,921) - Starting with JobEnvironment(job_id=3531636, hostname=c147, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:47:50,927) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531636/3531636_submitted.pkl
Trial-Index: 19
slurmstepd: error: *** JOB 3531636 ON c147 CANCELLED AT 2026-05-11T10:48:15 DUE TO TIME LIMIT ***
⚠ Job 3531633 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531633/3531633_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531633/3531633_0_log.out
----------------------------------------
submitit INFO (2026-05-10 22:45:49,795) - Starting with JobEnvironment(job_id=3531633, hostname=c96, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 22:45:49,797) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3531633/3531633_submitted.pkl
Trial-Index: 12
slurmstepd: error: *** JOB 3531633 ON c96 CANCELLED AT 2026-05-11T10:46:15 DUE TO TIME LIMIT ***
SOBOL, failed: 17, 3 done, ▁▇█, running 20 = ∑20/20, new result: VAL_LOSS: 0.034741, PARAMS: 34745.000000: 0%|░░░░░░░░░░| 3/1000 [30:16:55<3281:36:42, 11849.35s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 17, 4 done, ▄▇█▁, running 19 = ∑19/20, new result: VAL_LOSS: 0.022921, PARAMS: 30668.000000: 0%|░░░░░░░░░░| 4/1000 [30:54:05<9871:05:41, 35678.66s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 17, 5 done, ▄▇█▁▃, running 18 = ∑18/20, new result: VAL_LOSS: 0.010711, PARAMS: 28772.000000: 0%|░░░░░░░░░░| 5/1000 [37:44:46<6527:28:32, 23617.00s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 17, 6 done, ▄▇█▁▇▃, timeout/running 1/16 = ∑17/20, waiting for 17 jobs : 1%|░░░░░░░░░░| 6/1000 [38:45:48<6617:00:54, 23965.04s/it]
⚠ Job 3533156 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533156/3533156_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533156/3533156_0_log.out
----------------------------------------
submitit INFO (2026-05-12 00:32:43,388) - Starting with JobEnvironment(job_id=3533156, hostname=c109, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 00:32:43,390) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533156/3533156_submitted.pkl
Trial-Index: 20
slurmstepd: error: *** JOB 3533156 ON c109 CANCELLED AT 2026-05-12T12:33:05 DUE TO TIME LIMIT ***
SOBOL, failed: 18, 6 done, ▄▇█▁▇▃, running 16 = ∑16/20, waiting for 16 jobs : 1%|░░░░░░░░░░| 6/1000 [38:46:09<6617:00:54, 23965.04s/it]
⚠ Job 3533159 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533159/3533159_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533159/3533159_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:36:12,532) - Starting with JobEnvironment(job_id=3533159, hostname=c106, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:36:12,534) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533159/3533159_submitted.pkl
Trial-Index: 29
slurmstepd: error: *** JOB 3533159 ON c106 CANCELLED AT 2026-05-12T15:36:11 DUE TO TIME LIMIT ***
⚠ Job 3533157 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533157/3533157_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533157/3533157_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:33:11,102) - Starting with JobEnvironment(job_id=3533157, hostname=c94, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:33:11,104) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533157/3533157_submitted.pkl
Trial-Index: 30
slurmstepd: error: *** JOB 3533157 ON c94 CANCELLED AT 2026-05-12T15:33:11 DUE TO TIME LIMIT ***
⚠ Job 3533160 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533160/3533160_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533160/3533160_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:36:12,532) - Starting with JobEnvironment(job_id=3533160, hostname=c106, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:36:12,534) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533160/3533160_submitted.pkl
Trial-Index: 34
slurmstepd: error: *** JOB 3533160 ON c106 CANCELLED AT 2026-05-12T15:36:11 DUE TO TIME LIMIT ***
⚠ Job 3533161 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533161/3533161_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533161/3533161_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:36:42,457) - Starting with JobEnvironment(job_id=3533161, hostname=c148, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:36:42,458) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533161/3533161_submitted.pkl
Trial-Index: 21
slurmstepd: error: *** JOB 3533161 ON c148 CANCELLED AT 2026-05-12T15:36:41 DUE TO TIME LIMIT ***
SOBOL, failed: 22, 6 done, ▄▇█▁▇▃, timeout 12 = ∑12/20, waiting for 12 jobs : 1%|░░░░░░░░░░| 6/1000 [41:55:58<6617:00:54, 23965.04s/it]
⚠ Job 3533164 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533164/3533164_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533164/3533164_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:41:45,326) - Starting with JobEnvironment(job_id=3533164, hostname=c100, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:41:45,329) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533164/3533164_submitted.pkl
Trial-Index: 25
slurmstepd: error: *** JOB 3533164 ON c100 CANCELLED AT 2026-05-12T15:42:11 DUE TO TIME LIMIT ***
⚠ Job 3533167 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533167/3533167_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533167/3533167_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:44:47,385) - Starting with JobEnvironment(job_id=3533167, hostname=c86, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:44:47,388) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533167/3533167_submitted.pkl
Trial-Index: 35
slurmstepd: error: *** JOB 3533167 ON c86 CANCELLED AT 2026-05-12T15:45:11 DUE TO TIME LIMIT ***
⚠ Job 3533165 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533165/3533165_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533165/3533165_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:43:46,391) - Starting with JobEnvironment(job_id=3533165, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:43:46,395) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533165/3533165_submitted.pkl
Trial-Index: 22
slurmstepd: error: *** JOB 3533165 ON c99 CANCELLED AT 2026-05-12T15:44:11 DUE TO TIME LIMIT ***
⚠ Job 3533171 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533171/3533171_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533171/3533171_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:45:47,758) - Starting with JobEnvironment(job_id=3533171, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:45:47,764) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533171/3533171_submitted.pkl
Trial-Index: 28
slurmstepd: error: *** JOB 3533171 ON c108 CANCELLED AT 2026-05-12T15:46:11 DUE TO TIME LIMIT ***
⚠ Job 3533174 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533174/3533174_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533174/3533174_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:46:18,355) - Starting with JobEnvironment(job_id=3533174, hostname=c117, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:46:18,361) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533174/3533174_submitted.pkl
Trial-Index: 39
slurmstepd: error: *** JOB 3533174 ON c117 CANCELLED AT 2026-05-12T15:46:41 DUE TO TIME LIMIT ***
⚠ Job 3533166 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533166/3533166_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533166/3533166_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:44:17,191) - Starting with JobEnvironment(job_id=3533166, hostname=c65, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:44:17,197) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533166/3533166_submitted.pkl
Trial-Index: 26
slurmstepd: error: *** JOB 3533166 ON c65 CANCELLED AT 2026-05-12T15:44:41 DUE TO TIME LIMIT ***
⚠ Job 3533162 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533162/3533162_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533162/3533162_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:39:43,984) - Starting with JobEnvironment(job_id=3533162, hostname=c104, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:39:43,986) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533162/3533162_submitted.pkl
Trial-Index: 33
slurmstepd: error: *** JOB 3533162 ON c104 CANCELLED AT 2026-05-12T15:40:11 DUE TO TIME LIMIT ***
⚠ Job 3533168 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533168/3533168_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533168/3533168_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:45:18,025) - Starting with JobEnvironment(job_id=3533168, hostname=c117, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:45:18,029) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533168/3533168_submitted.pkl
Trial-Index: 32
slurmstepd: error: *** JOB 3533168 ON c117 CANCELLED AT 2026-05-12T15:45:41 DUE TO TIME LIMIT ***
⚠ Job 3533173 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533173/3533173_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533173/3533173_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:46:18,457) - Starting with JobEnvironment(job_id=3533173, hostname=c143, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:46:18,463) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533173/3533173_submitted.pkl
Trial-Index: 38
slurmstepd: error: *** JOB 3533173 ON c143 CANCELLED AT 2026-05-12T15:46:41 DUE TO TIME LIMIT ***
⚠ Job 3533172 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533172/3533172_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533172/3533172_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:45:48,445) - Starting with JobEnvironment(job_id=3533172, hostname=c34, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:45:48,451) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533172/3533172_submitted.pkl
Trial-Index: 37
slurmstepd: error: *** JOB 3533172 ON c34 CANCELLED AT 2026-05-12T15:46:11 DUE TO TIME LIMIT ***
⚠ Job 3533169 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533169/3533169_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533169/3533169_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:45:17,900) - Starting with JobEnvironment(job_id=3533169, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:45:17,904) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533169/3533169_submitted.pkl
Trial-Index: 31
slurmstepd: error: *** JOB 3533169 ON c108 CANCELLED AT 2026-05-12T15:45:41 DUE TO TIME LIMIT ***
⚠ Job 3533163 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533163/3533163_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533163/3533163_0_log.out
----------------------------------------
submitit INFO (2026-05-12 03:42:15,524) - Starting with JobEnvironment(job_id=3533163, hostname=c150, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 03:42:15,526) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3533163/3533163_submitted.pkl
Trial-Index: 24
slurmstepd: error: *** JOB 3533163 ON c150 CANCELLED AT 2026-05-12T15:42:41 DUE TO TIME LIMIT ***
SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, running/pending 10/10 = ∑20/20, new result: VAL_LOSS: 0.019894, PARAMS: 20582.000000: 1%|░░░░░░░░░░| 6/1000 [52:46:31<6617:00:54, 23965.04s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 34, 7 done, ▄▇█▁▇▃▄, running 19 = ∑19/20, new result: VAL_LOSS: 0.022520, PARAMS: 79270.000000 : 1%|░░░░░░░░░░| 7/1000 [56:19:51<9328:16:04, 33818.49s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 34, 8 done, ▄▇█▁▇▃▄▄, running 18 = ∑18/20, new result: VAL_LOSS: 0.010525, PARAMS: 69292.000000 : 1%|░░░░░░░░░░| 8/1000 [58:04:50<7475:01:42, 27127.12s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 34, 9 done, ▄▇█▁▇▃▇▄▄, running 17 = ∑17/20, new result: VAL_LOSS: 0.011913, PARAMS: 9652.000000 : 1%|░░░░░░░░░░| 9/1000 [59:39:22<5675:11:10, 20616.22s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 34, 10 done, ▄▇█▁▇▃▇▄▄▆, timeout/running 5/11 = ∑16/20, waiting for 16 jobs : 1%|░░░░░░░░░░| 10/1000 [60:37:33<4400:42:14, 16002.56s/it]
⚠ Job 3539658 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539658/3539658_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539658/3539658_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:51,974) - Starting with JobEnvironment(job_id=3539658, hostname=c71, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:51,976) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539658/3539658_submitted.pkl
Trial-Index: 49
slurmstepd: error: *** JOB 3539658 ON c71 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
⚠ Job 3539656 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539656/3539656_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539656/3539656_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:50,796) - Starting with JobEnvironment(job_id=3539656, hostname=c93, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:50,799) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539656/3539656_submitted.pkl
Trial-Index: 46
slurmstepd: error: *** JOB 3539656 ON c93 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
⚠ Job 3539659 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539659/3539659_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539659/3539659_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:50,958) - Starting with JobEnvironment(job_id=3539659, hostname=c70, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:50,960) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539659/3539659_submitted.pkl
Trial-Index: 53
slurmstepd: error: *** JOB 3539659 ON c70 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
⚠ Job 3539660 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539660/3539660_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539660/3539660_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:50,816) - Starting with JobEnvironment(job_id=3539660, hostname=c59, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:50,819) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539660/3539660_submitted.pkl
Trial-Index: 41
slurmstepd: error: *** JOB 3539660 ON c59 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
⚠ Job 3539655 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539655/3539655_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539655/3539655_0_log.out
----------------------------------------
submitit INFO (2026-05-12 22:29:51,836) - Starting with JobEnvironment(job_id=3539655, hostname=c96, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-12 22:29:51,839) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539655/3539655_submitted.pkl
Trial-Index: 43
slurmstepd: error: *** JOB 3539655 ON c96 CANCELLED AT 2026-05-13T10:30:09 DUE TO TIME LIMIT ***
SOBOL, failed: 39, 10 done, ▄▇█▁▇▃▇▄▄▆, running 11 = ∑11/20, waiting for 11 jobs : 1%|░░░░░░░░░░| 10/1000 [60:38:01<4400:42:14, 16002.56s/it]
⚠ Job 3539661 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539661/3539661_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539661/3539661_0_log.out
----------------------------------------
submitit INFO (2026-05-13 01:26:05,630) - Starting with JobEnvironment(job_id=3539661, hostname=c60, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 01:26:05,633) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539661/3539661_submitted.pkl
Trial-Index: 45
slurmstepd: error: *** JOB 3539661 ON c60 CANCELLED AT 2026-05-13T13:26:14 DUE TO TIME LIMIT ***
⚠ Job 3539663 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539663/3539663_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539663/3539663_0_log.out
----------------------------------------
submitit INFO (2026-05-13 01:26:05,361) - Starting with JobEnvironment(job_id=3539663, hostname=c57, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 01:26:05,363) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539663/3539663_submitted.pkl
Trial-Index: 50
slurmstepd: error: *** JOB 3539663 ON c57 CANCELLED AT 2026-05-13T13:26:14 DUE TO TIME LIMIT ***
SOBOL, failed: 41, 10 done, ▄▇█▁▇▃▇▄▄▆, running 9 = ∑9/20, waiting for 9 jobs : 1%|░░░░░░░░░░| 10/1000 [63:38:08<4400:42:14, 16002.56s/it]
⚠ Job 3539664 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539664/3539664_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539664/3539664_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:08:21,457) - Starting with JobEnvironment(job_id=3539664, hostname=c34, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:08:21,461) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539664/3539664_submitted.pkl
Trial-Index: 42
slurmstepd: error: *** JOB 3539664 ON c34 CANCELLED AT 2026-05-13T17:08:28 DUE TO TIME LIMIT ***
SOBOL, failed: 42, 10 done, ▄▇█▁▇▃▇▄▄▆, running/timeout 7/1 = ∑8/20, waiting for 8 jobs : 1%|░░░░░░░░░░| 10/1000 [67:28:00<4400:42:14, 16002.56s/it]
⚠ Job 3539665 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539665/3539665_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539665/3539665_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:18:32,340) - Starting with JobEnvironment(job_id=3539665, hostname=c147, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:18:32,342) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539665/3539665_submitted.pkl
Trial-Index: 44
slurmstepd: error: *** JOB 3539665 ON c147 CANCELLED AT 2026-05-13T17:19:00 DUE TO TIME LIMIT ***
SOBOL, failed: 43, 10 done, ▄▇█▁▇▃▇▄▄▆, running 7 = ∑7/20, waiting for 7 jobs : 1%|░░░░░░░░░░| 10/1000 [67:28:22<4400:42:14, 16002.56s/it]
⚠ Job 3539667 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539667/3539667_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539667/3539667_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:23:37,852) - Starting with JobEnvironment(job_id=3539667, hostname=c46, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:23:37,854) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539667/3539667_submitted.pkl
Trial-Index: 52
slurmstepd: error: *** JOB 3539667 ON c46 CANCELLED AT 2026-05-13T17:24:00 DUE TO TIME LIMIT ***
⚠ Job 3539666 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539666/3539666_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539666/3539666_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:23:37,225) - Starting with JobEnvironment(job_id=3539666, hostname=c111, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:23:37,227) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539666/3539666_submitted.pkl
Trial-Index: 48
slurmstepd: error: *** JOB 3539666 ON c111 CANCELLED AT 2026-05-13T17:24:00 DUE TO TIME LIMIT ***
⚠ Job 3539668 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539668/3539668_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539668/3539668_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:23:37,852) - Starting with JobEnvironment(job_id=3539668, hostname=c46, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:23:37,854) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539668/3539668_submitted.pkl
Trial-Index: 54
slurmstepd: error: *** JOB 3539668 ON c46 CANCELLED AT 2026-05-13T17:24:00 DUE TO TIME LIMIT ***
SOBOL, failed: 46, 10 done, ▄▇█▁▇▃▇▄▄▆, timeout/running 2/2 = ∑4/20, waiting for 4 jobs : 1%|░░░░░░░░░░| 10/1000 [67:48:01<4400:42:14, 16002.56s/it]
⚠ Job 3539669 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539669/3539669_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539669/3539669_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:34:48,508) - Starting with JobEnvironment(job_id=3539669, hostname=c55, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:34:48,511) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539669/3539669_submitted.pkl
Trial-Index: 55
slurmstepd: error: *** JOB 3539669 ON c55 CANCELLED AT 2026-05-13T17:35:00 DUE TO TIME LIMIT ***
⚠ Job 3539677 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539677/3539677_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539677/3539677_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:38:52,569) - Starting with JobEnvironment(job_id=3539677, hostname=c73, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:38:52,571) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539677/3539677_submitted.pkl
Trial-Index: 57
slurmstepd: error: *** JOB 3539677 ON c73 CANCELLED AT 2026-05-13T17:39:01 DUE TO TIME LIMIT ***
SOBOL, failed: 48, 10 done, ▄▇█▁▇▃▇▄▄▆, running 2 = ∑2/20, waiting for 2 jobs : 1%|░░░░░░░░░░| 10/1000 [67:48:25<4400:42:14, 16002.56s/it]
⚠ Job 3539678 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539678/3539678_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539678/3539678_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:41:25,346) - Starting with JobEnvironment(job_id=3539678, hostname=c107, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:41:25,348) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539678/3539678_submitted.pkl
Trial-Index: 59
slurmstepd: error: *** JOB 3539678 ON c107 CANCELLED AT 2026-05-13T17:41:31 DUE TO TIME LIMIT ***
⚠ Job 3539679 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539679/3539679_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539679/3539679_0_log.out
----------------------------------------
submitit INFO (2026-05-13 05:43:26,886) - Starting with JobEnvironment(job_id=3539679, hostname=c15, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 05:43:26,889) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3539679/3539679_submitted.pkl
Trial-Index: 58
slurmstepd: error: *** JOB 3539679 ON c15 CANCELLED AT 2026-05-13T17:43:31 DUE TO TIME LIMIT ***
SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, running 20 = ∑20/20, new result: VAL_LOSS: 0.027349, PARAMS: 28106.000000 : 1%|░░░░░░░░░░| 10/1000 [71:09:04<4400:42:14, 16002.56s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 50, 11 done, ▄▇█▁▇▃▇▄▄▆▂, running 19 = ∑19/20, new result: VAL_LOSS: 0.008205, PARAMS: 32913.000000 : 1%|░░░░░░░░░░| 11/1000 [71:24:22<6530:09:30, 23770.04s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 50, 12 done, ▄▇█▁▇▃▇▄▄▆▇▂, running 18 = ∑18/20, new result: VAL_LOSS: 0.017069, PARAMS: 27232.000000 : 1%|░░░░░░░░░░| 12/1000 [71:57:58<4615:40:23, 16818.24s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 50, 13 done, ▄▇█▁▇▃▇▄▄▆▇▂▅, running 17 = ∑17/20, new result: VAL_LOSS: 0.012173, PARAMS: 42708.000000 : 1%|░░░░░░░░░░| 13/1000 [73:52:15<3381:34:00, 12333.98s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 50, 14 done, ▄▇█▁▇▃▇▄▄▆▇▂▅▆, running 16 = ∑16/20, new result: VAL_LOSS: 0.026844, PARAMS: 17348.000000 : 1%|░░░░░░░░░░| 14/1000 [74:38:11<2925:02:59, 10679.70s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 50, 15 done, ▄▇█▁▇▃▇▄▄▆▂▇▂▅▆, running 15 = ∑15/20, new result: VAL_LOSS: 0.011546, PARAMS: 45766.000000 : 2%|░░░░░░░░░░| 15/1000 [78:00:36<2268:36:10, 8291.34s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 50, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, timeout/running 11/3 = ∑14/20, waiting for 14 jobs : 2%|░░░░░░░░░░| 16/1000 [80:06:55<2583:25:25, 9451.55s/it]
⚠ Job 3543793 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543793/3543793_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543793/3543793_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:57:59,080) - Starting with JobEnvironment(job_id=3543793, hostname=c90, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:57:59,083) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543793/3543793_submitted.pkl
Trial-Index: 74
slurmstepd: error: *** JOB 3543793 ON c90 CANCELLED AT 2026-05-14T05:57:58 DUE TO TIME LIMIT ***
⚠ Job 3543775 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543775/3543775_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543775/3543775_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:54:55,674) - Starting with JobEnvironment(job_id=3543775, hostname=c42, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:54:55,676) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543775/3543775_submitted.pkl
Trial-Index: 65
slurmstepd: error: *** JOB 3543775 ON c42 CANCELLED AT 2026-05-14T05:54:58 DUE TO TIME LIMIT ***
⚠ Job 3543789 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543789/3543789_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543789/3543789_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:56:27,018) - Starting with JobEnvironment(job_id=3543789, hostname=c89, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:56:27,019) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543789/3543789_submitted.pkl
Trial-Index: 62
slurmstepd: error: *** JOB 3543789 ON c89 CANCELLED AT 2026-05-14T05:56:28 DUE TO TIME LIMIT ***
⚠ Job 3543776 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543776/3543776_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543776/3543776_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:54:55,674) - Starting with JobEnvironment(job_id=3543776, hostname=c40, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:54:55,676) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543776/3543776_submitted.pkl
Trial-Index: 60
slurmstepd: error: *** JOB 3543776 ON c40 CANCELLED AT 2026-05-14T05:54:58 DUE TO TIME LIMIT ***
⚠ Job 3543777 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543777/3543777_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543777/3543777_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:55:25,401) - Starting with JobEnvironment(job_id=3543777, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:55:25,403) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543777/3543777_submitted.pkl
Trial-Index: 69
slurmstepd: error: *** JOB 3543777 ON c142 CANCELLED AT 2026-05-14T05:55:28 DUE TO TIME LIMIT ***
⚠ Job 3543778 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543778/3543778_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543778/3543778_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:55:25,378) - Starting with JobEnvironment(job_id=3543778, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:55:25,380) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543778/3543778_submitted.pkl
Trial-Index: 75
slurmstepd: error: *** JOB 3543778 ON c108 CANCELLED AT 2026-05-14T05:55:28 DUE TO TIME LIMIT ***
⚠ Job 3543794 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543794/3543794_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543794/3543794_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:58:29,223) - Starting with JobEnvironment(job_id=3543794, hostname=c71, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:58:29,225) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543794/3543794_submitted.pkl
Trial-Index: 68
slurmstepd: error: *** JOB 3543794 ON c71 CANCELLED AT 2026-05-14T05:58:28 DUE TO TIME LIMIT ***
⚠ Job 3543779 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543779/3543779_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543779/3543779_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:55:56,582) - Starting with JobEnvironment(job_id=3543779, hostname=c145, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:55:56,584) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543779/3543779_submitted.pkl
Trial-Index: 73
slurmstepd: error: *** JOB 3543779 ON c145 CANCELLED AT 2026-05-14T05:55:58 DUE TO TIME LIMIT ***
⚠ Job 3543791 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543791/3543791_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543791/3543791_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:57:27,796) - Starting with JobEnvironment(job_id=3543791, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:57:27,799) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543791/3543791_submitted.pkl
Trial-Index: 64
slurmstepd: error: *** JOB 3543791 ON c99 CANCELLED AT 2026-05-14T05:57:28 DUE TO TIME LIMIT ***
⚠ Job 3543773 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543773/3543773_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543773/3543773_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:54:55,086) - Starting with JobEnvironment(job_id=3543773, hostname=c117, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:54:55,088) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543773/3543773_submitted.pkl
Trial-Index: 66
slurmstepd: error: *** JOB 3543773 ON c117 CANCELLED AT 2026-05-14T05:54:58 DUE TO TIME LIMIT ***
⚠ Job 3543780 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543780/3543780_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543780/3543780_0_log.out
----------------------------------------
submitit INFO (2026-05-13 17:55:56,192) - Starting with JobEnvironment(job_id=3543780, hostname=c111, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 17:55:56,194) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543780/3543780_submitted.pkl
Trial-Index: 71
slurmstepd: error: *** JOB 3543780 ON c111 CANCELLED AT 2026-05-14T05:55:58 DUE TO TIME LIMIT ***
SOBOL, failed: 61, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, timeout 3 = ∑3/20, waiting for 3 jobs : 2%|░░░░░░░░░░| 16/1000 [80:16:56<2583:25:25, 9451.55s/it]
⚠ Job 3543799 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543799/3543799_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543799/3543799_0_log.out
----------------------------------------
submitit INFO (2026-05-13 18:00:00,947) - Starting with JobEnvironment(job_id=3543799, hostname=c94, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 18:00:00,950) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543799/3543799_submitted.pkl
Trial-Index: 77
slurmstepd: error: *** JOB 3543799 ON c94 CANCELLED AT 2026-05-14T06:00:28 DUE TO TIME LIMIT ***
⚠ Job 3543800 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543800/3543800_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543800/3543800_0_log.out
----------------------------------------
submitit INFO (2026-05-13 18:00:00,927) - Starting with JobEnvironment(job_id=3543800, hostname=c71, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 18:00:00,931) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543800/3543800_submitted.pkl
Trial-Index: 76
slurmstepd: error: *** JOB 3543800 ON c71 CANCELLED AT 2026-05-14T06:00:28 DUE TO TIME LIMIT ***
⚠ Job 3543802 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543802/3543802_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543802/3543802_0_log.out
----------------------------------------
submitit INFO (2026-05-13 18:03:03,040) - Starting with JobEnvironment(job_id=3543802, hostname=c114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-13 18:03:03,042) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3543802/3543802_submitted.pkl
Trial-Index: 78
slurmstepd: error: *** JOB 3543802 ON c114 CANCELLED AT 2026-05-14T06:03:28 DUE TO TIME LIMIT ***
SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running 20 = ∑20/20, new result: VAL_LOSS: 0.030514, PARAMS: 18878.000000 : 2%|░░░░░░░░░░| 16/1000 [83:31:14<2583:25:25, 9451.55s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 64, 17 done, █▁▇▃▇▄▄▆▂▇▂▆▅▆▂, running 19 = ∑19/20, new result: VAL_LOSS: 0.012584, PARAMS: 3038.000000 : 2%|░░░░░░░░░░| 17/1000 [84:11:54<3433:32:38, 12574.53s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 64, 18 done, ▁▇▄▇▄▄▇▃█▂▇▅▆▂▆, running 18 = ∑18/20, new result: VAL_LOSS: 0.017702, PARAMS: 94528.000000 : 2%|░░░░░░░░░░| 18/1000 [85:44:27<2599:19:42, 9529.11s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 64, 19 done, ▇▃▇▄▃▆▂█▁▆▅▆▅▁▆, running 17 = ∑17/20, new result: VAL_LOSS: 0.007035, PARAMS: 6835.000000 : 2%|░░░░░░░░░░| 19/1000 [86:53:12<2271:17:35, 8335.02s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 64, 20 done, ▃▆▄▃▆▂▇▁▆▅▆▄▁▆█, running 16 = ∑16/20, new result: VAL_LOSS: 0.011096, PARAMS: 69538.000000 : 2%|░░░░░░░░░░| 20/1000 [88:52:26<1924:50:18, 7070.83s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
SOBOL, failed: 64, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 15 = ∑15/20, waiting for 15 jobs : 2%|░░░░░░░░░░| 21/1000 [88:53:04<1929:40:22, 7095.84s/it]
⚠ Job 3545416 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545416/3545416_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545416/3545416_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:13:40,778) - Starting with JobEnvironment(job_id=3545416, hostname=c118, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:13:40,780) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545416/3545416_submitted.pkl
Trial-Index: 92
slurmstepd: error: *** JOB 3545416 ON c118 CANCELLED AT 2026-05-14T18:13:40 DUE TO TIME LIMIT ***
⚠ Job 3545415 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545415/3545415_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545415/3545415_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:13:10,352) - Starting with JobEnvironment(job_id=3545415, hostname=c114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:13:10,354) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545415/3545415_submitted.pkl
Trial-Index: 82
slurmstepd: error: *** JOB 3545415 ON c114 CANCELLED AT 2026-05-14T18:13:09 DUE TO TIME LIMIT ***
⚠ Job 3545418 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545418/3545418_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545418/3545418_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:17:44,523) - Starting with JobEnvironment(job_id=3545418, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:17:44,525) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545418/3545418_submitted.pkl
Trial-Index: 93
slurmstepd: error: *** JOB 3545418 ON c136 CANCELLED AT 2026-05-14T18:18:09 DUE TO TIME LIMIT ***
SOBOL, failed: 67, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 12 = ∑12/20, waiting for 12 jobs : 2%|░░░░░░░░░░| 21/1000 [92:27:12<1929:40:22, 7095.84s/it]
⚠ Job 3545420 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545420/3545420_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545420/3545420_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:21:16,042) - Starting with JobEnvironment(job_id=3545420, hostname=c120, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:21:16,044) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545420/3545420_submitted.pkl
Trial-Index: 85
slurmstepd: error: *** JOB 3545420 ON c120 CANCELLED AT 2026-05-14T18:21:39 DUE TO TIME LIMIT ***
⚠ Job 3545422 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545422/3545422_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545422/3545422_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:23:18,445) - Starting with JobEnvironment(job_id=3545422, hostname=c120, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:23:18,447) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545422/3545422_submitted.pkl
Trial-Index: 89
slurmstepd: error: *** JOB 3545422 ON c120 CANCELLED AT 2026-05-14T18:23:39 DUE TO TIME LIMIT ***
⚠ Job 3545423 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545423/3545423_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545423/3545423_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:25:20,472) - Starting with JobEnvironment(job_id=3545423, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:25:20,475) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545423/3545423_submitted.pkl
Trial-Index: 87
slurmstepd: error: *** JOB 3545423 ON c27 CANCELLED AT 2026-05-14T18:25:39 DUE TO TIME LIMIT ***
⚠ Job 3545421 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545421/3545421_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545421/3545421_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:21:16,304) - Starting with JobEnvironment(job_id=3545421, hostname=c106, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:21:16,306) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545421/3545421_submitted.pkl
Trial-Index: 94
slurmstepd: error: *** JOB 3545421 ON c106 CANCELLED AT 2026-05-14T18:21:39 DUE TO TIME LIMIT ***
⚠ Job 3545424 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545424/3545424_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545424/3545424_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:26:52,112) - Starting with JobEnvironment(job_id=3545424, hostname=c121, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:26:52,114) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545424/3545424_submitted.pkl
Trial-Index: 83
slurmstepd: error: *** JOB 3545424 ON c121 CANCELLED AT 2026-05-14T18:27:09 DUE TO TIME LIMIT ***
SOBOL, failed: 72, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running/timeout 6/1 = ∑7/20, waiting for 7 jobs : 2%|░░░░░░░░░░| 21/1000 [92:46:47<1929:40:22, 7095.84s/it]
⚠ Job 3545426 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545426/3545426_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545426/3545426_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:32:58,049) - Starting with JobEnvironment(job_id=3545426, hostname=c9, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:32:58,052) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545426/3545426_submitted.pkl
Trial-Index: 80
slurmstepd: error: *** JOB 3545426 ON c9 CANCELLED AT 2026-05-14T18:33:09 DUE TO TIME LIMIT ***
SOBOL, failed: 73, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 6 = ∑6/20, waiting for 6 jobs : 2%|░░░░░░░░░░| 21/1000 [92:47:09<1929:40:22, 7095.84s/it]
⚠ Job 3545428 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545428/3545428_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545428/3545428_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:40:34,210) - Starting with JobEnvironment(job_id=3545428, hostname=c133, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:40:34,212) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545428/3545428_submitted.pkl
Trial-Index: 86
slurmstepd: error: *** JOB 3545428 ON c133 CANCELLED AT 2026-05-14T18:40:39 DUE TO TIME LIMIT ***
⚠ Job 3545429 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545429/3545429_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545429/3545429_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:42:35,700) - Starting with JobEnvironment(job_id=3545429, hostname=c54, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:42:35,702) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545429/3545429_submitted.pkl
Trial-Index: 91
slurmstepd: error: *** JOB 3545429 ON c54 CANCELLED AT 2026-05-14T18:42:39 DUE TO TIME LIMIT ***
⚠ Job 3545430 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545430/3545430_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545430/3545430_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:43:05,647) - Starting with JobEnvironment(job_id=3545430, hostname=c123, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:43:05,649) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545430/3545430_submitted.pkl
Trial-Index: 81
slurmstepd: error: *** JOB 3545430 ON c123 CANCELLED AT 2026-05-14T18:43:09 DUE TO TIME LIMIT ***
⚠ Job 3545432 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545432/3545432_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545432/3545432_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:48:08,922) - Starting with JobEnvironment(job_id=3545432, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:48:08,923) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545432/3545432_submitted.pkl
Trial-Index: 99
slurmstepd: error: *** JOB 3545432 ON c108 CANCELLED AT 2026-05-14T18:48:09 DUE TO TIME LIMIT ***
⚠ Job 3545433 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545433/3545433_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545433/3545433_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:48:39,488) - Starting with JobEnvironment(job_id=3545433, hostname=c7, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:48:39,490) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545433/3545433_submitted.pkl
Trial-Index: 98
slurmstepd: error: *** JOB 3545433 ON c7 CANCELLED AT 2026-05-14T18:48:39 DUE TO TIME LIMIT ***
SOBOL, failed: 78, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, timeout 1 = ∑1/20, waiting for 1 job : 2%|░░░░░░░░░░| 21/1000 [93:06:48<1929:40:22, 7095.84s/it]
⚠ Job 3545434 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545434/3545434_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545434/3545434_0_log.out
----------------------------------------
submitit INFO (2026-05-14 06:50:11,616) - Starting with JobEnvironment(job_id=3545434, hostname=c49, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-14 06:50:11,619) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt_1/runs/additionLLM0/0/single_runs/3545434/3545434_submitted.pkl
Trial-Index: 97
slurmstepd: error: *** JOB 3545434 ON c49 CANCELLED AT 2026-05-14T18:50:39 DUE TO TIME LIMIT ***
BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 6 = ∑6/20, new result: VAL_LOSS: 0.680920, PARAMS: 3012.000000: 2%|░░░░░░░░░░| 21/1000 [99:41:06<1929:40:22, 7095.84s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
BOTORCH_MODULAR, failed: 79, 22 done, ▇▇▇▇▇▇▇▇▇▇▇▇▇█▁, running 5 = ∑5/20, new result: VAL_LOSS: 0.680991, PARAMS: 3012.000000: 2%|░░░░░░░░░░| 22/1000 [99:48:30<4522:23:52, 16646.86s/it]notify-send: /software/genoa/r24.04/util-linux/2.39-GCCcore-12.3.0/lib/libmount.so.1: version `MOUNT_2_40' not found (required by /lib64/libgio-2.0.so.0)
BOTORCH_MODULAR, failed: 79, 22 done, ▇▇▇▇▇▇▇▇▇▇▇▇▇█▁, running 5 = ∑5/20, new result: VAL_LOSS: 0.680991, PARAMS: 3012.000000: 2%|░░░░░░░░░░| 23/1000 [99:48:31<3198:14:32, 11784.72s/it]
2026-05-10 21:53:07 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, Started OmniOpt2 run...
2026-05-10 21:53:08 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #1/20
2026-05-10 21:53:08 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #2/20
2026-05-10 21:53:08 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #3/20
2026-05-10 21:53:08 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #4/20
2026-05-10 21:53:08 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #5/20
2026-05-10 21:53:08 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #6/20
2026-05-10 21:53:08 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #7/20
2026-05-10 21:53:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #8/20
2026-05-10 21:53:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #9/20
2026-05-10 21:53:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #10/20
2026-05-10 21:53:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #11/20
2026-05-10 21:53:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #12/20
2026-05-10 21:53:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #13/20
2026-05-10 21:53:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #14/20
2026-05-10 21:53:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #15/20
2026-05-10 21:53:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #16/20
2026-05-10 21:53:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #17/20
2026-05-10 21:53:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #18/20
2026-05-10 21:53:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #19/20
2026-05-10 21:53:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, getting new HP set #20/20
2026-05-10 21:53:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, requested 20 jobs, got 20, 0.19 s/job
2026-05-10 21:53:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #1/20 start
2026-05-10 21:53:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #2/20 start
2026-05-10 21:53:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #3/20 start
2026-05-10 21:53:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #4/20 start
2026-05-10 21:53:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #5/20 start
2026-05-10 21:53:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #6/20 start
2026-05-10 21:53:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #7/20 start
2026-05-10 21:53:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #8/20 start
2026-05-10 21:53:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #9/20 start
2026-05-10 21:53:16 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #10/20 start
2026-05-10 21:53:16 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #11/20 start
2026-05-10 21:53:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #12/20 start
2026-05-10 21:53:18 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #13/20 start
2026-05-10 21:53:18 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #14/20 start
2026-05-10 21:53:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #15/20 start
2026-05-10 21:53:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #16/20 start
2026-05-10 21:53:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #17/20 start
2026-05-10 21:53:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #18/20 start
2026-05-10 21:53:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #19/20 start
2026-05-10 21:53:27 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, eval #20/20 start
2026-05-10 21:53:28 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, starting new job
2026-05-10 21:53:28 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 1 = ∑1/20, started new job
2026-05-10 21:53:28 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 2 = ∑2/20, started new job
2026-05-10 21:53:28 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 2 = ∑2/20, starting new job
2026-05-10 21:53:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 3 = ∑3/20, started new job
2026-05-10 21:53:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 3 = ∑3/20, starting new job
2026-05-10 21:53:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 4 = ∑4/20, started new job
2026-05-10 21:53:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 4 = ∑4/20, starting new job
2026-05-10 21:53:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 5 = ∑5/20, started new job
2026-05-10 21:53:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 6 = ∑6/20, started new job
2026-05-10 21:53:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 7 = ∑7/20, started new job
2026-05-10 21:53:30 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 8 = ∑8/20, started new job
2026-05-10 21:53:30 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 9 = ∑9/20, started new job
2026-05-10 21:53:30 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 11 = ∑11/20, started new job
2026-05-10 21:53:30 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 12 = ∑12/20, started new job
2026-05-10 21:53:30 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 13 = ∑13/20, started new job
2026-05-10 21:53:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 14 = ∑14/20, started new job
2026-05-10 21:53:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 15 = ∑15/20, started new job
2026-05-10 21:53:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 16 = ∑16/20, started new job
2026-05-10 21:53:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, unknown 17 = ∑17/20, started new job
2026-05-10 21:53:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, pending/unknown 17/1 = ∑18/20, started new job
2026-05-10 21:53:32 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, pending/unknown 17/2 = ∑19/20, started new job
2026-05-10 21:53:33 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, pending/unknown 17/3 = ∑20/20, started new job
2026-05-10 21:53:34 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, pending/unknown 17/3 = ∑20/20, waiting for 20 jobs
2026-05-10 21:53:40 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-10 22:21:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, running/pending 2/18 = ∑20/20, waiting for 20 jobs
2026-05-10 22:31:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, running/pending 7/13 = ∑20/20, waiting for 20 jobs
2026-05-10 22:41:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, running/pending 10/10 = ∑20/20, waiting for 20 jobs
2026-05-10 22:51:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, running 20 = ∑20/20, waiting for 20 jobs
2026-05-10 23:40:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, running 20 = ∑20/20, new result: VAL_LOSS: 0.022503, PARAMS: 70022.000000
2026-05-10 23:40:20 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, 1 done, running 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-10 23:40:20 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, 1 done, running 19 = ∑19/20, waiting for 19 jobs
2026-05-11 06:41:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, 1 done, running 19 = ∑19/20, new result: VAL_LOSS: 0.007289, PARAMS: 30922.000000
2026-05-11 06:41:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, 2 done, ▁█, running 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-11 06:41:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, 2 done, ▁█, running 18 = ∑18/20, waiting for 18 jobs
2026-05-11 08:06:34 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, 2 done, ▁█, running 18 = ∑18/20, new result: VAL_LOSS: 0.006747, PARAMS: 34121.000000
2026-05-11 08:06:44 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, 3 done, ▁▇█, running 17 = ∑17/20, waiting for 18 jobs, finished 1 job
2026-05-11 08:06:44 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, 3 done, ▁▇█, running 17 = ∑17/20, waiting for 17 jobs
2026-05-11 10:22:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, 3 done, ▁▇█, timeout/running 1/16 = ∑17/20, job_failed
2026-05-11 10:22:27 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 1, 3 done, ▁▇█, running 16 = ∑16/20, waiting for 17 jobs, finished 1 job
2026-05-11 10:22:27 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 1, 3 done, ▁▇█, running 16 = ∑16/20, waiting for 16 jobs
2026-05-11 10:32:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 1, 3 done, ▁▇█, timeout/running 5/11 = ∑16/20, waiting for 16 jobs
2026-05-11 10:32:33 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 1, 3 done, ▁▇█, timeout/running 5/11 = ∑16/20, job_failed
2026-05-11 10:32:35 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 5, 3 done, ▁▇█, timeout/running 1/11 = ∑12/20, job_failed
2026-05-11 10:32:36 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 6, 3 done, ▁▇█, running 11 = ∑11/20, waiting for 16 jobs, finished 5 jobs
2026-05-11 10:32:36 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 6, 3 done, ▁▇█, running 11 = ∑11/20, waiting for 11 jobs
2026-05-11 10:42:18 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 6, 3 done, ▁▇█, timeout/running 3/8 = ∑11/20, waiting for 11 jobs
2026-05-11 10:42:34 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 6, 3 done, ▁▇█, timeout/running 3/8 = ∑11/20, job_failed
2026-05-11 10:42:36 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 9, 3 done, ▁▇█, running 8 = ∑8/20, waiting for 11 jobs, finished 3 jobs
2026-05-11 10:42:36 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 9, 3 done, ▁▇█, running 8 = ∑8/20, waiting for 8 jobs
2026-05-11 10:52:35 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 9, 3 done, ▁▇█, timeout 8 = ∑8/20, job_failed
2026-05-11 10:52:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, waiting for 8 jobs, finished 8 jobs
2026-05-11 10:52:46 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #1/20
2026-05-11 10:52:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #2/20
2026-05-11 10:52:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #3/20
2026-05-11 10:52:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #4/20
2026-05-11 10:52:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #5/20
2026-05-11 10:52:49 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #6/20
2026-05-11 10:52:49 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #7/20
2026-05-11 10:52:50 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #8/20
2026-05-11 10:52:50 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #9/20
2026-05-11 10:52:50 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #10/20
2026-05-11 10:52:51 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #11/20
2026-05-11 10:52:51 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #12/20
2026-05-11 10:52:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #13/20
2026-05-11 10:52:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #14/20
2026-05-11 10:52:53 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #15/20
2026-05-11 10:52:53 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #16/20
2026-05-11 10:52:53 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #17/20
2026-05-11 10:52:54 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #18/20
2026-05-11 10:52:54 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #19/20
2026-05-11 10:52:56 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, getting new HP set #20/20
2026-05-11 10:52:56 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, requested 20 jobs, got 20, 0.54 s/job
2026-05-11 10:52:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #1/20 start
2026-05-11 10:52:59 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #2/20 start
2026-05-11 10:53:00 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #3/20 start
2026-05-11 10:53:01 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #4/20 start
2026-05-11 10:53:03 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #5/20 start
2026-05-11 10:53:04 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #6/20 start
2026-05-11 10:53:05 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #7/20 start
2026-05-11 10:53:07 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #8/20 start
2026-05-11 10:53:09 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #9/20 start
2026-05-11 10:53:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #10/20 start
2026-05-11 10:53:14 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #11/20 start
2026-05-11 10:53:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #12/20 start
2026-05-11 10:53:16 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #13/20 start
2026-05-11 10:53:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #14/20 start
2026-05-11 10:53:19 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #15/20 start
2026-05-11 10:53:20 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #16/20 start
2026-05-11 10:53:21 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #17/20 start
2026-05-11 10:53:22 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #18/20 start
2026-05-11 10:53:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #19/20 start
2026-05-11 10:53:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, eval #20/20 start
2026-05-11 10:53:34 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, starting new job
2026-05-11 10:53:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending 11 = ∑11/20, started new job
2026-05-11 10:53:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending/unknown 12/1 = ∑13/20, started new job
2026-05-11 10:53:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending/unknown 13/1 = ∑14/20, started new job
2026-05-11 10:53:44 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending/unknown 13/3 = ∑16/20, started new job
2026-05-11 10:53:44 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending 16 = ∑16/20, started new job
2026-05-11 10:53:46 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending 16 = ∑16/20, starting new job
2026-05-11 10:53:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending/unknown 16/3 = ∑19/20, started new job
2026-05-11 10:53:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending/unknown 16/4 = ∑20/20, started new job
2026-05-11 10:53:50 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending/unknown 16/4 = ∑20/20, waiting for 20 jobs
2026-05-11 10:53:51 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-12 00:38:23 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, running/pending 1/19 = ∑20/20, waiting for 20 jobs
2026-05-12 03:38:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, running/pending 6/14 = ∑20/20, waiting for 20 jobs
2026-05-12 03:48:33 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, running 20 = ∑20/20, waiting for 20 jobs
2026-05-12 04:10:02 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 3 done, ▁▇█, running 20 = ∑20/20, new result: VAL_LOSS: 0.034741, PARAMS: 34745.000000
2026-05-12 04:10:18 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 4 done, ▄▇█▁, running 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-12 04:10:18 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 4 done, ▄▇█▁, running 19 = ∑19/20, waiting for 19 jobs
2026-05-12 04:47:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 4 done, ▄▇█▁, running 19 = ∑19/20, new result: VAL_LOSS: 0.022921, PARAMS: 30668.000000
2026-05-12 04:47:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 5 done, ▄▇█▁▃, running 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-12 04:47:25 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 5 done, ▄▇█▁▃, running 18 = ∑18/20, waiting for 18 jobs
2026-05-12 11:37:53 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 5 done, ▄▇█▁▃, running 18 = ∑18/20, new result: VAL_LOSS: 0.010711, PARAMS: 28772.000000
2026-05-12 11:38:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 6 done, ▄▇█▁▇▃, running 17 = ∑17/20, waiting for 18 jobs, finished 1 job
2026-05-12 11:38:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 6 done, ▄▇█▁▇▃, running 17 = ∑17/20, waiting for 17 jobs
2026-05-12 12:38:56 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 6 done, ▄▇█▁▇▃, timeout/running 1/16 = ∑17/20, waiting for 17 jobs
2026-05-12 12:39:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 17, 6 done, ▄▇█▁▇▃, timeout/running 1/16 = ∑17/20, job_failed
2026-05-12 12:39:14 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 18, 6 done, ▄▇█▁▇▃, running 16 = ∑16/20, waiting for 17 jobs, finished 1 job
2026-05-12 12:39:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 18, 6 done, ▄▇█▁▇▃, running 16 = ∑16/20, waiting for 16 jobs
2026-05-12 15:39:21 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 18, 6 done, ▄▇█▁▇▃, timeout/running 4/12 = ∑16/20, job_failed
2026-05-12 15:39:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 22, 6 done, ▄▇█▁▇▃, running 12 = ∑12/20, waiting for 16 jobs, finished 4 jobs
2026-05-12 15:39:27 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 22, 6 done, ▄▇█▁▇▃, running 12 = ∑12/20, waiting for 12 jobs
2026-05-12 15:49:05 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 22, 6 done, ▄▇█▁▇▃, timeout 12 = ∑12/20, waiting for 12 jobs
2026-05-12 15:49:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 22, 6 done, ▄▇█▁▇▃, timeout 12 = ∑12/20, job_failed
2026-05-12 15:49:43 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, waiting for 12 jobs, finished 12 jobs
2026-05-12 15:49:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #1/20
2026-05-12 15:49:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #2/20
2026-05-12 15:49:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #3/20
2026-05-12 15:49:49 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #4/20
2026-05-12 15:49:51 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #5/20
2026-05-12 15:49:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #6/20
2026-05-12 15:49:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #7/20
2026-05-12 15:49:53 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #8/20
2026-05-12 15:49:54 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #9/20
2026-05-12 15:49:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #10/20
2026-05-12 15:49:56 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #11/20
2026-05-12 15:49:57 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #12/20
2026-05-12 15:49:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #13/20
2026-05-12 15:49:59 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #14/20
2026-05-12 15:50:00 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #15/20
2026-05-12 15:50:00 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #16/20
2026-05-12 15:50:01 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #17/20
2026-05-12 15:50:02 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #18/20
2026-05-12 15:50:03 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #19/20
2026-05-12 15:50:04 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, getting new HP set #20/20
2026-05-12 15:50:05 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, requested 20 jobs, got 20, 0.93 s/job
2026-05-12 15:50:09 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #1/20 start
2026-05-12 15:50:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #2/20 start
2026-05-12 15:50:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #3/20 start
2026-05-12 15:50:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #4/20 start
2026-05-12 15:50:20 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #5/20 start
2026-05-12 15:50:23 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #6/20 start
2026-05-12 15:50:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #7/20 start
2026-05-12 15:50:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #8/20 start
2026-05-12 15:50:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #9/20 start
2026-05-12 15:50:34 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #10/20 start
2026-05-12 15:50:36 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #11/20 start
2026-05-12 15:50:39 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #12/20 start
2026-05-12 15:50:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #13/20 start
2026-05-12 15:50:43 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #14/20 start
2026-05-12 15:50:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #15/20 start
2026-05-12 15:50:50 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #16/20 start
2026-05-12 15:50:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #17/20 start
2026-05-12 15:50:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #18/20 start
2026-05-12 15:50:57 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #19/20 start
2026-05-12 15:51:00 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, eval #20/20 start
2026-05-12 15:51:19 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, starting new job
2026-05-12 15:51:38 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, pending 16 = ∑16/20, started new job
2026-05-12 15:51:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, pending 16 = ∑16/20, starting new job
2026-05-12 15:51:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, pending 20 = ∑20/20, started new job
2026-05-12 15:51:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-12 22:30:02 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2026-05-13 01:30:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, running/pending 10/10 = ∑20/20, waiting for 20 jobs
2026-05-13 02:39:38 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 6 done, ▄▇█▁▇▃, running/pending 10/10 = ∑20/20, new result: VAL_LOSS: 0.019894, PARAMS: 20582.000000
2026-05-13 02:40:02 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 7 done, ▄▇█▁▇▃▄, running/pending 9/10 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-13 02:40:03 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 7 done, ▄▇█▁▇▃▄, running/pending 9/10 = ∑19/20, waiting for 19 jobs
2026-05-13 05:10:23 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 7 done, ▄▇█▁▇▃▄, running/pending 10/9 = ∑19/20, waiting for 19 jobs
2026-05-13 05:20:22 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 7 done, ▄▇█▁▇▃▄, running/pending 11/8 = ∑19/20, waiting for 19 jobs
2026-05-13 05:30:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 7 done, ▄▇█▁▇▃▄, running/pending 14/5 = ∑19/20, waiting for 19 jobs
2026-05-13 05:40:23 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 7 done, ▄▇█▁▇▃▄, running/pending 17/2 = ∑19/20, waiting for 19 jobs
2026-05-13 05:50:25 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 7 done, ▄▇█▁▇▃▄, running 19 = ∑19/20, waiting for 19 jobs
2026-05-13 06:12:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 7 done, ▄▇█▁▇▃▄, running 19 = ∑19/20, new result: VAL_LOSS: 0.022520, PARAMS: 79270.000000
2026-05-13 06:13:23 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 8 done, ▄▇█▁▇▃▄▄, running 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-13 06:13:23 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 8 done, ▄▇█▁▇▃▄▄, running 18 = ∑18/20, waiting for 18 jobs
2026-05-13 07:57:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 8 done, ▄▇█▁▇▃▄▄, running 18 = ∑18/20, new result: VAL_LOSS: 0.010525, PARAMS: 69292.000000
2026-05-13 07:58:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 9 done, ▄▇█▁▇▃▇▄▄, running 17 = ∑17/20, waiting for 18 jobs, finished 1 job
2026-05-13 07:58:18 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 9 done, ▄▇█▁▇▃▇▄▄, running 17 = ∑17/20, waiting for 17 jobs
2026-05-13 09:32:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 9 done, ▄▇█▁▇▃▇▄▄, running 17 = ∑17/20, new result: VAL_LOSS: 0.011913, PARAMS: 9652.000000
2026-05-13 09:32:53 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 10 done, ▄▇█▁▇▃▇▄▄▆, running 16 = ∑16/20, waiting for 17 jobs, finished 1 job
2026-05-13 09:32:54 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 10 done, ▄▇█▁▇▃▇▄▄▆, running 16 = ∑16/20, waiting for 16 jobs
2026-05-13 10:30:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 10 done, ▄▇█▁▇▃▇▄▄▆, timeout/running 5/11 = ∑16/20, waiting for 16 jobs
2026-05-13 10:31:00 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 34, 10 done, ▄▇█▁▇▃▇▄▄▆, timeout/running 5/11 = ∑16/20, job_failed
2026-05-13 10:31:07 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 39, 10 done, ▄▇█▁▇▃▇▄▄▆, running 11 = ∑11/20, waiting for 16 jobs, finished 5 jobs
2026-05-13 10:31:07 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 39, 10 done, ▄▇█▁▇▃▇▄▄▆, running 11 = ∑11/20, waiting for 11 jobs
2026-05-13 13:31:09 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 39, 10 done, ▄▇█▁▇▃▇▄▄▆, timeout/running 2/9 = ∑11/20, job_failed
2026-05-13 13:31:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 41, 10 done, ▄▇█▁▇▃▇▄▄▆, running 9 = ∑9/20, waiting for 11 jobs, finished 2 jobs
2026-05-13 13:31:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 41, 10 done, ▄▇█▁▇▃▇▄▄▆, running 9 = ∑9/20, waiting for 9 jobs
2026-05-13 17:11:22 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 41, 10 done, ▄▇█▁▇▃▇▄▄▆, running/timeout 8/1 = ∑9/20, job_failed
2026-05-13 17:11:25 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 42, 10 done, ▄▇█▁▇▃▇▄▄▆, running 8 = ∑8/20, waiting for 9 jobs, finished 1 job
2026-05-13 17:11:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 42, 10 done, ▄▇█▁▇▃▇▄▄▆, running 8 = ∑8/20, waiting for 8 jobs
2026-05-13 17:21:07 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 42, 10 done, ▄▇█▁▇▃▇▄▄▆, running/timeout 7/1 = ∑8/20, waiting for 8 jobs
2026-05-13 17:21:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 42, 10 done, ▄▇█▁▇▃▇▄▄▆, running/timeout 7/1 = ∑8/20, job_failed
2026-05-13 17:21:27 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 43, 10 done, ▄▇█▁▇▃▇▄▄▆, running 7 = ∑7/20, waiting for 8 jobs, finished 1 job
2026-05-13 17:21:28 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 43, 10 done, ▄▇█▁▇▃▇▄▄▆, running 7 = ∑7/20, waiting for 7 jobs
2026-05-13 17:31:25 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 43, 10 done, ▄▇█▁▇▃▇▄▄▆, timeout/running 3/4 = ∑7/20, job_failed
2026-05-13 17:31:30 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 46, 10 done, ▄▇█▁▇▃▇▄▄▆, running 4 = ∑4/20, waiting for 7 jobs, finished 3 jobs
2026-05-13 17:31:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 46, 10 done, ▄▇█▁▇▃▇▄▄▆, running 4 = ∑4/20, waiting for 4 jobs
2026-05-13 17:41:09 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 46, 10 done, ▄▇█▁▇▃▇▄▄▆, timeout/running 2/2 = ∑4/20, waiting for 4 jobs
2026-05-13 17:41:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 46, 10 done, ▄▇█▁▇▃▇▄▄▆, timeout/running 2/2 = ∑4/20, job_failed
2026-05-13 17:41:30 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 48, 10 done, ▄▇█▁▇▃▇▄▄▆, running 2 = ∑2/20, waiting for 4 jobs, finished 2 jobs
2026-05-13 17:41:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 48, 10 done, ▄▇█▁▇▃▇▄▄▆, running 2 = ∑2/20, waiting for 2 jobs
2026-05-13 17:51:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 48, 10 done, ▄▇█▁▇▃▇▄▄▆, timeout 2 = ∑2/20, job_failed
2026-05-13 17:51:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, waiting for 2 jobs, finished 2 jobs
2026-05-13 17:51:34 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #1/20
2026-05-13 17:51:35 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #2/20
2026-05-13 17:51:36 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #3/20
2026-05-13 17:51:38 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #4/20
2026-05-13 17:51:39 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #5/20
2026-05-13 17:51:40 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #6/20
2026-05-13 17:51:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #7/20
2026-05-13 17:51:42 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #8/20
2026-05-13 17:51:43 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #9/20
2026-05-13 17:51:44 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #10/20
2026-05-13 17:51:46 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #11/20
2026-05-13 17:51:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #12/20
2026-05-13 17:51:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #13/20
2026-05-13 17:51:49 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #14/20
2026-05-13 17:51:50 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #15/20
2026-05-13 17:51:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #16/20
2026-05-13 17:51:53 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #17/20
2026-05-13 17:51:54 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #18/20
2026-05-13 17:51:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #19/20
2026-05-13 17:51:56 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, getting new HP set #20/20
2026-05-13 17:51:57 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, requested 20 jobs, got 20, 1.15 s/job
2026-05-13 17:52:01 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #1/20 start
2026-05-13 17:52:04 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #2/20 start
2026-05-13 17:52:08 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #3/20 start
2026-05-13 17:52:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #4/20 start
2026-05-13 17:52:14 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #5/20 start
2026-05-13 17:52:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #6/20 start
2026-05-13 17:52:21 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #7/20 start
2026-05-13 17:52:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #8/20 start
2026-05-13 17:52:27 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #9/20 start
2026-05-13 17:52:30 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #10/20 start
2026-05-13 17:52:34 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #11/20 start
2026-05-13 17:52:38 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #12/20 start
2026-05-13 17:52:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #13/20 start
2026-05-13 17:52:45 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #14/20 start
2026-05-13 17:52:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #15/20 start
2026-05-13 17:52:51 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #16/20 start
2026-05-13 17:52:54 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #17/20 start
2026-05-13 17:52:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #18/20 start
2026-05-13 17:53:01 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #19/20 start
2026-05-13 17:53:04 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, eval #20/20 start
2026-05-13 17:53:29 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, starting new job
2026-05-13 17:53:57 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, pending 15 = ∑15/20, started new job
2026-05-13 17:53:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, pending 16 = ∑16/20, started new job
2026-05-13 17:54:09 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, pending 16 = ∑16/20, starting new job
2026-05-13 17:54:16 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, pending 20 = ∑20/20, started new job
2026-05-13 17:54:19 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-13 17:55:06 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, running/pending 4/16 = ∑20/20, waiting for 20 jobs
2026-05-13 17:56:03 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, running/pending 10/10 = ∑20/20, waiting for 20 jobs
2026-05-13 17:57:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, running/pending 13/7 = ∑20/20, waiting for 20 jobs
2026-05-13 18:01:44 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, running/pending 19/1 = ∑20/20, waiting for 20 jobs
2026-05-13 18:09:19 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, running 20 = ∑20/20, waiting for 20 jobs
2026-05-13 21:02:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 10 done, ▄▇█▁▇▃▇▄▄▆, running 20 = ∑20/20, new result: VAL_LOSS: 0.027349, PARAMS: 28106.000000
2026-05-13 21:02:45 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 11 done, ▄▇█▁▇▃▇▄▄▆▂, running 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-13 21:02:45 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 11 done, ▄▇█▁▇▃▇▄▄▆▂, running 19 = ∑19/20, waiting for 19 jobs
2026-05-13 21:17:30 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 11 done, ▄▇█▁▇▃▇▄▄▆▂, running 19 = ∑19/20, new result: VAL_LOSS: 0.008205, PARAMS: 32913.000000
2026-05-13 21:17:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 12 done, ▄▇█▁▇▃▇▄▄▆▇▂, running 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-13 21:17:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 12 done, ▄▇█▁▇▃▇▄▄▆▇▂, running 18 = ∑18/20, waiting for 18 jobs
2026-05-13 21:51:05 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 12 done, ▄▇█▁▇▃▇▄▄▆▇▂, running 18 = ∑18/20, new result: VAL_LOSS: 0.017069, PARAMS: 27232.000000
2026-05-13 21:51:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 13 done, ▄▇█▁▇▃▇▄▄▆▇▂▅, running 17 = ∑17/20, waiting for 18 jobs, finished 1 job
2026-05-13 21:51:32 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 13 done, ▄▇█▁▇▃▇▄▄▆▇▂▅, running 17 = ∑17/20, waiting for 17 jobs
2026-05-13 23:45:22 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 13 done, ▄▇█▁▇▃▇▄▄▆▇▂▅, running 17 = ∑17/20, new result: VAL_LOSS: 0.012173, PARAMS: 42708.000000
2026-05-13 23:45:50 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 14 done, ▄▇█▁▇▃▇▄▄▆▇▂▅▆, running 16 = ∑16/20, waiting for 17 jobs, finished 1 job
2026-05-13 23:45:51 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 14 done, ▄▇█▁▇▃▇▄▄▆▇▂▅▆, running 16 = ∑16/20, waiting for 16 jobs
2026-05-14 00:31:19 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 14 done, ▄▇█▁▇▃▇▄▄▆▇▂▅▆, running 16 = ∑16/20, new result: VAL_LOSS: 0.026844, PARAMS: 17348.000000
2026-05-14 00:31:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 15 done, ▄▇█▁▇▃▇▄▄▆▂▇▂▅▆, running 15 = ∑15/20, waiting for 16 jobs, finished 1 job
2026-05-14 00:31:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 15 done, ▄▇█▁▇▃▇▄▄▆▂▇▂▅▆, running 15 = ∑15/20, waiting for 15 jobs
2026-05-14 03:53:43 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 15 done, ▄▇█▁▇▃▇▄▄▆▂▇▂▅▆, running 15 = ∑15/20, new result: VAL_LOSS: 0.011546, PARAMS: 45766.000000
2026-05-14 03:54:16 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running 14 = ∑14/20, waiting for 15 jobs, finished 1 job
2026-05-14 03:54:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running 14 = ∑14/20, waiting for 14 jobs
2026-05-14 06:00:02 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, timeout/running 11/3 = ∑14/20, waiting for 14 jobs
2026-05-14 06:00:28 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 50, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, timeout/running 11/3 = ∑14/20, job_failed
2026-05-14 06:00:42 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 61, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running 3 = ∑3/20, waiting for 14 jobs, finished 11 jobs
2026-05-14 06:00:43 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 61, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running 3 = ∑3/20, waiting for 3 jobs
2026-05-14 06:10:03 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 61, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, timeout 3 = ∑3/20, waiting for 3 jobs
2026-05-14 06:10:22 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 61, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, timeout 3 = ∑3/20, job_failed
2026-05-14 06:10:28 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, waiting for 3 jobs, finished 3 jobs
2026-05-14 06:10:32 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #1/20
2026-05-14 06:10:34 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #2/20
2026-05-14 06:10:35 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #3/20
2026-05-14 06:10:36 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #4/20
2026-05-14 06:10:38 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #5/20
2026-05-14 06:10:39 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #6/20
2026-05-14 06:10:40 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #7/20
2026-05-14 06:10:41 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #8/20
2026-05-14 06:10:42 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #9/20
2026-05-14 06:10:43 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #10/20
2026-05-14 06:10:44 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #11/20
2026-05-14 06:10:46 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #12/20
2026-05-14 06:10:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #13/20
2026-05-14 06:10:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #14/20
2026-05-14 06:10:49 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #15/20
2026-05-14 06:10:50 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #16/20
2026-05-14 06:10:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #17/20
2026-05-14 06:10:54 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #18/20
2026-05-14 06:10:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #19/20
2026-05-14 06:10:56 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, getting new HP set #20/20
2026-05-14 06:10:57 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, requested 20 jobs, got 20, 1.27 s/job
2026-05-14 06:11:03 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #1/20 start
2026-05-14 06:11:07 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #2/20 start
2026-05-14 06:11:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #3/20 start
2026-05-14 06:11:14 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #4/20 start
2026-05-14 06:11:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #5/20 start
2026-05-14 06:11:21 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #6/20 start
2026-05-14 06:11:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #7/20 start
2026-05-14 06:11:28 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #8/20 start
2026-05-14 06:11:31 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #9/20 start
2026-05-14 06:11:36 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #10/20 start
2026-05-14 06:11:40 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #11/20 start
2026-05-14 06:11:45 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #12/20 start
2026-05-14 06:11:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #13/20 start
2026-05-14 06:11:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #14/20 start
2026-05-14 06:11:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #15/20 start
2026-05-14 06:11:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #16/20 start
2026-05-14 06:12:02 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #17/20 start
2026-05-14 06:12:05 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #18/20 start
2026-05-14 06:12:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #19/20 start
2026-05-14 06:12:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, eval #20/20 start
2026-05-14 06:12:43 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, starting new job
2026-05-14 06:13:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, pending/running 14/1 = ∑15/20, started new job
2026-05-14 06:13:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, pending/running 15/1 = ∑16/20, started new job
2026-05-14 06:13:26 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, pending/running 15/1 = ∑16/20, starting new job
2026-05-14 06:13:34 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, pending/running 19/1 = ∑20/20, started new job
2026-05-14 06:13:37 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, pending/running 19/1 = ∑20/20, waiting for 20 jobs
2026-05-14 06:13:42 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running/pending 2/18 = ∑20/20, waiting for 20 jobs
2026-05-14 06:17:23 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running/pending 3/17 = ∑20/20, waiting for 20 jobs
2026-05-14 06:21:19 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running/pending 7/13 = ∑20/20, waiting for 20 jobs
2026-05-14 06:29:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running/pending 10/10 = ∑20/20, waiting for 20 jobs
2026-05-14 06:39:14 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running/pending 13/7 = ∑20/20, waiting for 20 jobs
2026-05-14 06:49:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running/pending 19/1 = ∑20/20, waiting for 20 jobs
2026-05-14 06:59:14 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running 20 = ∑20/20, waiting for 20 jobs
2026-05-14 09:24:22 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 16 done, ▇█▁▇▃▇▄▄▆▂▇▂▆▅▆, running 20 = ∑20/20, new result: VAL_LOSS: 0.030514, PARAMS: 18878.000000
2026-05-14 09:24:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 17 done, █▁▇▃▇▄▄▆▂▇▂▆▅▆▂, running 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-14 09:24:59 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 17 done, █▁▇▃▇▄▄▆▂▇▂▆▅▆▂, running 19 = ∑19/20, waiting for 19 jobs
2026-05-14 10:05:01 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 17 done, █▁▇▃▇▄▄▆▂▇▂▆▅▆▂, running 19 = ∑19/20, new result: VAL_LOSS: 0.012584, PARAMS: 3038.000000
2026-05-14 10:05:32 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 18 done, ▁▇▄▇▄▄▇▃█▂▇▅▆▂▆, running 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-14 10:05:33 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 18 done, ▁▇▄▇▄▄▇▃█▂▇▅▆▂▆, running 18 = ∑18/20, waiting for 18 jobs
2026-05-14 11:37:35 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 18 done, ▁▇▄▇▄▄▇▃█▂▇▅▆▂▆, running 18 = ∑18/20, new result: VAL_LOSS: 0.017702, PARAMS: 94528.000000
2026-05-14 11:38:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 19 done, ▇▃▇▄▃▆▂█▁▆▅▆▅▁▆, running 17 = ∑17/20, waiting for 18 jobs, finished 1 job
2026-05-14 11:38:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 19 done, ▇▃▇▄▃▆▂█▁▆▅▆▅▁▆, running 17 = ∑17/20, waiting for 17 jobs
2026-05-14 12:46:19 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 19 done, ▇▃▇▄▃▆▂█▁▆▅▆▅▁▆, running 17 = ∑17/20, new result: VAL_LOSS: 0.007035, PARAMS: 6835.000000
2026-05-14 12:46:54 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 20 done, ▃▆▄▃▆▂▇▁▆▅▆▄▁▆█, running 16 = ∑16/20, waiting for 17 jobs, finished 1 job
2026-05-14 12:46:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 20 done, ▃▆▄▃▆▂▇▁▆▅▆▄▁▆█, running 16 = ∑16/20, waiting for 16 jobs
2026-05-14 14:45:33 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 20 done, ▃▆▄▃▆▂▇▁▆▅▆▄▁▆█, running 16 = ∑16/20, new result: VAL_LOSS: 0.011096, PARAMS: 69538.000000
2026-05-14 14:46:10 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 15 = ∑15/20, waiting for 16 jobs, finished 1 job
2026-05-14 14:46:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 15 = ∑15/20, waiting for 15 jobs
2026-05-14 18:20:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 64, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, timeout/running 3/12 = ∑15/20, job_failed
2026-05-14 18:20:18 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 67, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 12 = ∑12/20, waiting for 15 jobs, finished 3 jobs
2026-05-14 18:20:19 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 67, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 12 = ∑12/20, waiting for 12 jobs
2026-05-14 18:30:14 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 67, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, timeout/running 5/7 = ∑12/20, job_failed
2026-05-14 18:30:23 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 72, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 7 = ∑7/20, waiting for 12 jobs, finished 5 jobs
2026-05-14 18:30:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 72, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 7 = ∑7/20, waiting for 7 jobs
2026-05-14 18:39:54 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 72, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running/timeout 6/1 = ∑7/20, waiting for 7 jobs
2026-05-14 18:40:11 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 72, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running/timeout 6/1 = ∑7/20, job_failed
2026-05-14 18:40:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 73, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 6 = ∑6/20, waiting for 7 jobs, finished 1 job
2026-05-14 18:40:16 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 73, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 6 = ∑6/20, waiting for 6 jobs
2026-05-14 18:50:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 73, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, timeout/running 5/1 = ∑6/20, job_failed
2026-05-14 18:50:23 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 78, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 1 = ∑1/20, waiting for 6 jobs, finished 5 jobs
2026-05-14 18:50:25 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 78, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 1 = ∑1/20, waiting for 1 job
2026-05-14 18:59:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 78, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, timeout 1 = ∑1/20, waiting for 1 job
2026-05-14 19:00:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 78, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, timeout 1 = ∑1/20, job_failed
2026-05-14 19:00:17 (d613248e-3a44-40c9-82d1-9af61b828cdd): SOBOL, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, waiting for 1 job, finished 1 job
2026-05-14 19:00:45 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, getting new HP set #1/20
2026-05-14 19:00:47 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, getting new HP set #2/20
2026-05-14 19:00:48 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, getting new HP set #3/20
2026-05-14 19:00:52 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, getting new HP set #4/20
2026-05-14 19:01:08 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, getting new HP set #5/20
2026-05-14 19:01:37 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, getting new HP set #6/20
2026-05-14 19:01:38 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, getting new HP set #7/20
2026-05-14 19:01:40 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, getting new HP set #8/20
2026-05-14 19:01:53 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, requested 20 jobs, got 7, 13.21 s/job
2026-05-14 19:01:58 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, eval #1/7 start
2026-05-14 19:02:03 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, eval #2/7 start
2026-05-14 19:02:07 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, eval #3/7 start
2026-05-14 19:02:12 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, eval #4/7 start
2026-05-14 19:02:15 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, eval #5/7 start
2026-05-14 19:02:19 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, eval #6/7 start
2026-05-14 19:02:24 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, eval #7/7 start
2026-05-14 19:02:38 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, starting new job
2026-05-14 19:02:50 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, pending 6 = ∑6/20, started new job
2026-05-14 19:02:55 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, pending 6 = ∑6/20, waiting for 6 jobs
2026-05-14 19:14:14 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 6 = ∑6/20, waiting for 6 jobs
2026-05-15 01:34:13 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 21 done, ▆▄▃▆▂▇▁▆▅▆▄▆▁▆█, running 6 = ∑6/20, new result: VAL_LOSS: 0.680920, PARAMS: 3012.000000
2026-05-15 01:35:06 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 22 done, ▇▇▇▇▇▇▇▇▇▇▇▇▇█▁, running 5 = ∑5/20, waiting for 6 jobs, finished 1 job
2026-05-15 01:35:07 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 22 done, ▇▇▇▇▇▇▇▇▇▇▇▇▇█▁, running 5 = ∑5/20, waiting for 5 jobs
2026-05-15 01:41:37 (d613248e-3a44-40c9-82d1-9af61b828cdd): BOTORCH_MODULAR, failed: 79, 22 done, ▇▇▇▇▇▇▇▇▇▇▇▇▇█▁, running 5 = ∑5/20, new result: VAL_LOSS: 0.680991, PARAMS: 3012.000000