⚠ Job 3514200 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514200/3514200_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514200/3514200_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,530) - Starting with JobEnvironment(job_id=3514200, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,535) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514200/3514200_submitted.pkl
Trial-Index: 2
slurmstepd: error: *** JOB 3514200 ON c140 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514208 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514208/3514208_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514208/3514208_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,440) - Starting with JobEnvironment(job_id=3514208, hostname=c130, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,443) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514208/3514208_submitted.pkl
Trial-Index: 5
slurmstepd: error: *** JOB 3514208 ON c130 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514213 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514213/3514213_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514213/3514213_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,675) - Starting with JobEnvironment(job_id=3514213, hostname=c6, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,681) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514213/3514213_submitted.pkl
Trial-Index: 10
slurmstepd: error: *** JOB 3514213 ON c6 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514202 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514202/3514202_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514202/3514202_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,422) - Starting with JobEnvironment(job_id=3514202, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,425) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514202/3514202_submitted.pkl
Trial-Index: 9
slurmstepd: error: *** JOB 3514202 ON c139 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514199 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514199/3514199_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514199/3514199_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:24,575) - Starting with JobEnvironment(job_id=3514199, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:24,630) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514199/3514199_submitted.pkl
Trial-Index: 4
slurmstepd: error: *** JOB 3514199 ON c140 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514204 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514204/3514204_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514204/3514204_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,406) - Starting with JobEnvironment(job_id=3514204, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,408) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514204/3514204_submitted.pkl
Trial-Index: 3
slurmstepd: error: *** JOB 3514204 ON c99 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514207 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514207/3514207_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514207/3514207_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,813) - Starting with JobEnvironment(job_id=3514207, hostname=c91, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,819) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514207/3514207_submitted.pkl
Trial-Index: 11
slurmstepd: error: *** JOB 3514207 ON c91 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514201 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514201/3514201_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514201/3514201_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,422) - Starting with JobEnvironment(job_id=3514201, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,425) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514201/3514201_submitted.pkl
Trial-Index: 0
slurmstepd: error: *** JOB 3514201 ON c139 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514217 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514217/3514217_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514217/3514217_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:29:24,899) - Starting with JobEnvironment(job_id=3514217, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:29:24,905) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514217/3514217_submitted.pkl
Trial-Index: 19
slurmstepd: error: *** JOB 3514217 ON c136 CANCELLED AT 2026-05-10T00:29:24 DUE TO TIME LIMIT ***
⚠ Job 3514210 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514210/3514210_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514210/3514210_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,706) - Starting with JobEnvironment(job_id=3514210, hostname=c114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,712) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514210/3514210_submitted.pkl
Trial-Index: 13
slurmstepd: error: *** JOB 3514210 ON c114 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514209 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514209/3514209_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514209/3514209_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,706) - Starting with JobEnvironment(job_id=3514209, hostname=c114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,711) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514209/3514209_submitted.pkl
Trial-Index: 14
slurmstepd: error: *** JOB 3514209 ON c114 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514205 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514205/3514205_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514205/3514205_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,406) - Starting with JobEnvironment(job_id=3514205, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,408) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514205/3514205_submitted.pkl
Trial-Index: 6
slurmstepd: error: *** JOB 3514205 ON c99 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514214 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514214/3514214_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514214/3514214_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,615) - Starting with JobEnvironment(job_id=3514214, hostname=c3, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,622) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514214/3514214_submitted.pkl
Trial-Index: 16
slurmstepd: error: *** JOB 3514214 ON c3 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514216 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514216/3514216_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514216/3514216_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:29:24,279) - Starting with JobEnvironment(job_id=3514216, hostname=c151, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:29:24,290) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514216/3514216_submitted.pkl
Trial-Index: 18
slurmstepd: error: *** JOB 3514216 ON c151 CANCELLED AT 2026-05-10T00:29:24 DUE TO TIME LIMIT ***
⚠ Job 3514215 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514215/3514215_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514215/3514215_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,615) - Starting with JobEnvironment(job_id=3514215, hostname=c3, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,622) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514215/3514215_submitted.pkl
Trial-Index: 17
slurmstepd: error: *** JOB 3514215 ON c3 CANCELLED AT 2026-05-10T00:28:55 DUE TO TIME LIMIT ***
⚠ Job 3514212 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514212/3514212_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514212/3514212_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,577) - Starting with JobEnvironment(job_id=3514212, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,585) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514212/3514212_submitted.pkl
Trial-Index: 12
slurmstepd: error: *** JOB 3514212 ON c27 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514198 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514198/3514198_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514198/3514198_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,752) - Starting with JobEnvironment(job_id=3514198, hostname=c154, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,758) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514198/3514198_submitted.pkl
Trial-Index: 1
slurmstepd: error: *** JOB 3514198 ON c154 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3524176 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524176/3524176_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524176/3524176_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,932) - Starting with JobEnvironment(job_id=3524176, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,934) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524176/3524176_submitted.pkl
Trial-Index: 35
slurmstepd: error: *** JOB 3524176 ON c108 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524180 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524180/3524180_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524180/3524180_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:32,579) - Starting with JobEnvironment(job_id=3524180, hostname=c25, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:32,580) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524180/3524180_submitted.pkl
Trial-Index: 27
slurmstepd: error: *** JOB 3524180 ON c25 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524177 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524177/3524177_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524177/3524177_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,879) - Starting with JobEnvironment(job_id=3524177, hostname=c101, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,881) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524177/3524177_submitted.pkl
Trial-Index: 31
slurmstepd: error: *** JOB 3524177 ON c101 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524178 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524178/3524178_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524178/3524178_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:32,253) - Starting with JobEnvironment(job_id=3524178, hostname=c76, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:32,257) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524178/3524178_submitted.pkl
Trial-Index: 26
slurmstepd: error: *** JOB 3524178 ON c76 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524184 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524184/3524184_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524184/3524184_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:32:02,810) - Starting with JobEnvironment(job_id=3524184, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:32:02,812) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524184/3524184_submitted.pkl
Trial-Index: 36
slurmstepd: error: *** JOB 3524184 ON c144 CANCELLED AT 2026-05-10T12:32:15 DUE TO TIME LIMIT ***
⚠ Job 3524170 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524170/3524170_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524170/3524170_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,610) - Starting with JobEnvironment(job_id=3524170, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,612) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524170/3524170_submitted.pkl
Trial-Index: 22
slurmstepd: error: *** JOB 3524170 ON c139 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524168 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524168/3524168_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524168/3524168_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,548) - Starting with JobEnvironment(job_id=3524168, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,550) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524168/3524168_submitted.pkl
Trial-Index: 20
slurmstepd: error: *** JOB 3524168 ON c140 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524186 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524186/3524186_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524186/3524186_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:32:02,499) - Starting with JobEnvironment(job_id=3524186, hostname=c126, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:32:02,501) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524186/3524186_submitted.pkl
Trial-Index: 37
slurmstepd: error: *** JOB 3524186 ON c126 CANCELLED AT 2026-05-10T12:32:15 DUE TO TIME LIMIT ***
⚠ Job 3524172 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524172/3524172_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524172/3524172_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,582) - Starting with JobEnvironment(job_id=3524172, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,584) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524172/3524172_submitted.pkl
Trial-Index: 29
slurmstepd: error: *** JOB 3524172 ON c136 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524171 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524171/3524171_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524171/3524171_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,610) - Starting with JobEnvironment(job_id=3524171, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,612) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524171/3524171_submitted.pkl
Trial-Index: 28
slurmstepd: error: *** JOB 3524171 ON c139 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524169 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524169/3524169_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524169/3524169_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,548) - Starting with JobEnvironment(job_id=3524169, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,550) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524169/3524169_submitted.pkl
Trial-Index: 23
slurmstepd: error: *** JOB 3524169 ON c140 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524183 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524183/3524183_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524183/3524183_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:32:01,934) - Starting with JobEnvironment(job_id=3524183, hostname=c151, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:32:01,937) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524183/3524183_submitted.pkl
Trial-Index: 25
slurmstepd: error: *** JOB 3524183 ON c151 CANCELLED AT 2026-05-10T12:32:15 DUE TO TIME LIMIT ***
⚠ Job 3524174 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524174/3524174_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524174/3524174_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,345) - Starting with JobEnvironment(job_id=3524174, hostname=c131, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,347) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524174/3524174_submitted.pkl
Trial-Index: 32
slurmstepd: error: *** JOB 3524174 ON c131 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524181 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524181/3524181_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524181/3524181_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:32,492) - Starting with JobEnvironment(job_id=3524181, hostname=c17, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:32,493) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524181/3524181_submitted.pkl
Trial-Index: 21
slurmstepd: error: *** JOB 3524181 ON c17 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524182 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524182/3524182_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524182/3524182_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:31,801) - Starting with JobEnvironment(job_id=3524182, hostname=c11, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:31,803) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524182/3524182_submitted.pkl
Trial-Index: 30
slurmstepd: error: *** JOB 3524182 ON c11 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524175 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524175/3524175_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524175/3524175_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,932) - Starting with JobEnvironment(job_id=3524175, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,934) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524175/3524175_submitted.pkl
Trial-Index: 33
slurmstepd: error: *** JOB 3524175 ON c108 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524179 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524179/3524179_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524179/3524179_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:31,629) - Starting with JobEnvironment(job_id=3524179, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:31,632) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524179/3524179_submitted.pkl
Trial-Index: 34
slurmstepd: error: *** JOB 3524179 ON c27 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524187 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524187/3524187_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524187/3524187_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:32:02,499) - Starting with JobEnvironment(job_id=3524187, hostname=c126, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:32:02,501) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524187/3524187_submitted.pkl
Trial-Index: 38
slurmstepd: error: *** JOB 3524187 ON c126 CANCELLED AT 2026-05-10T12:32:15 DUE TO TIME LIMIT ***
⚠ Job 3528402 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528402/3528402_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528402/3528402_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,664) - Starting with JobEnvironment(job_id=3528402, hostname=c151, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,666) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528402/3528402_submitted.pkl
Trial-Index: 43
slurmstepd: error: *** JOB 3528402 ON c151 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528419 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528419/3528419_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528419/3528419_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,702) - Starting with JobEnvironment(job_id=3528419, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,704) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528419/3528419_submitted.pkl
Trial-Index: 57
slurmstepd: error: *** JOB 3528419 ON c99 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528405 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528405/3528405_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528405/3528405_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,658) - Starting with JobEnvironment(job_id=3528405, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,660) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528405/3528405_submitted.pkl
Trial-Index: 41
slurmstepd: error: *** JOB 3528405 ON c142 CANCELLED AT 2026-05-11T00:36:32 DUE TO TIME LIMIT ***
⚠ Job 3528418 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528418/3528418_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528418/3528418_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,741) - Starting with JobEnvironment(job_id=3528418, hostname=c100, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,743) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528418/3528418_submitted.pkl
Trial-Index: 56
slurmstepd: error: *** JOB 3528418 ON c100 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528421 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528421/3528421_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528421/3528421_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:58,345) - Starting with JobEnvironment(job_id=3528421, hostname=c96, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:58,347) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528421/3528421_submitted.pkl
Trial-Index: 59
slurmstepd: error: *** JOB 3528421 ON c96 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528408 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528408/3528408_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528408/3528408_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,723) - Starting with JobEnvironment(job_id=3528408, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,725) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528408/3528408_submitted.pkl
Trial-Index: 51
slurmstepd: error: *** JOB 3528408 ON c139 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528407 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528407/3528407_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528407/3528407_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,723) - Starting with JobEnvironment(job_id=3528407, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,725) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528407/3528407_submitted.pkl
Trial-Index: 46
slurmstepd: error: *** JOB 3528407 ON c140 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528404 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528404/3528404_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528404/3528404_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,658) - Starting with JobEnvironment(job_id=3528404, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,660) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528404/3528404_submitted.pkl
Trial-Index: 53
slurmstepd: error: *** JOB 3528404 ON c142 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528410 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528410/3528410_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528410/3528410_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,666) - Starting with JobEnvironment(job_id=3528410, hostname=c138, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,668) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528410/3528410_submitted.pkl
Trial-Index: 44
slurmstepd: error: *** JOB 3528410 ON c138 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528415 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528415/3528415_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528415/3528415_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,703) - Starting with JobEnvironment(job_id=3528415, hostname=c101, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,705) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528415/3528415_submitted.pkl
Trial-Index: 50
slurmstepd: error: *** JOB 3528415 ON c101 CANCELLED AT 2026-05-11T00:37:02 DUE TO TIME LIMIT ***
⚠ Job 3528406 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528406/3528406_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528406/3528406_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,723) - Starting with JobEnvironment(job_id=3528406, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,725) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528406/3528406_submitted.pkl
Trial-Index: 42
slurmstepd: error: *** JOB 3528406 ON c140 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528416 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528416/3528416_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528416/3528416_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,703) - Starting with JobEnvironment(job_id=3528416, hostname=c101, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,705) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528416/3528416_submitted.pkl
Trial-Index: 52
slurmstepd: error: *** JOB 3528416 ON c101 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528413 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528413/3528413_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528413/3528413_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,631) - Starting with JobEnvironment(job_id=3528413, hostname=c102, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,633) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528413/3528413_submitted.pkl
Trial-Index: 54
slurmstepd: error: *** JOB 3528413 ON c102 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528403 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528403/3528403_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528403/3528403_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,659) - Starting with JobEnvironment(job_id=3528403, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,661) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528403/3528403_submitted.pkl
Trial-Index: 48
slurmstepd: error: *** JOB 3528403 ON c144 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528420 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528420/3528420_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528420/3528420_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,702) - Starting with JobEnvironment(job_id=3528420, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,704) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528420/3528420_submitted.pkl
Trial-Index: 58
slurmstepd: error: *** JOB 3528420 ON c99 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528409 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528409/3528409_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528409/3528409_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,723) - Starting with JobEnvironment(job_id=3528409, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,725) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528409/3528409_submitted.pkl
Trial-Index: 47
slurmstepd: error: *** JOB 3528409 ON c139 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528417 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528417/3528417_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528417/3528417_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,741) - Starting with JobEnvironment(job_id=3528417, hostname=c100, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,743) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528417/3528417_submitted.pkl
Trial-Index: 49
slurmstepd: error: *** JOB 3528417 ON c100 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528412 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528412/3528412_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528412/3528412_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:58,175) - Starting with JobEnvironment(job_id=3528412, hostname=c105, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:58,177) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528412/3528412_submitted.pkl
Trial-Index: 45
slurmstepd: error: *** JOB 3528412 ON c105 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3531808 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531808/3531808_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531808/3531808_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:06:33,740) - Starting with JobEnvironment(job_id=3531808, hostname=c12, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:06:33,742) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531808/3531808_submitted.pkl
Trial-Index: 72
slurmstepd: error: *** JOB 3531808 ON c12 CANCELLED AT 2026-05-11T14:06:48 DUE TO TIME LIMIT ***
⚠ Job 3531807 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531807/3531807_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531807/3531807_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:03:32,711) - Starting with JobEnvironment(job_id=3531807, hostname=c111, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:03:32,713) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531807/3531807_submitted.pkl
Trial-Index: 61
slurmstepd: error: *** JOB 3531807 ON c111 CANCELLED AT 2026-05-11T14:03:48 DUE TO TIME LIMIT ***
⚠ Job 3531806 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531806/3531806_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531806/3531806_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:03:31,967) - Starting with JobEnvironment(job_id=3531806, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:03:31,969) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531806/3531806_submitted.pkl
Trial-Index: 68
slurmstepd: error: *** JOB 3531806 ON c136 CANCELLED AT 2026-05-11T14:03:48 DUE TO TIME LIMIT ***
⚠ Job 3531809 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531809/3531809_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531809/3531809_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:11:04,535) - Starting with JobEnvironment(job_id=3531809, hostname=c51, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:11:04,537) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531809/3531809_submitted.pkl
Trial-Index: 62
slurmstepd: error: *** JOB 3531809 ON c51 CANCELLED AT 2026-05-11T14:11:18 DUE TO TIME LIMIT ***
⚠ Job 3531812 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531812/3531812_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531812/3531812_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:35:11,961) - Starting with JobEnvironment(job_id=3531812, hostname=c148, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:35:11,963) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531812/3531812_submitted.pkl
Trial-Index: 66
slurmstepd: error: *** JOB 3531812 ON c148 CANCELLED AT 2026-05-11T14:35:19 DUE TO TIME LIMIT ***
⚠ Job 3531811 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531811/3531811_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531811/3531811_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:34:11,583) - Starting with JobEnvironment(job_id=3531811, hostname=c143, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:34:11,585) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531811/3531811_submitted.pkl
Trial-Index: 65
slurmstepd: error: *** JOB 3531811 ON c143 CANCELLED AT 2026-05-11T14:34:19 DUE TO TIME LIMIT ***
⚠ Job 3531810 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531810/3531810_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531810/3531810_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:28:40,064) - Starting with JobEnvironment(job_id=3531810, hostname=c41, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:28:40,066) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531810/3531810_submitted.pkl
Trial-Index: 67
slurmstepd: error: *** JOB 3531810 ON c41 CANCELLED AT 2026-05-11T14:28:48 DUE TO TIME LIMIT ***
⚠ Job 3531813 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531813/3531813_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531813/3531813_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:37:42,639) - Starting with JobEnvironment(job_id=3531813, hostname=c101, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:37:42,641) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531813/3531813_submitted.pkl
Trial-Index: 75
slurmstepd: error: *** JOB 3531813 ON c101 CANCELLED AT 2026-05-11T14:37:49 DUE TO TIME LIMIT ***
⚠ Job 3531814 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531814/3531814_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531814/3531814_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:42:15,504) - Starting with JobEnvironment(job_id=3531814, hostname=c57, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:42:15,506) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531814/3531814_submitted.pkl
Trial-Index: 63
slurmstepd: error: *** JOB 3531814 ON c57 CANCELLED AT 2026-05-11T14:42:19 DUE TO TIME LIMIT ***
⚠ Job 3531815 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531815/3531815_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531815/3531815_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:04:54,062) - Starting with JobEnvironment(job_id=3531815, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:04:54,064) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531815/3531815_submitted.pkl
Trial-Index: 73
slurmstepd: error: *** JOB 3531815 ON c136 CANCELLED AT 2026-05-11T15:05:19 DUE TO TIME LIMIT ***
⚠ Job 3531820 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531820/3531820_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531820/3531820_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:16:27,570) - Starting with JobEnvironment(job_id=3531820, hostname=c112, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:16:27,572) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531820/3531820_submitted.pkl
Trial-Index: 69
slurmstepd: error: *** JOB 3531820 ON c112 CANCELLED AT 2026-05-11T15:16:49 DUE TO TIME LIMIT ***
⚠ Job 3531818 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531818/3531818_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531818/3531818_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:10:55,833) - Starting with JobEnvironment(job_id=3531818, hostname=c53, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:10:55,835) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531818/3531818_submitted.pkl
Trial-Index: 70
slurmstepd: error: *** JOB 3531818 ON c53 CANCELLED AT 2026-05-11T15:11:19 DUE TO TIME LIMIT ***
⚠ Job 3531817 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531817/3531817_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531817/3531817_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:10:56,638) - Starting with JobEnvironment(job_id=3531817, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:10:56,640) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531817/3531817_submitted.pkl
Trial-Index: 64
slurmstepd: error: *** JOB 3531817 ON c144 CANCELLED AT 2026-05-11T15:11:19 DUE TO TIME LIMIT ***
⚠ Job 3531816 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531816/3531816_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531816/3531816_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:08:55,097) - Starting with JobEnvironment(job_id=3531816, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:08:55,099) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531816/3531816_submitted.pkl
Trial-Index: 74
slurmstepd: error: *** JOB 3531816 ON c99 CANCELLED AT 2026-05-11T15:09:19 DUE TO TIME LIMIT ***
⚠ Job 3531819 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531819/3531819_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531819/3531819_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:11:26,253) - Starting with JobEnvironment(job_id=3531819, hostname=c93, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:11:26,255) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531819/3531819_submitted.pkl
Trial-Index: 71
slurmstepd: error: *** JOB 3531819 ON c93 CANCELLED AT 2026-05-11T15:11:49 DUE TO TIME LIMIT ***
⚠ Job 3531824 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531824/3531824_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531824/3531824_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:19:27,568) - Starting with JobEnvironment(job_id=3531824, hostname=c96, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:19:27,570) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531824/3531824_submitted.pkl
Trial-Index: 77
slurmstepd: error: *** JOB 3531824 ON c96 CANCELLED AT 2026-05-11T15:19:49 DUE TO TIME LIMIT ***
⚠ Job 3531822 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531822/3531822_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531822/3531822_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:18:27,516) - Starting with JobEnvironment(job_id=3531822, hostname=c37, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:18:27,518) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531822/3531822_submitted.pkl
Trial-Index: 78
slurmstepd: error: *** JOB 3531822 ON c37 CANCELLED AT 2026-05-11T15:18:49 DUE TO TIME LIMIT ***
⚠ Job 3531823 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531823/3531823_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531823/3531823_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:19:28,140) - Starting with JobEnvironment(job_id=3531823, hostname=c120, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:19:28,142) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531823/3531823_submitted.pkl
Trial-Index: 79
slurmstepd: error: *** JOB 3531823 ON c120 CANCELLED AT 2026-05-11T15:19:49 DUE TO TIME LIMIT ***
⚠ Job 3531821 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531821/3531821_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531821/3531821_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:18:27,922) - Starting with JobEnvironment(job_id=3531821, hostname=c104, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:18:27,924) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531821/3531821_submitted.pkl
Trial-Index: 76
slurmstepd: error: *** JOB 3531821 ON c104 CANCELLED AT 2026-05-11T15:18:49 DUE TO TIME LIMIT ***
To cancel, press CTRLc, then run 'scancel 3514197'
⠋Importing logging...⠋Importing warnings...⠋Importing argparse...⠋Importing datetime...⠋Importing dataclass...⠋Importing socket...⠋Importing stat...⠋Importing pwd...⠋Importing base64...⠋Importing json...⠋Importing yaml...⠋Importing toml...⠋Importing csv...⠋Importing ast...⠋Importing rich.table...⠋Importing rich print...⠋Importing rich.pretty...⠋Importing pformat...⠋Importing rich.prompt...⠋Importing types.FunctionType...⠋Importing typing...⠋Importing ThreadPoolExecutor...⠋Importing submitit.LocalExecutor...⠋Importing submitit.Job...⠋Importing importlib.util...⠋Importing platform...⠋Importing inspect frame info...⠋Importing pathlib.Path...⠋Importing uuid...⠋Importing cowsay...⠋Importing shutil...⠋Importing itertools.combinations...⠋Importing os.listdir...⠋Importing os.path...⠋Importing PIL.Image...⠋Importing sixel...⠋Importing subprocess...⠋Importing tqdm...⠙Importing beartype...
[WARNING 05-09 12:27:23] ax.storage.sqa_store.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
⠋Importing rendering stuff...⠋Importing statistics...⠋Trying to import pyfiglet...⠋Importing helpers...⠋Importing pareto...⠋Parsing arguments...⠋Importing torch...⠋Importing numpy...⠋Importing ax...⠋Importing ax.core.generator_run...⠋Importing Cont_X_trans and Y_trans from ax.adapter.registry...⠋Importing ax.core.arm...⠋Importing ax.core.objective...⠋Importing ax.core.Metric...⠋Importing ax.exceptions.core...⠋Importing ax.exceptions.generation_strategy...⠋Importing CORE_DECODER_REGISTRY...⠋Trying ax.generation_strategy.generation_node...⠋Importing GenerationStep, GenerationStrategy from generation_strategy...⠋Importing GenerationNode from generation_node...⠋Importing ExternalGenerationNode...⠋Importing MinTrials...⠋Importing GeneratorSpec...⠋Importing Generators from ax.generation_strategy.registry...⠋Importing load_experiment...⠋Importing save_experiment...⠋Importing save_experiment_to_db...⠋Importing TrialStatus...⠋Importing Data...⠋Importing Experiment...⠋Importing parameter types...⠋Importing TParameterization...⠋Importing pandas...⠋Importing AxClient and ObjectiveProperties...⠋Importing RandomForestRegressor...⠋Importing botorch...⠋Importing submitit...⠋Importing ax logger...⠋Importing SQL-Storage-Stuff...
Run-UUID: 2979ba8f-99c6-4f82-b9fd-4fa51152ec39
____ _ ____ _ ___
/ __ \ (_)/ __ \ | | |__ \
| | | |_ __ ___ _ __ _| | | |_ __ | |_ ) |
| | | | '_ ` _ \| '_ \| | | | | '_ \| __| / /
| |__| | | | | | | | | | | |__| | |_) | |_ / /_
\____/|_| |_| |_|_| |_|_|\____/| .__/ \__|____|
| |
|_|
⠋Writing worker creation log...
omniopt --partition=alpha --experiment_name=tinyLogicLLM5 --mem_gb=32 --time=7200 --worker_timeout=720 --max_eval=1000 --num_parallel_jobs=20 --gpus=1 --num_random_steps=20 --follow --live_share --send_anonymized_usage_stats --result_names 'VAL_LOSS=min PARAMS=min' --run_program='cHl0aG9uMyAvZGF0YS9ob3JzZS93cy9zMzgxMTE0MS1ncm9rMy90cmFpbi5weSAtLWJhdGNoLXNpemU9JShiYXRjaF9zaXplKSAtLXRvcG8gLS10b3BvLWV2ZXJ5IDEgLS1rZWxwLWV2ZXJ5PTEgLS1uLWxheWVycz0lKG5fbGF5ZXJzKSAtLWQtbW9kZWw9JShkX21vZGVsKSAtLW4taGVhZHM9JShuX2hlYWRzKSAtLXRva2VuaXplcl9pbml0aWFsX25yPTEwMDAgLS1scj0lKGxyKSAtLWJhdGNoZXMtcGVyLWVwb2NoPSUoYmF0Y2hlc19wZXJfZXBvY2gpIC0tdmFsLWJhdGNoZXM9JSh2YWxfYmF0Y2hlcykgLS1lcG9jaHM9JShlcG9jaHMpIC0tdGFzaz10dXJuc3RpbGUgLS1tYXgtc2VxLWxlbiAlKG1heF9zZXF1ZW5jZV9sZW5ndGgpIC0tbWF4LXR1cm5zdGlsZXMgJShtYXhfdHVybnN0aWxlcykgLS1zY2hlZHVsZXI9Y29zaW5lIC0tZmZuPSUoZmZuKQo=' --run_program_once='cHl0aG9uMyAvZGF0YS9ob3JzZS93cy9zMzgxMTE0MS1ncm9rMi90cmFpbi5weSAtLWV4aXQtYWZ0ZXItaW5zdGFsbA==' --cpus_per_task=1 --nodes_per_job=1 --revert_to_random_when_seemingly_exhausted --model=BOTORCH_MODULAR --n_estimators_randomforest=100 --run_mode=local --show_sixel_general --show_sixel_trial_index_result --show_sixel_scatter --occ_type=euclid --main_process_gb=8 --nr_evals_per_arm=1 --max_nr_of_zero_results=50 --slurm_signal_delay_s=0 --max_failed_jobs=0 --max_attempts_for_generation=20 --num_restarts=20 --raw_samples=1024 --max_abandoned_retrial=20 --max_num_of_parallel_sruns=16 --number_of_generators=1 --generate_all_jobs_at_once --parameter batch_size range 16 512 int false --parameter ffn range 2 32 int false --parameter max_turnstiles fixed 128 --parameter epochs range 10 400 int false --parameter n_layers range 1 2 int false --parameter d_model range 2 3 int false --parameter max_sequence_length range 128 256 int false --parameter val_batches fixed 256 --parameter lr range 0.0000001 0.1 float false --parameter n_heads range 1 32 int false --parameter batches_per_epoch fixed 30
⠋Disabling logging...⠋Setting run folder...⠋Creating folder /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0...⠋Writing revert_to_random_when_seemingly_exhausted file ...⠋Writing username state file...⠋Writing result names file...⠋Writing result min/max file...
Exiting after installation
Executing command: python3 /data/horse/ws/s3811141-grok2/train.py --exit-after-install
Setup script completed successfully ✅
⠋Saving state files...Run-folder: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0⠋Writing live_share file if it is present...⠋Writing job_start_time file...⠹Writing git information⠋Checking max_eval...⠋Calculating number of steps...⠋Adding excluded nodes...⠋Initializing ax_client...⠋Setting orchestrator...See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=tinyLogicLLM5&run_nr=0 for live-results.You have 1 CPUs available for the main process.Using CUDA device NVIDIA H100.Generation strategy: SOBOL for 20 steps and then BOTORCH_MODULAR for 980 steps.
Run-Program: python3 /data/horse/ws/s3811141-grok3/train.py --batch-size=%(batch_size) --topo --topo-every 1 --kelp-every=1 --n-layers=%(n_layers) --d-model=%(d_model) --n-heads=%(n_heads) --tokenizer_initial_nr=1000 --lr=%(lr) --batches-per-epoch=%(batches_per_epoch) --val-batches=%(val_batches) --epochs=%(epochs) --task=turnstile --max-seq-len %(max_sequence_length) --max-turnstiles %(max_turnstiles) --scheduler=cosine --ffn=%(ffn)
Experiment parameters
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃Name ┃Type ┃Lower bound┃Upper bound┃Values┃Type ┃Log Scale?┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│batch_size │range│16 │512 ││int │No │
│ffn │range│2 │32 ││int │No │
│max_turnstiles │fixed│││128 │││
│epochs │range│10 │400 ││int │No │
│n_layers │range│1 │2 ││int │No │
│d_model │range│2 │3 ││int │No │
│max_sequence_length│range│128 │256 ││int │No │
│val_batches │fixed│││256 │││
│lr │range│1e-07 │0.1 ││float│No │
│n_heads │range│1 │32 ││int │No │
│batches_per_epoch │fixed│││30 │││
└─────────────────────┴───────┴─────────────┴─────────────┴────────┴───────┴────────────┘
Result-Names
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃Result-Name┃Min or max?┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│VAL_LOSS │ min│
│PARAMS │ min│
└─────────────┴─────────────┘
⠋Write files and show overview
SOBOL, 1 done, running 19 = ∑19/20, waiting for 19 jobs : 0%|░░░░░░░░░░| 1/1000 [1:36:50<1611:26:18, 5806.99s/it]
Exit-Code 3, this means: Invalid exit code detected
SOBOL, 3 done, running 17 = ∑17/20, waiting for 17 jobs : 0%|░░░░░░░░░░| 3/1000 [11:25:10<3681:27:56, 13293.16s/it]
⚠ Job 3514198 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514198/3514198_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514198/3514198_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,752) - Starting with JobEnvironment(job_id=3514198, hostname=c154, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,758) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514198/3514198_submitted.pkl
Trial-Index: 1
slurmstepd: error: *** JOB 3514198 ON c154 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514199 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514199/3514199_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514199/3514199_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:24,575) - Starting with JobEnvironment(job_id=3514199, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:24,630) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514199/3514199_submitted.pkl
Trial-Index: 4
slurmstepd: error: *** JOB 3514199 ON c140 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514212 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514212/3514212_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514212/3514212_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,577) - Starting with JobEnvironment(job_id=3514212, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,585) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514212/3514212_submitted.pkl
Trial-Index: 12
slurmstepd: error: *** JOB 3514212 ON c27 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514215 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514215/3514215_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514215/3514215_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,615) - Starting with JobEnvironment(job_id=3514215, hostname=c3, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,622) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514215/3514215_submitted.pkl
Trial-Index: 17
slurmstepd: error: *** JOB 3514215 ON c3 CANCELLED AT 2026-05-10T00:28:55 DUE TO TIME LIMIT ***
⚠ Job 3514201 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514201/3514201_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514201/3514201_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,422) - Starting with JobEnvironment(job_id=3514201, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,425) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514201/3514201_submitted.pkl
Trial-Index: 0
slurmstepd: error: *** JOB 3514201 ON c139 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514217 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514217/3514217_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514217/3514217_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:29:24,899) - Starting with JobEnvironment(job_id=3514217, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:29:24,905) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514217/3514217_submitted.pkl
Trial-Index: 19
slurmstepd: error: *** JOB 3514217 ON c136 CANCELLED AT 2026-05-10T00:29:24 DUE TO TIME LIMIT ***
⚠ Job 3514209 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514209/3514209_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514209/3514209_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,706) - Starting with JobEnvironment(job_id=3514209, hostname=c114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,711) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514209/3514209_submitted.pkl
Trial-Index: 14
slurmstepd: error: *** JOB 3514209 ON c114 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514213 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514213/3514213_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514213/3514213_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,675) - Starting with JobEnvironment(job_id=3514213, hostname=c6, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,681) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514213/3514213_submitted.pkl
Trial-Index: 10
slurmstepd: error: *** JOB 3514213 ON c6 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514216 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514216/3514216_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514216/3514216_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:29:24,279) - Starting with JobEnvironment(job_id=3514216, hostname=c151, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:29:24,290) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514216/3514216_submitted.pkl
Trial-Index: 18
slurmstepd: error: *** JOB 3514216 ON c151 CANCELLED AT 2026-05-10T00:29:24 DUE TO TIME LIMIT ***
⚠ Job 3514214 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514214/3514214_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514214/3514214_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,615) - Starting with JobEnvironment(job_id=3514214, hostname=c3, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,622) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514214/3514214_submitted.pkl
Trial-Index: 16
slurmstepd: error: *** JOB 3514214 ON c3 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514202 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514202/3514202_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514202/3514202_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,422) - Starting with JobEnvironment(job_id=3514202, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,425) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514202/3514202_submitted.pkl
Trial-Index: 9
slurmstepd: error: *** JOB 3514202 ON c139 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514210 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514210/3514210_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514210/3514210_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,706) - Starting with JobEnvironment(job_id=3514210, hostname=c114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,712) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514210/3514210_submitted.pkl
Trial-Index: 13
slurmstepd: error: *** JOB 3514210 ON c114 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514200 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514200/3514200_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514200/3514200_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,530) - Starting with JobEnvironment(job_id=3514200, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,535) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514200/3514200_submitted.pkl
Trial-Index: 2
slurmstepd: error: *** JOB 3514200 ON c140 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514208 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514208/3514208_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514208/3514208_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:53,440) - Starting with JobEnvironment(job_id=3514208, hostname=c130, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:53,443) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514208/3514208_submitted.pkl
Trial-Index: 5
slurmstepd: error: *** JOB 3514208 ON c130 CANCELLED AT 2026-05-10T00:28:54 DUE TO TIME LIMIT ***
⚠ Job 3514207 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514207/3514207_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514207/3514207_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,813) - Starting with JobEnvironment(job_id=3514207, hostname=c91, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,819) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514207/3514207_submitted.pkl
Trial-Index: 11
slurmstepd: error: *** JOB 3514207 ON c91 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514204 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514204/3514204_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514204/3514204_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,406) - Starting with JobEnvironment(job_id=3514204, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,408) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514204/3514204_submitted.pkl
Trial-Index: 3
slurmstepd: error: *** JOB 3514204 ON c99 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
⚠ Job 3514205 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514205/3514205_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514205/3514205_0_log.out
----------------------------------------
submitit INFO (2026-05-09 12:28:23,406) - Starting with JobEnvironment(job_id=3514205, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,408) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514205/3514205_submitted.pkl
Trial-Index: 6
slurmstepd: error: *** JOB 3514205 ON c99 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***
SOBOL, failed: 17, 5 done, timeout 18 = ∑18/20, waiting for 18 jobs : 0%|░░░░░░░░░░| 5/1000 [24:06:28<3521:12:36, 12740.06s/it]
⚠ Job 3524181 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524181/3524181_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524181/3524181_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:32,492) - Starting with JobEnvironment(job_id=3524181, hostname=c17, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:32,493) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524181/3524181_submitted.pkl
Trial-Index: 21
slurmstepd: error: *** JOB 3524181 ON c17 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524183 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524183/3524183_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524183/3524183_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:32:01,934) - Starting with JobEnvironment(job_id=3524183, hostname=c151, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:32:01,937) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524183/3524183_submitted.pkl
Trial-Index: 25
slurmstepd: error: *** JOB 3524183 ON c151 CANCELLED AT 2026-05-10T12:32:15 DUE TO TIME LIMIT ***
⚠ Job 3524182 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524182/3524182_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524182/3524182_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:31,801) - Starting with JobEnvironment(job_id=3524182, hostname=c11, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:31,803) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524182/3524182_submitted.pkl
Trial-Index: 30
slurmstepd: error: *** JOB 3524182 ON c11 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524174 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524174/3524174_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524174/3524174_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,345) - Starting with JobEnvironment(job_id=3524174, hostname=c131, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,347) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524174/3524174_submitted.pkl
Trial-Index: 32
slurmstepd: error: *** JOB 3524174 ON c131 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524176 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524176/3524176_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524176/3524176_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,932) - Starting with JobEnvironment(job_id=3524176, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,934) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524176/3524176_submitted.pkl
Trial-Index: 35
slurmstepd: error: *** JOB 3524176 ON c108 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524169 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524169/3524169_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524169/3524169_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,548) - Starting with JobEnvironment(job_id=3524169, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,550) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524169/3524169_submitted.pkl
Trial-Index: 23
slurmstepd: error: *** JOB 3524169 ON c140 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524172 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524172/3524172_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524172/3524172_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,582) - Starting with JobEnvironment(job_id=3524172, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,584) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524172/3524172_submitted.pkl
Trial-Index: 29
slurmstepd: error: *** JOB 3524172 ON c136 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524170 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524170/3524170_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524170/3524170_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,610) - Starting with JobEnvironment(job_id=3524170, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,612) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524170/3524170_submitted.pkl
Trial-Index: 22
slurmstepd: error: *** JOB 3524170 ON c139 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524168 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524168/3524168_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524168/3524168_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,548) - Starting with JobEnvironment(job_id=3524168, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,550) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524168/3524168_submitted.pkl
Trial-Index: 20
slurmstepd: error: *** JOB 3524168 ON c140 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524175 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524175/3524175_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524175/3524175_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,932) - Starting with JobEnvironment(job_id=3524175, hostname=c108, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,934) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524175/3524175_submitted.pkl
Trial-Index: 33
slurmstepd: error: *** JOB 3524175 ON c108 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524179 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524179/3524179_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524179/3524179_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:31,629) - Starting with JobEnvironment(job_id=3524179, hostname=c27, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:31,632) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524179/3524179_submitted.pkl
Trial-Index: 34
slurmstepd: error: *** JOB 3524179 ON c27 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524171 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524171/3524171_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524171/3524171_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,610) - Starting with JobEnvironment(job_id=3524171, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,612) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524171/3524171_submitted.pkl
Trial-Index: 28
slurmstepd: error: *** JOB 3524171 ON c139 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524177 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524177/3524177_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524177/3524177_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:01,879) - Starting with JobEnvironment(job_id=3524177, hostname=c101, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:01,881) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524177/3524177_submitted.pkl
Trial-Index: 31
slurmstepd: error: *** JOB 3524177 ON c101 CANCELLED AT 2026-05-10T12:31:15 DUE TO TIME LIMIT ***
⚠ Job 3524180 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524180/3524180_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524180/3524180_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:32,579) - Starting with JobEnvironment(job_id=3524180, hostname=c25, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:32,580) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524180/3524180_submitted.pkl
Trial-Index: 27
slurmstepd: error: *** JOB 3524180 ON c25 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524178 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524178/3524178_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524178/3524178_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:31:32,253) - Starting with JobEnvironment(job_id=3524178, hostname=c76, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:31:32,257) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524178/3524178_submitted.pkl
Trial-Index: 26
slurmstepd: error: *** JOB 3524178 ON c76 CANCELLED AT 2026-05-10T12:31:45 DUE TO TIME LIMIT ***
⚠ Job 3524186 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524186/3524186_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524186/3524186_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:32:02,499) - Starting with JobEnvironment(job_id=3524186, hostname=c126, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:32:02,501) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524186/3524186_submitted.pkl
Trial-Index: 37
slurmstepd: error: *** JOB 3524186 ON c126 CANCELLED AT 2026-05-10T12:32:15 DUE TO TIME LIMIT ***
⚠ Job 3524184 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524184/3524184_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524184/3524184_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:32:02,810) - Starting with JobEnvironment(job_id=3524184, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:32:02,812) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524184/3524184_submitted.pkl
Trial-Index: 36
slurmstepd: error: *** JOB 3524184 ON c144 CANCELLED AT 2026-05-10T12:32:15 DUE TO TIME LIMIT ***
⚠ Job 3524187 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524187/3524187_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524187/3524187_0_log.out
----------------------------------------
submitit INFO (2026-05-10 00:32:02,499) - Starting with JobEnvironment(job_id=3524187, hostname=c126, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 00:32:02,501) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3524187/3524187_submitted.pkl
Trial-Index: 38
slurmstepd: error: *** JOB 3524187 ON c126 CANCELLED AT 2026-05-10T12:32:15 DUE TO TIME LIMIT ***
SOBOL, failed: 35, 7 done, timeout 18 = ∑18/20, waiting for 18 jobs : 1%|░░░░░░░░░░| 7/1000 [36:18:15<5479:01:52, 19863.56s/it]
⚠ Job 3528416 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528416/3528416_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528416/3528416_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,703) - Starting with JobEnvironment(job_id=3528416, hostname=c101, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,705) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528416/3528416_submitted.pkl
Trial-Index: 52
slurmstepd: error: *** JOB 3528416 ON c101 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528405 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528405/3528405_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528405/3528405_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,658) - Starting with JobEnvironment(job_id=3528405, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,660) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528405/3528405_submitted.pkl
Trial-Index: 41
slurmstepd: error: *** JOB 3528405 ON c142 CANCELLED AT 2026-05-11T00:36:32 DUE TO TIME LIMIT ***
⚠ Job 3528406 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528406/3528406_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528406/3528406_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,723) - Starting with JobEnvironment(job_id=3528406, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,725) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528406/3528406_submitted.pkl
Trial-Index: 42
slurmstepd: error: *** JOB 3528406 ON c140 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528415 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528415/3528415_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528415/3528415_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,703) - Starting with JobEnvironment(job_id=3528415, hostname=c101, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,705) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528415/3528415_submitted.pkl
Trial-Index: 50
slurmstepd: error: *** JOB 3528415 ON c101 CANCELLED AT 2026-05-11T00:37:02 DUE TO TIME LIMIT ***
⚠ Job 3528418 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528418/3528418_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528418/3528418_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,741) - Starting with JobEnvironment(job_id=3528418, hostname=c100, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,743) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528418/3528418_submitted.pkl
Trial-Index: 56
slurmstepd: error: *** JOB 3528418 ON c100 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528413 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528413/3528413_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528413/3528413_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,631) - Starting with JobEnvironment(job_id=3528413, hostname=c102, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,633) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528413/3528413_submitted.pkl
Trial-Index: 54
slurmstepd: error: *** JOB 3528413 ON c102 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528407 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528407/3528407_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528407/3528407_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,723) - Starting with JobEnvironment(job_id=3528407, hostname=c140, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,725) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528407/3528407_submitted.pkl
Trial-Index: 46
slurmstepd: error: *** JOB 3528407 ON c140 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528421 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528421/3528421_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528421/3528421_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:58,345) - Starting with JobEnvironment(job_id=3528421, hostname=c96, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:58,347) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528421/3528421_submitted.pkl
Trial-Index: 59
slurmstepd: error: *** JOB 3528421 ON c96 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528403 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528403/3528403_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528403/3528403_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,659) - Starting with JobEnvironment(job_id=3528403, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,661) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528403/3528403_submitted.pkl
Trial-Index: 48
slurmstepd: error: *** JOB 3528403 ON c144 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528408 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528408/3528408_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528408/3528408_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,723) - Starting with JobEnvironment(job_id=3528408, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,725) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528408/3528408_submitted.pkl
Trial-Index: 51
slurmstepd: error: *** JOB 3528408 ON c139 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528410 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528410/3528410_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528410/3528410_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,666) - Starting with JobEnvironment(job_id=3528410, hostname=c138, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,668) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528410/3528410_submitted.pkl
Trial-Index: 44
slurmstepd: error: *** JOB 3528410 ON c138 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528419 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528419/3528419_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528419/3528419_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,702) - Starting with JobEnvironment(job_id=3528419, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,704) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528419/3528419_submitted.pkl
Trial-Index: 57
slurmstepd: error: *** JOB 3528419 ON c99 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528420 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528420/3528420_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528420/3528420_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,702) - Starting with JobEnvironment(job_id=3528420, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,704) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528420/3528420_submitted.pkl
Trial-Index: 58
slurmstepd: error: *** JOB 3528420 ON c99 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528404 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528404/3528404_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528404/3528404_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,658) - Starting with JobEnvironment(job_id=3528404, hostname=c142, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,660) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528404/3528404_submitted.pkl
Trial-Index: 53
slurmstepd: error: *** JOB 3528404 ON c142 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528409 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528409/3528409_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528409/3528409_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,723) - Starting with JobEnvironment(job_id=3528409, hostname=c139, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,725) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528409/3528409_submitted.pkl
Trial-Index: 47
slurmstepd: error: *** JOB 3528409 ON c139 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528402 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528402/3528402_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528402/3528402_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:27,664) - Starting with JobEnvironment(job_id=3528402, hostname=c151, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:27,666) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528402/3528402_submitted.pkl
Trial-Index: 43
slurmstepd: error: *** JOB 3528402 ON c151 CANCELLED AT 2026-05-11T00:36:31 DUE TO TIME LIMIT ***
⚠ Job 3528417 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528417/3528417_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528417/3528417_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:57,741) - Starting with JobEnvironment(job_id=3528417, hostname=c100, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:57,743) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528417/3528417_submitted.pkl
Trial-Index: 49
slurmstepd: error: *** JOB 3528417 ON c100 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
⚠ Job 3528412 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528412/3528412_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528412/3528412_0_log.out
----------------------------------------
submitit INFO (2026-05-10 12:36:58,175) - Starting with JobEnvironment(job_id=3528412, hostname=c105, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-10 12:36:58,177) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3528412/3528412_submitted.pkl
Trial-Index: 45
slurmstepd: error: *** JOB 3528412 ON c105 CANCELLED AT 2026-05-11T00:37:01 DUE TO TIME LIMIT ***
SOBOL, failed: 53, 8 done, running 19 = ∑19/20, waiting for 19 jobs : 1%|░░░░░░░░░░| 8/1000 [45:10:28<7089:37:23, 25728.47s/it]
⚠ Job 3531806 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531806/3531806_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531806/3531806_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:03:31,967) - Starting with JobEnvironment(job_id=3531806, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:03:31,969) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531806/3531806_submitted.pkl
Trial-Index: 68
slurmstepd: error: *** JOB 3531806 ON c136 CANCELLED AT 2026-05-11T14:03:48 DUE TO TIME LIMIT ***
⚠ Job 3531807 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531807/3531807_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531807/3531807_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:03:32,711) - Starting with JobEnvironment(job_id=3531807, hostname=c111, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:03:32,713) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531807/3531807_submitted.pkl
Trial-Index: 61
slurmstepd: error: *** JOB 3531807 ON c111 CANCELLED AT 2026-05-11T14:03:48 DUE TO TIME LIMIT ***
⚠ Job 3531808 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531808/3531808_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531808/3531808_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:06:33,740) - Starting with JobEnvironment(job_id=3531808, hostname=c12, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:06:33,742) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531808/3531808_submitted.pkl
Trial-Index: 72
slurmstepd: error: *** JOB 3531808 ON c12 CANCELLED AT 2026-05-11T14:06:48 DUE TO TIME LIMIT ***
SOBOL, failed: 56, 8 done, running/timeout 15/1 = ∑16/20, waiting for 16 jobs : 1%|░░░░░░░░░░| 8/1000 [49:48:46<7089:37:23, 25728.47s/it]
⚠ Job 3531809 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531809/3531809_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531809/3531809_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:11:04,535) - Starting with JobEnvironment(job_id=3531809, hostname=c51, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:11:04,537) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531809/3531809_submitted.pkl
Trial-Index: 62
slurmstepd: error: *** JOB 3531809 ON c51 CANCELLED AT 2026-05-11T14:11:18 DUE TO TIME LIMIT ***
SOBOL, failed: 57, 8 done, timeout/running 3/12 = ∑15/20, waiting for 15 jobs : 1%|░░░░░░░░░░| 8/1000 [50:08:47<7089:37:23, 25728.47s/it]
⚠ Job 3531812 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531812/3531812_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531812/3531812_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:35:11,961) - Starting with JobEnvironment(job_id=3531812, hostname=c148, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:35:11,963) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531812/3531812_submitted.pkl
Trial-Index: 66
slurmstepd: error: *** JOB 3531812 ON c148 CANCELLED AT 2026-05-11T14:35:19 DUE TO TIME LIMIT ***
⚠ Job 3531811 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531811/3531811_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531811/3531811_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:34:11,583) - Starting with JobEnvironment(job_id=3531811, hostname=c143, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:34:11,585) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531811/3531811_submitted.pkl
Trial-Index: 65
slurmstepd: error: *** JOB 3531811 ON c143 CANCELLED AT 2026-05-11T14:34:19 DUE TO TIME LIMIT ***
⚠ Job 3531810 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531810/3531810_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531810/3531810_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:28:40,064) - Starting with JobEnvironment(job_id=3531810, hostname=c41, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:28:40,066) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531810/3531810_submitted.pkl
Trial-Index: 67
slurmstepd: error: *** JOB 3531810 ON c41 CANCELLED AT 2026-05-11T14:28:48 DUE TO TIME LIMIT ***
SOBOL, failed: 60, 8 done, timeout/running 2/10 = ∑12/20, waiting for 12 jobs : 1%|░░░░░░░░░░| 8/1000 [50:18:47<7089:37:23, 25728.47s/it]
⚠ Job 3531813 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531813/3531813_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531813/3531813_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:37:42,639) - Starting with JobEnvironment(job_id=3531813, hostname=c101, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:37:42,641) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531813/3531813_submitted.pkl
Trial-Index: 75
slurmstepd: error: *** JOB 3531813 ON c101 CANCELLED AT 2026-05-11T14:37:49 DUE TO TIME LIMIT ***
⚠ Job 3531814 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531814/3531814_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531814/3531814_0_log.out
----------------------------------------
submitit INFO (2026-05-11 02:42:15,504) - Starting with JobEnvironment(job_id=3531814, hostname=c57, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 02:42:15,506) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531814/3531814_submitted.pkl
Trial-Index: 63
slurmstepd: error: *** JOB 3531814 ON c57 CANCELLED AT 2026-05-11T14:42:19 DUE TO TIME LIMIT ***
SOBOL, failed: 62, 8 done, running/timeout 9/1 = ∑10/20, waiting for 10 jobs : 1%|░░░░░░░░░░| 8/1000 [50:38:48<7089:37:23, 25728.47s/it]
⚠ Job 3531815 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531815/3531815_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531815/3531815_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:04:54,062) - Starting with JobEnvironment(job_id=3531815, hostname=c136, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:04:54,064) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531815/3531815_submitted.pkl
Trial-Index: 73
slurmstepd: error: *** JOB 3531815 ON c136 CANCELLED AT 2026-05-11T15:05:19 DUE TO TIME LIMIT ***
SOBOL, failed: 63, 8 done, timeout/running 5/4 = ∑9/20, waiting for 9 jobs : 1%|░░░░░░░░░░| 8/1000 [50:48:48<7089:37:23, 25728.47s/it]
⚠ Job 3531817 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531817/3531817_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531817/3531817_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:10:56,638) - Starting with JobEnvironment(job_id=3531817, hostname=c144, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:10:56,640) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531817/3531817_submitted.pkl
Trial-Index: 64
slurmstepd: error: *** JOB 3531817 ON c144 CANCELLED AT 2026-05-11T15:11:19 DUE TO TIME LIMIT ***
⚠ Job 3531816 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531816/3531816_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531816/3531816_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:08:55,097) - Starting with JobEnvironment(job_id=3531816, hostname=c99, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:08:55,099) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531816/3531816_submitted.pkl
Trial-Index: 74
slurmstepd: error: *** JOB 3531816 ON c99 CANCELLED AT 2026-05-11T15:09:19 DUE TO TIME LIMIT ***
⚠ Job 3531818 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531818/3531818_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531818/3531818_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:10:55,833) - Starting with JobEnvironment(job_id=3531818, hostname=c53, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:10:55,835) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531818/3531818_submitted.pkl
Trial-Index: 70
slurmstepd: error: *** JOB 3531818 ON c53 CANCELLED AT 2026-05-11T15:11:19 DUE TO TIME LIMIT ***
⚠ Job 3531820 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531820/3531820_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531820/3531820_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:16:27,570) - Starting with JobEnvironment(job_id=3531820, hostname=c112, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:16:27,572) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531820/3531820_submitted.pkl
Trial-Index: 69
slurmstepd: error: *** JOB 3531820 ON c112 CANCELLED AT 2026-05-11T15:16:49 DUE TO TIME LIMIT ***
⚠ Job 3531819 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531819/3531819_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531819/3531819_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:11:26,253) - Starting with JobEnvironment(job_id=3531819, hostname=c93, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:11:26,255) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531819/3531819_submitted.pkl
Trial-Index: 71
slurmstepd: error: *** JOB 3531819 ON c93 CANCELLED AT 2026-05-11T15:11:49 DUE TO TIME LIMIT ***
SOBOL, failed: 68, 8 done, timeout 4 = ∑4/20, waiting for 4 jobs : 1%|░░░░░░░░░░| 8/1000 [50:58:50<7089:37:23, 25728.47s/it]
⚠ Job 3531824 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531824/3531824_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531824/3531824_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:19:27,568) - Starting with JobEnvironment(job_id=3531824, hostname=c96, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:19:27,570) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531824/3531824_submitted.pkl
Trial-Index: 77
slurmstepd: error: *** JOB 3531824 ON c96 CANCELLED AT 2026-05-11T15:19:49 DUE TO TIME LIMIT ***
⚠ Job 3531822 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531822/3531822_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531822/3531822_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:18:27,516) - Starting with JobEnvironment(job_id=3531822, hostname=c37, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:18:27,518) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531822/3531822_submitted.pkl
Trial-Index: 78
slurmstepd: error: *** JOB 3531822 ON c37 CANCELLED AT 2026-05-11T15:18:49 DUE TO TIME LIMIT ***
⚠ Job 3531823 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531823/3531823_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531823/3531823_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:19:28,140) - Starting with JobEnvironment(job_id=3531823, hostname=c120, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:19:28,142) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531823/3531823_submitted.pkl
Trial-Index: 79
slurmstepd: error: *** JOB 3531823 ON c120 CANCELLED AT 2026-05-11T15:19:49 DUE TO TIME LIMIT ***
⚠ Job 3531821 (task: 0) with path /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531821/3531821_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531821/3531821_0_log.out
----------------------------------------
submitit INFO (2026-05-11 03:18:27,922) - Starting with JobEnvironment(job_id=3531821, hostname=c104, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-11 03:18:27,924) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3531821/3531821_submitted.pkl
Trial-Index: 76
slurmstepd: error: *** JOB 3531821 ON c104 CANCELLED AT 2026-05-11T15:18:49 DUE TO TIME LIMIT ***
SOBOL, failed: 72, 8 done, running 20 = ∑20/20, new result: VAL_LOSS: 0.693252, PARAMS: 32808.000000: 1%|░░░░░░░░░░| 9/1000 [74:00:28<13803:53:52, 50145.34s/it]
2026-05-09 12:28:04 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, Started OmniOpt2 run...
2026-05-09 12:28:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #1/20
2026-05-09 12:28:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #2/20
2026-05-09 12:28:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #3/20
2026-05-09 12:28:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #4/20
2026-05-09 12:28:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #5/20
2026-05-09 12:28:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #6/20
2026-05-09 12:28:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #7/20
2026-05-09 12:28:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #8/20
2026-05-09 12:28:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #9/20
2026-05-09 12:28:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #10/20
2026-05-09 12:28:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #11/20
2026-05-09 12:28:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #12/20
2026-05-09 12:28:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #13/20
2026-05-09 12:28:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #14/20
2026-05-09 12:28:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #15/20
2026-05-09 12:28:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #16/20
2026-05-09 12:28:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #17/20
2026-05-09 12:28:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #18/20
2026-05-09 12:28:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #19/20
2026-05-09 12:28:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, getting new HP set #20/20
2026-05-09 12:28:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, requested 20 jobs, got 20, 0.22 s/job
2026-05-09 12:28:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #1/20 start
2026-05-09 12:28:10 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #2/20 start
2026-05-09 12:28:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #3/20 start
2026-05-09 12:28:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #4/20 start
2026-05-09 12:28:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #5/20 start
2026-05-09 12:28:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #6/20 start
2026-05-09 12:28:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #7/20 start
2026-05-09 12:28:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #8/20 start
2026-05-09 12:28:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #9/20 start
2026-05-09 12:28:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #10/20 start
2026-05-09 12:28:14 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #11/20 start
2026-05-09 12:28:14 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #12/20 start
2026-05-09 12:28:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #13/20 start
2026-05-09 12:28:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #14/20 start
2026-05-09 12:28:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #15/20 start
2026-05-09 12:28:17 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #16/20 start
2026-05-09 12:28:18 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #17/20 start
2026-05-09 12:28:18 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #18/20 start
2026-05-09 12:28:19 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #19/20 start
2026-05-09 12:28:19 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, eval #20/20 start
2026-05-09 12:28:20 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, starting new job
2026-05-09 12:28:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, unknown 1 = ∑1/20, started new job
2026-05-09 12:28:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, unknown 4 = ∑4/20, started new job
2026-05-09 12:28:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, unknown 5 = ∑5/20, started new job
2026-05-09 12:28:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, unknown 6 = ∑6/20, starting new job
2026-05-09 12:28:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, unknown 6 = ∑6/20, started new job
2026-05-09 12:28:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, unknown 7 = ∑7/20, started new job
2026-05-09 12:28:22 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, unknown 8 = ∑8/20, started new job
2026-05-09 12:28:22 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, unknown 9 = ∑9/20, started new job
2026-05-09 12:28:22 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, unknown 10 = ∑10/20, started new job
2026-05-09 12:28:23 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/1 = ∑12/20, started new job
2026-05-09 12:28:23 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/2 = ∑13/20, started new job
2026-05-09 12:28:23 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/3 = ∑14/20, started new job
2026-05-09 12:28:23 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/4 = ∑15/20, started new job
2026-05-09 12:28:24 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/5 = ∑16/20, started new job
2026-05-09 12:28:24 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/6 = ∑17/20, started new job
2026-05-09 12:28:24 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/7 = ∑18/20, started new job
2026-05-09 12:28:24 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/8 = ∑19/20, started new job
2026-05-09 12:28:24 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/9 = ∑20/20, started new job
2026-05-09 12:28:25 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending/unknown 10/1/9 = ∑20/20, waiting for 20 jobs
2026-05-09 12:28:29 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending 10/10 = ∑20/20, waiting for 20 jobs
2026-05-09 12:29:04 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running/pending 18/2 = ∑20/20, waiting for 20 jobs
2026-05-09 12:29:44 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running 20 = ∑20/20, waiting for 20 jobs
2026-05-09 14:04:51 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, running 20 = ∑20/20, new result: VAL_LOSS: 0.693155, PARAMS: 42624.000000
2026-05-09 14:04:53 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, 1 done, running 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-09 14:04:53 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, 1 done, running 19 = ∑19/20, waiting for 19 jobs
2026-05-09 22:26:29 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, 1 done, running 19 = ∑19/20, new result: VAL_LOSS: 0.693097, PARAMS: 3136.000000
2026-05-09 22:26:36 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, 2 done, running 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-09 22:26:36 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, 2 done, running 18 = ∑18/20, waiting for 18 jobs
2026-05-09 23:53:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, 2 done, running 18 = ∑18/20, new result: VAL_LOSS: 0.693089, PARAMS: 8796.000000
2026-05-09 23:53:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, 3 done, running 17 = ∑17/20, waiting for 18 jobs, finished 1 job
2026-05-09 23:53:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, 3 done, running 17 = ∑17/20, waiting for 17 jobs
2026-05-10 00:29:48 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, 3 done, timeout 17 = ∑17/20, job_failed
2026-05-10 00:29:55 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 16, 3 done, timeout 1 = ∑1/20, job_failed
2026-05-10 00:29:56 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, waiting for 17 jobs, finished 17 jobs
2026-05-10 00:29:58 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #1/20
2026-05-10 00:29:59 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #2/20
2026-05-10 00:29:59 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #3/20
2026-05-10 00:29:59 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #4/20
2026-05-10 00:30:00 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #5/20
2026-05-10 00:30:00 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #6/20
2026-05-10 00:30:00 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #7/20
2026-05-10 00:30:01 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #8/20
2026-05-10 00:30:01 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #9/20
2026-05-10 00:30:01 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #10/20
2026-05-10 00:30:01 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #11/20
2026-05-10 00:30:02 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #12/20
2026-05-10 00:30:02 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #13/20
2026-05-10 00:30:02 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #14/20
2026-05-10 00:30:03 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #15/20
2026-05-10 00:30:03 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #16/20
2026-05-10 00:30:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #17/20
2026-05-10 00:30:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #18/20
2026-05-10 00:30:06 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #19/20
2026-05-10 00:30:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, getting new HP set #20/20
2026-05-10 00:30:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, requested 20 jobs, got 20, 0.46 s/job
2026-05-10 00:30:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #1/20 start
2026-05-10 00:30:10 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #2/20 start
2026-05-10 00:30:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #3/20 start
2026-05-10 00:30:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #4/20 start
2026-05-10 00:30:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #5/20 start
2026-05-10 00:30:14 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #6/20 start
2026-05-10 00:30:15 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #7/20 start
2026-05-10 00:30:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #8/20 start
2026-05-10 00:30:18 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #9/20 start
2026-05-10 00:30:19 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #10/20 start
2026-05-10 00:30:20 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #11/20 start
2026-05-10 00:30:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #12/20 start
2026-05-10 00:30:23 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #13/20 start
2026-05-10 00:30:23 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #14/20 start
2026-05-10 00:30:24 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #15/20 start
2026-05-10 00:30:25 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #16/20 start
2026-05-10 00:30:26 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #17/20 start
2026-05-10 00:30:27 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #18/20 start
2026-05-10 00:30:28 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #19/20 start
2026-05-10 00:30:29 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, eval #20/20 start
2026-05-10 00:30:36 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, starting new job
2026-05-10 00:30:43 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, pending 12 = ∑12/20, started new job
2026-05-10 00:30:44 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, pending/unknown 12/2 = ∑14/20, started new job
2026-05-10 00:30:45 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, pending/unknown 12/4 = ∑16/20, started new job
2026-05-10 00:30:46 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, pending 16 = ∑16/20, started new job
2026-05-10 00:30:47 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, pending 16 = ∑16/20, starting new job
2026-05-10 00:30:49 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, pending/unknown 16/2 = ∑18/20, started new job
2026-05-10 00:30:49 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, pending/unknown 16/4 = ∑20/20, started new job
2026-05-10 00:30:51 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-10 00:31:01 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, running/pending 10/10 = ∑20/20, waiting for 20 jobs
2026-05-10 00:31:38 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, running/pending 15/5 = ∑20/20, waiting for 20 jobs
2026-05-10 00:32:28 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, running 20 = ∑20/20, waiting for 20 jobs
2026-05-10 01:26:30 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 3 done, running 20 = ∑20/20, new result: VAL_LOSS: 0.693795, PARAMS: 5720.000000
2026-05-10 01:26:41 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 4 done, running 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-10 01:26:41 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 4 done, running 19 = ∑19/20, waiting for 19 jobs
2026-05-10 06:12:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 4 done, running 19 = ∑19/20, new result: VAL_LOSS: 0.693265, PARAMS: 11296.000000
2026-05-10 06:12:28 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 5 done, running 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-10 06:12:28 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 5 done, running 18 = ∑18/20, waiting for 18 jobs
2026-05-10 12:34:32 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 5 done, timeout 18 = ∑18/20, waiting for 18 jobs
2026-05-10 12:34:55 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 17, 5 done, timeout 18 = ∑18/20, job_failed
2026-05-10 12:35:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, waiting for 18 jobs, finished 18 jobs
2026-05-10 12:35:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #1/20
2026-05-10 12:35:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #2/20
2026-05-10 12:35:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #3/20
2026-05-10 12:35:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #4/20
2026-05-10 12:35:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #5/20
2026-05-10 12:35:10 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #6/20
2026-05-10 12:35:10 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #7/20
2026-05-10 12:35:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #8/20
2026-05-10 12:35:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #9/20
2026-05-10 12:35:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #10/20
2026-05-10 12:35:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #11/20
2026-05-10 12:35:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #12/20
2026-05-10 12:35:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #13/20
2026-05-10 12:35:14 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #14/20
2026-05-10 12:35:14 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #15/20
2026-05-10 12:35:15 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #16/20
2026-05-10 12:35:15 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #17/20
2026-05-10 12:35:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #18/20
2026-05-10 12:35:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #19/20
2026-05-10 12:35:17 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, getting new HP set #20/20
2026-05-10 12:35:17 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, requested 20 jobs, got 20, 0.50 s/job
2026-05-10 12:35:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #1/20 start
2026-05-10 12:35:23 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #2/20 start
2026-05-10 12:35:24 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #3/20 start
2026-05-10 12:35:25 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #4/20 start
2026-05-10 12:35:27 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #5/20 start
2026-05-10 12:35:28 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #6/20 start
2026-05-10 12:35:30 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #7/20 start
2026-05-10 12:35:32 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #8/20 start
2026-05-10 12:35:34 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #9/20 start
2026-05-10 12:35:35 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #10/20 start
2026-05-10 12:35:36 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #11/20 start
2026-05-10 12:35:38 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #12/20 start
2026-05-10 12:35:40 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #13/20 start
2026-05-10 12:35:42 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #14/20 start
2026-05-10 12:35:44 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #15/20 start
2026-05-10 12:35:46 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #16/20 start
2026-05-10 12:35:47 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #17/20 start
2026-05-10 12:35:49 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #18/20 start
2026-05-10 12:35:50 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #19/20 start
2026-05-10 12:35:52 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, eval #20/20 start
2026-05-10 12:36:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, starting new job
2026-05-10 12:36:17 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, pending 16 = ∑16/20, started new job
2026-05-10 12:36:24 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, pending 16 = ∑16/20, starting new job
2026-05-10 12:36:27 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, pending/unknown 16/4 = ∑20/20, started new job
2026-05-10 12:36:31 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, running/pending 10/10 = ∑20/20, waiting for 20 jobs
2026-05-10 12:37:01 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, running 20 = ∑20/20, waiting for 20 jobs
2026-05-10 21:57:20 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 5 done, running 20 = ∑20/20, new result: VAL_LOSS: 0.680355, PARAMS: 17118.000000
2026-05-10 21:57:40 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 6 done, running 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-10 21:57:41 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 6 done, running 19 = ∑19/20, waiting for 19 jobs
2026-05-10 22:59:52 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 6 done, running 19 = ∑19/20, new result: VAL_LOSS: 0.693259, PARAMS: 4670.000000
2026-05-10 23:00:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 7 done, running 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-10 23:00:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 7 done, running 18 = ∑18/20, waiting for 18 jobs
2026-05-11 00:46:19 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 7 done, timeout 18 = ∑18/20, waiting for 18 jobs
2026-05-11 00:46:45 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 35, 7 done, timeout 18 = ∑18/20, job_failed
2026-05-11 00:46:59 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, waiting for 18 jobs, finished 18 jobs
2026-05-11 00:47:02 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #1/20
2026-05-11 00:47:03 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #2/20
2026-05-11 00:47:04 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #3/20
2026-05-11 00:47:04 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #4/20
2026-05-11 00:47:05 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #5/20
2026-05-11 00:47:06 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #6/20
2026-05-11 00:47:06 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #7/20
2026-05-11 00:47:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #8/20
2026-05-11 00:47:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #9/20
2026-05-11 00:47:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #10/20
2026-05-11 00:47:10 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #11/20
2026-05-11 00:47:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #12/20
2026-05-11 00:47:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #13/20
2026-05-11 00:47:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #14/20
2026-05-11 00:47:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #15/20
2026-05-11 00:47:14 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #16/20
2026-05-11 00:47:15 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #17/20
2026-05-11 00:47:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #18/20
2026-05-11 00:47:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #19/20
2026-05-11 00:47:17 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, getting new HP set #20/20
2026-05-11 00:47:18 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, requested 20 jobs, got 20, 0.82 s/job
2026-05-11 00:47:23 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #1/20 start
2026-05-11 00:47:25 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #2/20 start
2026-05-11 00:47:27 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #3/20 start
2026-05-11 00:47:30 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #4/20 start
2026-05-11 00:47:34 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #5/20 start
2026-05-11 00:47:36 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #6/20 start
2026-05-11 00:47:39 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #7/20 start
2026-05-11 00:47:41 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #8/20 start
2026-05-11 00:47:44 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #9/20 start
2026-05-11 00:47:47 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #10/20 start
2026-05-11 00:47:49 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #11/20 start
2026-05-11 00:47:51 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #12/20 start
2026-05-11 00:47:53 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #13/20 start
2026-05-11 00:47:57 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #14/20 start
2026-05-11 00:48:00 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #15/20 start
2026-05-11 00:48:04 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #16/20 start
2026-05-11 00:48:06 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #17/20 start
2026-05-11 00:48:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #18/20 start
2026-05-11 00:48:10 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #19/20 start
2026-05-11 00:48:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, eval #20/20 start
2026-05-11 00:48:28 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, starting new job
2026-05-11 00:48:45 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, pending 14 = ∑14/20, started new job
2026-05-11 00:48:48 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, pending 15 = ∑15/20, started new job
2026-05-11 00:48:49 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, pending 16 = ∑16/20, started new job
2026-05-11 00:48:55 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, pending 16 = ∑16/20, starting new job
2026-05-11 00:49:00 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, pending 20 = ∑20/20, started new job
2026-05-11 00:49:02 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-11 01:36:14 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, running/pending 1/19 = ∑20/20, waiting for 20 jobs
2026-05-11 02:06:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, running/pending 3/17 = ∑20/20, waiting for 20 jobs
2026-05-11 02:16:15 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, running/pending 5/15 = ∑20/20, waiting for 20 jobs
2026-05-11 02:36:16 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, running/pending 8/12 = ∑20/20, waiting for 20 jobs
2026-05-11 02:46:17 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, running/pending 10/10 = ∑20/20, waiting for 20 jobs
2026-05-11 03:06:19 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, running/pending 11/9 = ∑20/20, waiting for 20 jobs
2026-05-11 03:16:19 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, running/pending 15/5 = ∑20/20, waiting for 20 jobs
2026-05-11 03:26:20 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, running 20 = ∑20/20, waiting for 20 jobs
2026-05-11 09:37:58 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 7 done, running 20 = ∑20/20, new result: VAL_LOSS: 0.669041, PARAMS: 11204.000000
2026-05-11 09:38:30 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 8 done, running 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-11 09:38:31 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 8 done, running 19 = ∑19/20, waiting for 19 jobs
2026-05-11 14:07:06 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 53, 8 done, timeout/running 3/16 = ∑19/20, job_failed
2026-05-11 14:07:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 56, 8 done, running 16 = ∑16/20, waiting for 19 jobs, finished 3 jobs
2026-05-11 14:07:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 56, 8 done, running 16 = ∑16/20, waiting for 16 jobs
2026-05-11 14:16:50 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 56, 8 done, running/timeout 15/1 = ∑16/20, waiting for 16 jobs
2026-05-11 14:17:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 56, 8 done, running/timeout 15/1 = ∑16/20, job_failed
2026-05-11 14:17:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 57, 8 done, running 15 = ∑15/20, waiting for 16 jobs, finished 1 job
2026-05-11 14:17:10 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 57, 8 done, running 15 = ∑15/20, waiting for 15 jobs
2026-05-11 14:36:51 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 57, 8 done, timeout/running 3/12 = ∑15/20, waiting for 15 jobs
2026-05-11 14:37:09 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 57, 8 done, timeout/running 3/12 = ∑15/20, job_failed
2026-05-11 14:37:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 60, 8 done, running 12 = ∑12/20, waiting for 15 jobs, finished 3 jobs
2026-05-11 14:37:14 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 60, 8 done, running 12 = ∑12/20, waiting for 12 jobs
2026-05-11 14:46:51 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 60, 8 done, timeout/running 2/10 = ∑12/20, waiting for 12 jobs
2026-05-11 14:47:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 60, 8 done, timeout/running 2/10 = ∑12/20, job_failed
2026-05-11 14:47:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 62, 8 done, running 10 = ∑10/20, waiting for 12 jobs, finished 2 jobs
2026-05-11 14:47:12 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 62, 8 done, running 10 = ∑10/20, waiting for 10 jobs
2026-05-11 15:06:52 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 62, 8 done, running/timeout 9/1 = ∑10/20, waiting for 10 jobs
2026-05-11 15:07:08 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 62, 8 done, running/timeout 9/1 = ∑10/20, job_failed
2026-05-11 15:07:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 63, 8 done, running 9 = ∑9/20, waiting for 10 jobs, finished 1 job
2026-05-11 15:07:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 63, 8 done, running 9 = ∑9/20, waiting for 9 jobs
2026-05-11 15:16:52 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 63, 8 done, timeout/running 5/4 = ∑9/20, waiting for 9 jobs
2026-05-11 15:17:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 63, 8 done, timeout/running 5/4 = ∑9/20, job_failed
2026-05-11 15:17:17 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 68, 8 done, running 4 = ∑4/20, waiting for 9 jobs, finished 5 jobs
2026-05-11 15:17:18 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 68, 8 done, running 4 = ∑4/20, waiting for 4 jobs
2026-05-11 15:26:54 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 68, 8 done, timeout 4 = ∑4/20, waiting for 4 jobs
2026-05-11 15:27:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 68, 8 done, timeout 4 = ∑4/20, job_failed
2026-05-11 15:27:18 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, waiting for 4 jobs, finished 4 jobs
2026-05-11 15:27:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #1/20
2026-05-11 15:27:22 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #2/20
2026-05-11 15:27:25 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #3/20
2026-05-11 15:27:26 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #4/20
2026-05-11 15:27:26 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #5/20
2026-05-11 15:27:27 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #6/20
2026-05-11 15:27:29 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #7/20
2026-05-11 15:27:30 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #8/20
2026-05-11 15:27:31 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #9/20
2026-05-11 15:27:33 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #10/20
2026-05-11 15:27:34 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #11/20
2026-05-11 15:27:35 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #12/20
2026-05-11 15:27:36 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #13/20
2026-05-11 15:27:37 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #14/20
2026-05-11 15:27:38 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #15/20
2026-05-11 15:27:39 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #16/20
2026-05-11 15:27:40 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #17/20
2026-05-11 15:27:41 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #18/20
2026-05-11 15:27:42 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #19/20
2026-05-11 15:27:43 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, getting new HP set #20/20
2026-05-11 15:27:44 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, requested 20 jobs, got 20, 1.15 s/job
2026-05-11 15:27:47 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #1/20 start
2026-05-11 15:27:51 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #2/20 start
2026-05-11 15:27:55 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #3/20 start
2026-05-11 15:27:58 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #4/20 start
2026-05-11 15:28:01 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #5/20 start
2026-05-11 15:28:03 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #6/20 start
2026-05-11 15:28:07 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #7/20 start
2026-05-11 15:28:11 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #8/20 start
2026-05-11 15:28:13 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #9/20 start
2026-05-11 15:28:17 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #10/20 start
2026-05-11 15:28:21 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #11/20 start
2026-05-11 15:28:24 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #12/20 start
2026-05-11 15:28:29 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #13/20 start
2026-05-11 15:28:31 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #14/20 start
2026-05-11 15:28:40 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #15/20 start
2026-05-11 15:28:43 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #16/20 start
2026-05-11 15:28:47 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #17/20 start
2026-05-11 15:28:50 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #18/20 start
2026-05-11 15:28:55 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #19/20 start
2026-05-11 15:28:58 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, eval #20/20 start
2026-05-11 15:29:20 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, starting new job
2026-05-11 15:29:41 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, pending 15 = ∑15/20, started new job
2026-05-11 15:29:42 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, pending 16 = ∑16/20, started new job
2026-05-11 15:29:52 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, pending 16 = ∑16/20, starting new job
2026-05-11 15:29:57 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, pending/unknown 19/1 = ∑20/20, started new job
2026-05-11 15:30:01 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, pending/unknown 19/1 = ∑20/20, waiting for 20 jobs
2026-05-11 15:30:02 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, pending 20 = ∑20/20, waiting for 20 jobs
2026-05-12 03:50:58 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, running/pending 9/11 = ∑20/20, waiting for 20 jobs
2026-05-12 04:00:59 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, running 20 = ∑20/20, waiting for 20 jobs
2026-05-12 14:28:31 (dbd6dd28-3604-4ab4-a330-8f5d7c543648): SOBOL, failed: 72, 8 done, running 20 = ∑20/20, new result: VAL_LOSS: 0.693252, PARAMS: 32808.000000
submitit INFO (2026-05-09 12:28:23,752) - Starting with JobEnvironment(job_id=3514198, hostname=c154, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-09 12:28:23,758) - Loading pickle: /data/horse/ws/s3811141-grok3/omniopt/runs/tinyLogicLLM5/0/single_runs/3514198/3514198_submitted.pkl
Trial-Index: 1
slurmstepd: error: *** JOB 3514198 ON c154 CANCELLED AT 2026-05-10T00:28:24 DUE TO TIME LIMIT ***