Parameters ranked by absolute Pearson correlation with each result. Stronger correlations (|r| ≥ 0.7) suggest the parameter has more influence on the outcome.
Convergence Analysis
Parameter statistics
Parameter
Min
Max
Mean
Std Dev
Count
run_time
136
3548
1439.4508
945.8402
508
OO_Info_ZEIT
118.858
3514.7824
1390.063
937.8496
508
RESULT
0.001
0.0073
0.0026
0.0011
508
epochs
100
300
260.842
48.9588
804
filters
50
150
107.6306
40.1747
804
window
5
15
10.0323
4.1882
804
stride
1
5
2.0547
1.6578
804
dense
100
280
220.0547
65.0769
804
batchsz
4
32
11.1219
9.0404
804
lr
0
0.0005
0.0002
0.0001
804
âš Job 40547855 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547855/40547855_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547855/40547855_0_log.out
----------------------------------------
submitit INFO (2026-11-02 16:44:53,491) - Starting with JobEnvironment(job_id=40547855, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 16:44:53,491) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547855/40547855_submitted.pkl
Trial-Index: 130
[2026-11-02T17:44:56.004] error: *** JOB 40547855 ON n1450 CANCELLED AT 2026-11-02T17:44:56 DUE TO TIME LIMIT ***
âš Job 40547849 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547849/40547849_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547849/40547849_0_log.out
----------------------------------------
submitit INFO (2026-11-02 16:44:41,292) - Starting with JobEnvironment(job_id=40547849, hostname=n1609, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 16:44:41,293) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547849/40547849_submitted.pkl
Trial-Index: 112
[2026-11-02T17:44:56.004] error: *** JOB 40547849 ON n1609 CANCELLED AT 2026-11-02T17:44:56 DUE TO TIME LIMIT ***
âš Job 40547850 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547850/40547850_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547850/40547850_0_log.out
----------------------------------------
submitit INFO (2026-11-02 16:44:41,293) - Starting with JobEnvironment(job_id=40547850, hostname=n1609, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 16:44:41,293) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547850/40547850_submitted.pkl
Trial-Index: 121
[2026-11-02T17:44:56.004] error: *** JOB 40547850 ON n1609 CANCELLED AT 2026-11-02T17:44:56 DUE TO TIME LIMIT ***
âš Job 40547825 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547825/40547825_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547825/40547825_0_log.out
----------------------------------------
submitit INFO (2026-11-02 16:44:13,044) - Starting with JobEnvironment(job_id=40547825, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 16:44:13,045) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40547825/40547825_submitted.pkl
Trial-Index: 100
[2026-11-02T17:44:26.003] error: *** JOB 40547825 ON n1713 CANCELLED AT 2026-11-02T17:44:26 DUE TO TIME LIMIT ***
âš Job 40548373 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548373/40548373_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548373/40548373_0_log.out
----------------------------------------
submitit INFO (2026-11-02 17:58:39,239) - Starting with JobEnvironment(job_id=40548373, hostname=n1669, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 17:58:39,239) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548373/40548373_submitted.pkl
Trial-Index: 170
[2026-11-02T18:58:57.004] error: *** JOB 40548373 ON n1669 CANCELLED AT 2026-11-02T18:58:57 DUE TO TIME LIMIT ***
âš Job 40548331 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548331/40548331_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548331/40548331_0_log.out
----------------------------------------
submitit INFO (2026-11-02 17:56:38,809) - Starting with JobEnvironment(job_id=40548331, hostname=n1635, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 17:56:38,810) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548331/40548331_submitted.pkl
Trial-Index: 160
[2026-11-02T18:56:56.040] error: *** JOB 40548331 ON n1635 CANCELLED AT 2026-11-02T18:56:56 DUE TO TIME LIMIT ***
âš Job 40548335 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548335/40548335_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548335/40548335_0_log.out
----------------------------------------
submitit INFO (2026-11-02 17:57:09,251) - Starting with JobEnvironment(job_id=40548335, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 17:57:09,251) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548335/40548335_submitted.pkl
Trial-Index: 163
[2026-11-02T18:57:27.005] error: *** JOB 40548335 ON n1627 CANCELLED AT 2026-11-02T18:57:27 DUE TO TIME LIMIT ***
âš Job 40548369 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548369/40548369_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548369/40548369_0_log.out
----------------------------------------
submitit INFO (2026-11-02 17:58:39,237) - Starting with JobEnvironment(job_id=40548369, hostname=n1669, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 17:58:39,240) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548369/40548369_submitted.pkl
Trial-Index: 199
[2026-11-02T18:58:57.004] error: *** JOB 40548369 ON n1669 CANCELLED AT 2026-11-02T18:58:57 DUE TO TIME LIMIT ***
âš Job 40548363 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548363/40548363_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548363/40548363_0_log.out
----------------------------------------
submitit INFO (2026-11-02 17:58:08,941) - Starting with JobEnvironment(job_id=40548363, hostname=n1664, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 17:58:08,943) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548363/40548363_submitted.pkl
Trial-Index: 195
[2026-11-02T18:58:26.006] error: *** JOB 40548363 ON n1664 CANCELLED AT 2026-11-02T18:58:26 DUE TO TIME LIMIT ***
âš Job 40548325 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548325/40548325_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548325/40548325_0_log.out
----------------------------------------
submitit INFO (2026-11-02 17:56:38,819) - Starting with JobEnvironment(job_id=40548325, hostname=n1635, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 17:56:38,820) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40548325/40548325_submitted.pkl
Trial-Index: 162
[2026-11-02T18:56:56.040] error: *** JOB 40548325 ON n1635 CANCELLED AT 2026-11-02T18:56:56 DUE TO TIME LIMIT ***
âš Job 40549495 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549495/40549495_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549495/40549495_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:09,836) - Starting with JobEnvironment(job_id=40549495, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:09,837) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549495/40549495_submitted.pkl
Trial-Index: 200
[2026-11-02T20:13:27.005] error: *** JOB 40549495 ON n1700 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549492 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549492/40549492_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549492/40549492_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:09,836) - Starting with JobEnvironment(job_id=40549492, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:09,837) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549492/40549492_submitted.pkl
Trial-Index: 233
[2026-11-02T20:13:27.004] error: *** JOB 40549492 ON n1700 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549479 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549479/40549479_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549479/40549479_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:12:52,178) - Starting with JobEnvironment(job_id=40549479, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:12:52,179) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549479/40549479_submitted.pkl
Trial-Index: 215
[2026-11-02T20:12:57.037] error: *** JOB 40549479 ON n1713 CANCELLED AT 2026-11-02T20:12:57 DUE TO TIME LIMIT ***
âš Job 40549494 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549494/40549494_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549494/40549494_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:09,836) - Starting with JobEnvironment(job_id=40549494, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:09,837) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549494/40549494_submitted.pkl
Trial-Index: 206
[2026-11-02T20:13:27.004] error: *** JOB 40549494 ON n1700 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549499 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549499/40549499_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549499/40549499_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:09,836) - Starting with JobEnvironment(job_id=40549499, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:09,837) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549499/40549499_submitted.pkl
Trial-Index: 202
[2026-11-02T20:13:27.005] error: *** JOB 40549499 ON n1700 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549489 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549489/40549489_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549489/40549489_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:12:53,906) - Starting with JobEnvironment(job_id=40549489, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:12:53,907) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549489/40549489_submitted.pkl
Trial-Index: 222
[2026-11-02T20:12:57.037] error: *** JOB 40549489 ON n1700 CANCELLED AT 2026-11-02T20:12:57 DUE TO TIME LIMIT ***
âš Job 40549493 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549493/40549493_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549493/40549493_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:09,836) - Starting with JobEnvironment(job_id=40549493, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:09,837) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549493/40549493_submitted.pkl
Trial-Index: 208
[2026-11-02T20:13:27.004] error: *** JOB 40549493 ON n1700 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549506 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549506/40549506_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549506/40549506_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:16,054) - Starting with JobEnvironment(job_id=40549506, hostname=n1634, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:16,054) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549506/40549506_submitted.pkl
Trial-Index: 242
[2026-11-02T20:13:27.007] error: *** JOB 40549506 ON n1634 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549514 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549514/40549514_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549514/40549514_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:16,020) - Starting with JobEnvironment(job_id=40549514, hostname=n1634, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:16,021) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549514/40549514_submitted.pkl
Trial-Index: 243
[2026-11-02T20:13:27.007] error: *** JOB 40549514 ON n1634 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549500 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549500/40549500_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549500/40549500_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:09,836) - Starting with JobEnvironment(job_id=40549500, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:09,837) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549500/40549500_submitted.pkl
Trial-Index: 210
[2026-11-02T20:13:27.005] error: *** JOB 40549500 ON n1700 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549515 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549515/40549515_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549515/40549515_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:16,061) - Starting with JobEnvironment(job_id=40549515, hostname=n1634, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:16,062) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549515/40549515_submitted.pkl
Trial-Index: 234
[2026-11-02T20:13:27.007] error: *** JOB 40549515 ON n1634 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549505 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549505/40549505_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549505/40549505_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:16,025) - Starting with JobEnvironment(job_id=40549505, hostname=n1634, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:16,026) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549505/40549505_submitted.pkl
Trial-Index: 203
[2026-11-02T20:13:27.006] error: *** JOB 40549505 ON n1634 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40549488 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549488/40549488_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549488/40549488_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:12:59,350) - Starting with JobEnvironment(job_id=40549488, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:12:59,351) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549488/40549488_submitted.pkl
Trial-Index: 240
[2026-11-02T20:12:57.036] error: *** JOB 40549488 ON n1713 CANCELLED AT 2026-11-02T20:12:57 DUE TO TIME LIMIT ***
âš Job 40549476 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549476/40549476_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549476/40549476_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:12:52,178) - Starting with JobEnvironment(job_id=40549476, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:12:52,179) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549476/40549476_submitted.pkl
Trial-Index: 205
[2026-11-02T20:12:57.036] error: *** JOB 40549476 ON n1713 CANCELLED AT 2026-11-02T20:12:57 DUE TO TIME LIMIT ***
âš Job 40549480 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549480/40549480_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549480/40549480_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:12:59,350) - Starting with JobEnvironment(job_id=40549480, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:12:59,351) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549480/40549480_submitted.pkl
Trial-Index: 217
[2026-11-02T20:12:57.037] error: *** JOB 40549480 ON n1713 CANCELLED AT 2026-11-02T20:12:57 DUE TO TIME LIMIT ***
âš Job 40549509 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549509/40549509_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549509/40549509_0_log.out
----------------------------------------
submitit INFO (2026-11-02 19:13:16,067) - Starting with JobEnvironment(job_id=40549509, hostname=n1634, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 19:13:16,068) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40549509/40549509_submitted.pkl
Trial-Index: 228
[2026-11-02T20:13:27.007] error: *** JOB 40549509 ON n1634 CANCELLED AT 2026-11-02T20:13:27 DUE TO TIME LIMIT ***
âš Job 40550316 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550316/40550316_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550316/40550316_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:21:19,291) - Starting with JobEnvironment(job_id=40550316, hostname=n1677, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:21:19,292) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550316/40550316_submitted.pkl
Trial-Index: 299
[2026-11-02T21:21:27.009] error: *** JOB 40550316 ON n1677 CANCELLED AT 2026-11-02T21:21:27 DUE TO TIME LIMIT ***
âš Job 40550319 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550319/40550319_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550319/40550319_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:21:18,920) - Starting with JobEnvironment(job_id=40550319, hostname=n1677, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:21:18,921) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550319/40550319_submitted.pkl
Trial-Index: 277
[2026-11-02T21:21:27.009] error: *** JOB 40550319 ON n1677 CANCELLED AT 2026-11-02T21:21:27 DUE TO TIME LIMIT ***
âš Job 40550292 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550292/40550292_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550292/40550292_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:21:23,931) - Starting with JobEnvironment(job_id=40550292, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:21:23,932) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550292/40550292_submitted.pkl
Trial-Index: 290
[2026-11-02T21:21:27.009] error: *** JOB 40550292 ON n1700 CANCELLED AT 2026-11-02T21:21:27 DUE TO TIME LIMIT ***
âš Job 40550278 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550278/40550278_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550278/40550278_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:20:56,705) - Starting with JobEnvironment(job_id=40550278, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:20:56,705) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550278/40550278_submitted.pkl
Trial-Index: 295
[2026-11-02T21:20:57.034] error: *** JOB 40550278 ON n1700 CANCELLED AT 2026-11-02T21:20:57 DUE TO TIME LIMIT ***
âš Job 40550276 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550276/40550276_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550276/40550276_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:20:56,705) - Starting with JobEnvironment(job_id=40550276, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:20:56,706) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550276/40550276_submitted.pkl
Trial-Index: 253
[2026-11-02T21:20:57.034] error: *** JOB 40550276 ON n1700 CANCELLED AT 2026-11-02T21:20:57 DUE TO TIME LIMIT ***
âš Job 40550317 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550317/40550317_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550317/40550317_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:21:18,920) - Starting with JobEnvironment(job_id=40550317, hostname=n1677, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:21:18,921) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550317/40550317_submitted.pkl
Trial-Index: 288
[2026-11-02T21:21:27.010] error: *** JOB 40550317 ON n1677 CANCELLED AT 2026-11-02T21:21:27 DUE TO TIME LIMIT ***
âš Job 40550303 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550303/40550303_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550303/40550303_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:21:24,255) - Starting with JobEnvironment(job_id=40550303, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:21:24,256) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550303/40550303_submitted.pkl
Trial-Index: 291
[2026-11-02T21:21:27.008] error: *** JOB 40550303 ON n1700 CANCELLED AT 2026-11-02T21:21:27 DUE TO TIME LIMIT ***
âš Job 40550324 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550324/40550324_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550324/40550324_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:21:19,155) - Starting with JobEnvironment(job_id=40550324, hostname=n1649, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:21:19,156) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550324/40550324_submitted.pkl
Trial-Index: 283
[2026-11-02T21:21:27.010] error: *** JOB 40550324 ON n1649 CANCELLED AT 2026-11-02T21:21:27 DUE TO TIME LIMIT ***
âš Job 40550313 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550313/40550313_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550313/40550313_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:21:18,920) - Starting with JobEnvironment(job_id=40550313, hostname=n1677, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:21:18,921) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550313/40550313_submitted.pkl
Trial-Index: 271
[2026-11-02T21:21:27.009] error: *** JOB 40550313 ON n1677 CANCELLED AT 2026-11-02T21:21:27 DUE TO TIME LIMIT ***
âš Job 40550282 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550282/40550282_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550282/40550282_0_log.out
----------------------------------------
submitit INFO (2026-11-02 20:21:24,259) - Starting with JobEnvironment(job_id=40550282, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 20:21:24,260) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550282/40550282_submitted.pkl
Trial-Index: 289
[2026-11-02T21:21:27.007] error: *** JOB 40550282 ON n1700 CANCELLED AT 2026-11-02T21:21:27 DUE TO TIME LIMIT ***
âš Job 40550977 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550977/40550977_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550977/40550977_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:22,290) - Starting with JobEnvironment(job_id=40550977, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:22,290) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550977/40550977_submitted.pkl
Trial-Index: 318
[2026-11-02T22:32:27.006] error: *** JOB 40550977 ON n1700 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550982 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550982/40550982_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550982/40550982_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:30,641) - Starting with JobEnvironment(job_id=40550982, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:30,642) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550982/40550982_submitted.pkl
Trial-Index: 341
[2026-11-02T22:32:27.007] error: *** JOB 40550982 ON n1705 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550998 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550998/40550998_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550998/40550998_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:41,974) - Starting with JobEnvironment(job_id=40550998, hostname=n1694, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:41,977) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550998/40550998_submitted.pkl
Trial-Index: 330
[2026-11-02T22:32:58.005] error: *** JOB 40550998 ON n1694 CANCELLED AT 2026-11-02T22:32:58 DUE TO TIME LIMIT ***
âš Job 40550967 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550967/40550967_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550967/40550967_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:21,247) - Starting with JobEnvironment(job_id=40550967, hostname=n1701, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:21,248) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550967/40550967_submitted.pkl
Trial-Index: 337
[2026-11-02T22:32:27.006] error: *** JOB 40550967 ON n1701 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550961 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550961/40550961_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550961/40550961_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:22,003) - Starting with JobEnvironment(job_id=40550961, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:22,005) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550961/40550961_submitted.pkl
Trial-Index: 303
[2026-11-02T22:32:27.006] error: *** JOB 40550961 ON n1702 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550987 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550987/40550987_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550987/40550987_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:31,836) - Starting with JobEnvironment(job_id=40550987, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:31,837) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550987/40550987_submitted.pkl
Trial-Index: 324
[2026-11-02T22:32:27.013] error: *** JOB 40550987 ON n1700 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550968 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550968/40550968_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550968/40550968_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:21,247) - Starting with JobEnvironment(job_id=40550968, hostname=n1701, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:21,248) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550968/40550968_submitted.pkl
Trial-Index: 335
[2026-11-02T22:32:27.006] error: *** JOB 40550968 ON n1701 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550979 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550979/40550979_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550979/40550979_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:28,693) - Starting with JobEnvironment(job_id=40550979, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:28,694) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550979/40550979_submitted.pkl
Trial-Index: 312
[2026-11-02T22:32:27.007] error: *** JOB 40550979 ON n1711 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550981 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550981/40550981_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550981/40550981_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:30,641) - Starting with JobEnvironment(job_id=40550981, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:30,642) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550981/40550981_submitted.pkl
Trial-Index: 306
[2026-11-02T22:32:27.007] error: *** JOB 40550981 ON n1705 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550992 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550992/40550992_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550992/40550992_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:42,316) - Starting with JobEnvironment(job_id=40550992, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:42,319) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550992/40550992_submitted.pkl
Trial-Index: 302
[2026-11-02T22:32:58.004] error: *** JOB 40550992 ON n1700 CANCELLED AT 2026-11-02T22:32:58 DUE TO TIME LIMIT ***
âš Job 40550962 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550962/40550962_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550962/40550962_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:22,003) - Starting with JobEnvironment(job_id=40550962, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:22,003) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550962/40550962_submitted.pkl
Trial-Index: 314
[2026-11-02T22:32:27.006] error: *** JOB 40550962 ON n1702 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550991 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550991/40550991_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550991/40550991_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:43,304) - Starting with JobEnvironment(job_id=40550991, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:43,305) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550991/40550991_submitted.pkl
Trial-Index: 340
[2026-11-02T22:32:58.004] error: *** JOB 40550991 ON n1700 CANCELLED AT 2026-11-02T22:32:58 DUE TO TIME LIMIT ***
âš Job 40550975 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550975/40550975_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550975/40550975_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:22,290) - Starting with JobEnvironment(job_id=40550975, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:22,291) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550975/40550975_submitted.pkl
Trial-Index: 326
[2026-11-02T22:32:27.006] error: *** JOB 40550975 ON n1700 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40550978 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550978/40550978_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550978/40550978_0_log.out
----------------------------------------
submitit INFO (2026-11-02 21:32:22,290) - Starting with JobEnvironment(job_id=40550978, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 21:32:22,291) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40550978/40550978_submitted.pkl
Trial-Index: 331
[2026-11-02T22:32:27.012] error: *** JOB 40550978 ON n1700 CANCELLED AT 2026-11-02T22:32:27 DUE TO TIME LIMIT ***
âš Job 40551246 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551246/40551246_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551246/40551246_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:34,859) - Starting with JobEnvironment(job_id=40551246, hostname=n1532, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:34,860) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551246/40551246_submitted.pkl
Trial-Index: 389
[2026-11-02T23:45:28.007] error: *** JOB 40551246 ON n1532 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551241 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551241/40551241_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551241/40551241_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:26,344) - Starting with JobEnvironment(job_id=40551241, hostname=n1694, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:26,346) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551241/40551241_submitted.pkl
Trial-Index: 363
[2026-11-02T23:45:28.008] error: *** JOB 40551241 ON n1694 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551251 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551251/40551251_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551251/40551251_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:54,553) - Starting with JobEnvironment(job_id=40551251, hostname=n1530, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:54,554) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551251/40551251_submitted.pkl
Trial-Index: 371
[2026-11-02T23:45:28.008] error: *** JOB 40551251 ON n1530 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551209 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551209/40551209_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551209/40551209_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:44:50,601) - Starting with JobEnvironment(job_id=40551209, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:44:50,601) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551209/40551209_submitted.pkl
Trial-Index: 377
[2026-11-02T23:44:58.007] error: *** JOB 40551209 ON n1711 CANCELLED AT 2026-11-02T23:44:58 DUE TO TIME LIMIT ***
âš Job 40551229 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551229/40551229_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551229/40551229_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:10,227) - Starting with JobEnvironment(job_id=40551229, hostname=n1658, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:10,228) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551229/40551229_submitted.pkl
Trial-Index: 362
[2026-11-02T23:45:28.006] error: *** JOB 40551229 ON n1658 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551235 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551235/40551235_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551235/40551235_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:10,275) - Starting with JobEnvironment(job_id=40551235, hostname=n1658, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:10,275) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551235/40551235_submitted.pkl
Trial-Index: 394
[2026-11-02T23:45:28.006] error: *** JOB 40551235 ON n1658 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551226 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551226/40551226_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551226/40551226_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:10,260) - Starting with JobEnvironment(job_id=40551226, hostname=n1658, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:10,260) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551226/40551226_submitted.pkl
Trial-Index: 359
[2026-11-02T23:45:28.006] error: *** JOB 40551226 ON n1658 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551217 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551217/40551217_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551217/40551217_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:05,869) - Starting with JobEnvironment(job_id=40551217, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:05,870) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551217/40551217_submitted.pkl
Trial-Index: 383
[2026-11-02T23:45:28.005] error: *** JOB 40551217 ON n1711 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551244 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551244/40551244_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551244/40551244_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:34,860) - Starting with JobEnvironment(job_id=40551244, hostname=n1532, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:34,860) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551244/40551244_submitted.pkl
Trial-Index: 381
[2026-11-02T23:45:28.007] error: *** JOB 40551244 ON n1532 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551214 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551214/40551214_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551214/40551214_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:44:50,601) - Starting with JobEnvironment(job_id=40551214, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:44:50,601) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551214/40551214_submitted.pkl
Trial-Index: 395
[2026-11-02T23:44:58.007] error: *** JOB 40551214 ON n1711 CANCELLED AT 2026-11-02T23:44:58 DUE TO TIME LIMIT ***
âš Job 40551234 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551234/40551234_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551234/40551234_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:10,280) - Starting with JobEnvironment(job_id=40551234, hostname=n1658, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:10,280) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551234/40551234_submitted.pkl
Trial-Index: 366
[2026-11-02T23:45:28.006] error: *** JOB 40551234 ON n1658 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551219 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551219/40551219_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551219/40551219_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:05,869) - Starting with JobEnvironment(job_id=40551219, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:05,870) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551219/40551219_submitted.pkl
Trial-Index: 379
[2026-11-02T23:45:28.005] error: *** JOB 40551219 ON n1711 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551207 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551207/40551207_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551207/40551207_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:44:50,601) - Starting with JobEnvironment(job_id=40551207, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:44:50,601) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551207/40551207_submitted.pkl
Trial-Index: 350
[2026-11-02T23:44:58.007] error: *** JOB 40551207 ON n1711 CANCELLED AT 2026-11-02T23:44:58 DUE TO TIME LIMIT ***
âš Job 40551254 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551254/40551254_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551254/40551254_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:54,556) - Starting with JobEnvironment(job_id=40551254, hostname=n1530, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:54,557) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551254/40551254_submitted.pkl
Trial-Index: 365
[2026-11-02T23:45:28.008] error: *** JOB 40551254 ON n1530 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551249 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551249/40551249_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551249/40551249_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:54,555) - Starting with JobEnvironment(job_id=40551249, hostname=n1530, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:54,556) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551249/40551249_submitted.pkl
Trial-Index: 352
[2026-11-02T23:45:28.008] error: *** JOB 40551249 ON n1530 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40551245 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551245/40551245_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551245/40551245_0_log.out
----------------------------------------
submitit INFO (2026-11-02 22:45:34,857) - Starting with JobEnvironment(job_id=40551245, hostname=n1532, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-02 22:45:34,858) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40551245/40551245_submitted.pkl
Trial-Index: 356
[2026-11-02T23:45:28.007] error: *** JOB 40551245 ON n1532 CANCELLED AT 2026-11-02T23:45:28 DUE TO TIME LIMIT ***
âš Job 40552230 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552230/40552230_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552230/40552230_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:41,430) - Starting with JobEnvironment(job_id=40552230, hostname=n1588, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:41,431) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552230/40552230_submitted.pkl
Trial-Index: 433
[2026-11-03T01:05:58.008] error: *** JOB 40552230 ON n1588 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552210 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552210/40552210_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552210/40552210_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:33,269) - Starting with JobEnvironment(job_id=40552210, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:33,270) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552210/40552210_submitted.pkl
Trial-Index: 421
[2026-11-03T01:05:28.042] error: *** JOB 40552210 ON n1659 CANCELLED AT 2026-11-03T01:05:28 DUE TO TIME LIMIT ***
âš Job 40552217 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552217/40552217_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552217/40552217_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:33,283) - Starting with JobEnvironment(job_id=40552217, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:33,284) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552217/40552217_submitted.pkl
Trial-Index: 403
[2026-11-03T01:05:27.008] error: *** JOB 40552217 ON n1659 CANCELLED AT 2026-11-03T01:05:27 DUE TO TIME LIMIT ***
âš Job 40552212 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552212/40552212_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552212/40552212_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:33,276) - Starting with JobEnvironment(job_id=40552212, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:33,277) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552212/40552212_submitted.pkl
Trial-Index: 402
[2026-11-03T01:05:27.008] error: *** JOB 40552212 ON n1659 CANCELLED AT 2026-11-03T01:05:27 DUE TO TIME LIMIT ***
âš Job 40552225 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552225/40552225_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552225/40552225_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:46,587) - Starting with JobEnvironment(job_id=40552225, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:46,588) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552225/40552225_submitted.pkl
Trial-Index: 445
[2026-11-03T01:05:58.006] error: *** JOB 40552225 ON n1659 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552218 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552218/40552218_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552218/40552218_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:33,267) - Starting with JobEnvironment(job_id=40552218, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:33,268) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552218/40552218_submitted.pkl
Trial-Index: 437
[2026-11-03T01:05:27.008] error: *** JOB 40552218 ON n1659 CANCELLED AT 2026-11-03T01:05:27 DUE TO TIME LIMIT ***
âš Job 40552222 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552222/40552222_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552222/40552222_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:46,587) - Starting with JobEnvironment(job_id=40552222, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:46,588) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552222/40552222_submitted.pkl
Trial-Index: 409
[2026-11-03T01:05:58.006] error: *** JOB 40552222 ON n1659 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552215 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552215/40552215_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552215/40552215_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:33,240) - Starting with JobEnvironment(job_id=40552215, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:33,241) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552215/40552215_submitted.pkl
Trial-Index: 447
[2026-11-03T01:05:27.008] error: *** JOB 40552215 ON n1659 CANCELLED AT 2026-11-03T01:05:27 DUE TO TIME LIMIT ***
âš Job 40552227 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552227/40552227_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552227/40552227_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:41,429) - Starting with JobEnvironment(job_id=40552227, hostname=n1588, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:41,429) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552227/40552227_submitted.pkl
Trial-Index: 436
[2026-11-03T01:05:58.008] error: *** JOB 40552227 ON n1588 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552226 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552226/40552226_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552226/40552226_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:41,430) - Starting with JobEnvironment(job_id=40552226, hostname=n1588, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:41,431) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552226/40552226_submitted.pkl
Trial-Index: 431
[2026-11-03T01:05:58.009] error: *** JOB 40552226 ON n1588 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552228 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552228/40552228_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552228/40552228_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:41,431) - Starting with JobEnvironment(job_id=40552228, hostname=n1588, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:41,431) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552228/40552228_submitted.pkl
Trial-Index: 444
[2026-11-03T01:05:58.008] error: *** JOB 40552228 ON n1588 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552236 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552236/40552236_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552236/40552236_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:45,001) - Starting with JobEnvironment(job_id=40552236, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:45,002) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552236/40552236_submitted.pkl
Trial-Index: 408
[2026-11-03T01:05:58.008] error: *** JOB 40552236 ON n1713 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552213 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552213/40552213_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552213/40552213_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:33,273) - Starting with JobEnvironment(job_id=40552213, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:33,274) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552213/40552213_submitted.pkl
Trial-Index: 404
[2026-11-03T01:05:27.008] error: *** JOB 40552213 ON n1659 CANCELLED AT 2026-11-03T01:05:27 DUE TO TIME LIMIT ***
âš Job 40552211 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552211/40552211_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552211/40552211_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:33,251) - Starting with JobEnvironment(job_id=40552211, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:33,253) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552211/40552211_submitted.pkl
Trial-Index: 443
[2026-11-03T01:05:27.008] error: *** JOB 40552211 ON n1659 CANCELLED AT 2026-11-03T01:05:27 DUE TO TIME LIMIT ***
âš Job 40552241 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552241/40552241_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552241/40552241_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:45,729) - Starting with JobEnvironment(job_id=40552241, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:45,730) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552241/40552241_submitted.pkl
Trial-Index: 411
[2026-11-03T01:05:58.009] error: *** JOB 40552241 ON n1711 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552240 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552240/40552240_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552240/40552240_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:45,729) - Starting with JobEnvironment(job_id=40552240, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:45,730) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552240/40552240_submitted.pkl
Trial-Index: 427
[2026-11-03T01:05:58.010] error: *** JOB 40552240 ON n1711 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552238 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552238/40552238_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552238/40552238_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:45,001) - Starting with JobEnvironment(job_id=40552238, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:45,002) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552238/40552238_submitted.pkl
Trial-Index: 406
[2026-11-03T01:05:58.009] error: *** JOB 40552238 ON n1713 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552235 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552235/40552235_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552235/40552235_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:45,001) - Starting with JobEnvironment(job_id=40552235, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:45,002) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552235/40552235_submitted.pkl
Trial-Index: 401
[2026-11-03T01:05:58.008] error: *** JOB 40552235 ON n1713 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552216 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552216/40552216_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552216/40552216_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:33,262) - Starting with JobEnvironment(job_id=40552216, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:33,270) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552216/40552216_submitted.pkl
Trial-Index: 416
[2026-11-03T01:05:27.008] error: *** JOB 40552216 ON n1659 CANCELLED AT 2026-11-03T01:05:27 DUE TO TIME LIMIT ***
âš Job 40552233 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552233/40552233_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552233/40552233_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:52,303) - Starting with JobEnvironment(job_id=40552233, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:52,304) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552233/40552233_submitted.pkl
Trial-Index: 415
[2026-11-03T01:05:58.008] error: *** JOB 40552233 ON n1714 CANCELLED AT 2026-11-03T01:05:58 DUE TO TIME LIMIT ***
âš Job 40552214 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552214/40552214_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552214/40552214_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:05:33,247) - Starting with JobEnvironment(job_id=40552214, hostname=n1659, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:05:33,248) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552214/40552214_submitted.pkl
Trial-Index: 439
[2026-11-03T01:05:27.008] error: *** JOB 40552214 ON n1659 CANCELLED AT 2026-11-03T01:05:27 DUE TO TIME LIMIT ***
âš Job 40552259 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552259/40552259_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552259/40552259_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:10,108) - Starting with JobEnvironment(job_id=40552259, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:10,110) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552259/40552259_submitted.pkl
Trial-Index: 424
[2026-11-03T01:06:28.008] error: *** JOB 40552259 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552250 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552250/40552250_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552250/40552250_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:07,699) - Starting with JobEnvironment(job_id=40552250, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:07,700) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552250/40552250_submitted.pkl
Trial-Index: 420
[2026-11-03T01:06:28.007] error: *** JOB 40552250 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552248 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552248/40552248_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552248/40552248_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:07,699) - Starting with JobEnvironment(job_id=40552248, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:07,700) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552248/40552248_submitted.pkl
Trial-Index: 410
[2026-11-03T01:06:28.007] error: *** JOB 40552248 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552251 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552251/40552251_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552251/40552251_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:07,699) - Starting with JobEnvironment(job_id=40552251, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:07,700) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552251/40552251_submitted.pkl
Trial-Index: 423
[2026-11-03T01:06:28.007] error: *** JOB 40552251 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552245 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552245/40552245_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552245/40552245_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:07,699) - Starting with JobEnvironment(job_id=40552245, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:07,700) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552245/40552245_submitted.pkl
Trial-Index: 400
[2026-11-03T01:06:28.007] error: *** JOB 40552245 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552247 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552247/40552247_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552247/40552247_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:07,699) - Starting with JobEnvironment(job_id=40552247, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:07,700) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552247/40552247_submitted.pkl
Trial-Index: 407
[2026-11-03T01:06:28.007] error: *** JOB 40552247 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552246 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552246/40552246_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552246/40552246_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:07,699) - Starting with JobEnvironment(job_id=40552246, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:07,700) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552246/40552246_submitted.pkl
Trial-Index: 425
[2026-11-03T01:06:29.033] error: *** JOB 40552246 ON n1711 CANCELLED AT 2026-11-03T01:06:29 DUE TO TIME LIMIT ***
âš Job 40552255 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552255/40552255_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552255/40552255_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:10,108) - Starting with JobEnvironment(job_id=40552255, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:10,110) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552255/40552255_submitted.pkl
Trial-Index: 419
[2026-11-03T01:06:28.008] error: *** JOB 40552255 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552242 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552242/40552242_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552242/40552242_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:03,822) - Starting with JobEnvironment(job_id=40552242, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:03,823) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552242/40552242_submitted.pkl
Trial-Index: 448
[2026-11-03T01:06:28.006] error: *** JOB 40552242 ON n1714 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552253 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552253/40552253_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552253/40552253_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:10,108) - Starting with JobEnvironment(job_id=40552253, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:10,109) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552253/40552253_submitted.pkl
Trial-Index: 405
[2026-11-03T01:06:28.008] error: *** JOB 40552253 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552258 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552258/40552258_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552258/40552258_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:10,108) - Starting with JobEnvironment(job_id=40552258, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:10,110) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552258/40552258_submitted.pkl
Trial-Index: 422
[2026-11-03T01:06:28.008] error: *** JOB 40552258 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552256 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552256/40552256_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552256/40552256_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:10,108) - Starting with JobEnvironment(job_id=40552256, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:10,110) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552256/40552256_submitted.pkl
Trial-Index: 428
[2026-11-03T01:06:28.008] error: *** JOB 40552256 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552252 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552252/40552252_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552252/40552252_0_log.out
----------------------------------------
submitit INFO (2026-11-03 00:06:10,108) - Starting with JobEnvironment(job_id=40552252, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 00:06:10,109) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552252/40552252_submitted.pkl
Trial-Index: 418
[2026-11-03T01:06:28.008] error: *** JOB 40552252 ON n1711 CANCELLED AT 2026-11-03T01:06:28 DUE TO TIME LIMIT ***
âš Job 40552985 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552985/40552985_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552985/40552985_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:02,203) - Starting with JobEnvironment(job_id=40552985, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:02,204) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552985/40552985_submitted.pkl
Trial-Index: 450
[2026-11-03T02:27:57.043] error: *** JOB 40552985 ON n1705 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552976 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552976/40552976_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552976/40552976_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:27:41,984) - Starting with JobEnvironment(job_id=40552976, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:27:41,984) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552976/40552976_submitted.pkl
Trial-Index: 452
[2026-11-03T02:27:57.041] error: *** JOB 40552976 ON n1714 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552987 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552987/40552987_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552987/40552987_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:02,203) - Starting with JobEnvironment(job_id=40552987, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:02,204) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552987/40552987_submitted.pkl
Trial-Index: 473
[2026-11-03T02:27:57.043] error: *** JOB 40552987 ON n1705 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552994 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552994/40552994_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552994/40552994_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:10,324) - Starting with JobEnvironment(job_id=40552994, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:10,325) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552994/40552994_submitted.pkl
Trial-Index: 489
[2026-11-03T02:28:27.005] error: *** JOB 40552994 ON n1705 CANCELLED AT 2026-11-03T02:28:27 DUE TO TIME LIMIT ***
âš Job 40552977 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552977/40552977_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552977/40552977_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:27:41,984) - Starting with JobEnvironment(job_id=40552977, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:27:41,985) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552977/40552977_submitted.pkl
Trial-Index: 498
[2026-11-03T02:27:57.042] error: *** JOB 40552977 ON n1714 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552990 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552990/40552990_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552990/40552990_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:02,203) - Starting with JobEnvironment(job_id=40552990, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:02,204) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552990/40552990_submitted.pkl
Trial-Index: 499
[2026-11-03T02:27:57.043] error: *** JOB 40552990 ON n1705 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552982 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552982/40552982_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552982/40552982_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:27:41,706) - Starting with JobEnvironment(job_id=40552982, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:27:41,707) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552982/40552982_submitted.pkl
Trial-Index: 484
[2026-11-03T02:27:57.042] error: *** JOB 40552982 ON n1450 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552975 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552975/40552975_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552975/40552975_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:27:41,984) - Starting with JobEnvironment(job_id=40552975, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:27:41,985) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552975/40552975_submitted.pkl
Trial-Index: 464
[2026-11-03T02:27:57.041] error: *** JOB 40552975 ON n1714 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552980 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552980/40552980_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552980/40552980_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:27:41,706) - Starting with JobEnvironment(job_id=40552980, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:27:41,707) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552980/40552980_submitted.pkl
Trial-Index: 460
[2026-11-03T02:27:57.043] error: *** JOB 40552980 ON n1450 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552988 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552988/40552988_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552988/40552988_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:02,203) - Starting with JobEnvironment(job_id=40552988, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:02,204) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552988/40552988_submitted.pkl
Trial-Index: 490
[2026-11-03T02:27:57.043] error: *** JOB 40552988 ON n1705 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552969 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552969/40552969_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552969/40552969_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:27:41,984) - Starting with JobEnvironment(job_id=40552969, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:27:41,984) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552969/40552969_submitted.pkl
Trial-Index: 470
[2026-11-03T02:27:57.041] error: *** JOB 40552969 ON n1714 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552978 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552978/40552978_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552978/40552978_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:27:45,807) - Starting with JobEnvironment(job_id=40552978, hostname=n1692, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:27:45,807) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552978/40552978_submitted.pkl
Trial-Index: 461
[2026-11-03T02:27:57.043] error: *** JOB 40552978 ON n1692 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552999 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552999/40552999_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552999/40552999_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:09,234) - Starting with JobEnvironment(job_id=40552999, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:09,235) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552999/40552999_submitted.pkl
Trial-Index: 479
[2026-11-03T02:28:27.005] error: *** JOB 40552999 ON n1627 CANCELLED AT 2026-11-03T02:28:27 DUE TO TIME LIMIT ***
âš Job 40553003 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553003/40553003_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553003/40553003_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:12,453) - Starting with JobEnvironment(job_id=40553003, hostname=n1615, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:12,453) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553003/40553003_submitted.pkl
Trial-Index: 486
[2026-11-03T02:28:27.005] error: *** JOB 40553003 ON n1615 CANCELLED AT 2026-11-03T02:28:27 DUE TO TIME LIMIT ***
âš Job 40552995 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552995/40552995_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552995/40552995_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:09,234) - Starting with JobEnvironment(job_id=40552995, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:09,235) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552995/40552995_submitted.pkl
Trial-Index: 485
[2026-11-03T02:28:27.005] error: *** JOB 40552995 ON n1627 CANCELLED AT 2026-11-03T02:28:27 DUE TO TIME LIMIT ***
âš Job 40553005 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553005/40553005_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553005/40553005_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:23,904) - Starting with JobEnvironment(job_id=40553005, hostname=n1615, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:23,906) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553005/40553005_submitted.pkl
Trial-Index: 456
[2026-11-03T02:28:27.005] error: *** JOB 40553005 ON n1615 CANCELLED AT 2026-11-03T02:28:27 DUE TO TIME LIMIT ***
âš Job 40552993 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552993/40552993_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552993/40552993_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:02,203) - Starting with JobEnvironment(job_id=40552993, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:02,204) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552993/40552993_submitted.pkl
Trial-Index: 458
[2026-11-03T02:27:57.043] error: *** JOB 40552993 ON n1705 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552972 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552972/40552972_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552972/40552972_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:27:41,984) - Starting with JobEnvironment(job_id=40552972, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:27:41,985) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552972/40552972_submitted.pkl
Trial-Index: 465
[2026-11-03T02:27:57.041] error: *** JOB 40552972 ON n1714 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40552989 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552989/40552989_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552989/40552989_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:02,203) - Starting with JobEnvironment(job_id=40552989, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:02,204) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552989/40552989_submitted.pkl
Trial-Index: 468
[2026-11-03T02:27:57.043] error: *** JOB 40552989 ON n1705 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40553007 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553007/40553007_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553007/40553007_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:27,258) - Starting with JobEnvironment(job_id=40553007, hostname=n1605, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:27,261) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553007/40553007_submitted.pkl
Trial-Index: 453
[2026-11-03T02:28:27.006] error: *** JOB 40553007 ON n1605 CANCELLED AT 2026-11-03T02:28:27 DUE TO TIME LIMIT ***
âš Job 40553006 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553006/40553006_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553006/40553006_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:23,904) - Starting with JobEnvironment(job_id=40553006, hostname=n1615, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:23,905) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553006/40553006_submitted.pkl
Trial-Index: 462
[2026-11-03T02:28:27.006] error: *** JOB 40553006 ON n1615 CANCELLED AT 2026-11-03T02:28:27 DUE TO TIME LIMIT ***
âš Job 40553008 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553008/40553008_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553008/40553008_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:27,258) - Starting with JobEnvironment(job_id=40553008, hostname=n1605, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:27,259) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553008/40553008_submitted.pkl
Trial-Index: 455
[2026-11-03T02:28:27.007] error: *** JOB 40553008 ON n1605 CANCELLED AT 2026-11-03T02:28:27 DUE TO TIME LIMIT ***
âš Job 40553009 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553009/40553009_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553009/40553009_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:27,258) - Starting with JobEnvironment(job_id=40553009, hostname=n1605, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:27,259) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553009/40553009_submitted.pkl
Trial-Index: 475
[2026-11-03T02:28:27.007] error: *** JOB 40553009 ON n1605 CANCELLED AT 2026-11-03T02:28:27 DUE TO TIME LIMIT ***
âš Job 40552992 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552992/40552992_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552992/40552992_0_log.out
----------------------------------------
submitit INFO (2026-11-03 01:28:02,203) - Starting with JobEnvironment(job_id=40552992, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 01:28:02,204) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40552992/40552992_submitted.pkl
Trial-Index: 474
[2026-11-03T02:27:57.044] error: *** JOB 40552992 ON n1705 CANCELLED AT 2026-11-03T02:27:57 DUE TO TIME LIMIT ***
âš Job 40553688 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553688/40553688_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553688/40553688_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,222) - Starting with JobEnvironment(job_id=40553688, hostname=n1165, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,222) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553688/40553688_submitted.pkl
Trial-Index: 501
[2026-11-03T03:43:27.017] error: *** JOB 40553688 ON n1165 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553674 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553674/40553674_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553674/40553674_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:15,733) - Starting with JobEnvironment(job_id=40553674, hostname=n1261, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:15,735) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553674/40553674_submitted.pkl
Trial-Index: 547
[2026-11-03T03:43:27.015] error: *** JOB 40553674 ON n1261 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553700 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553700/40553700_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553700/40553700_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,309) - Starting with JobEnvironment(job_id=40553700, hostname=n1114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,310) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553700/40553700_submitted.pkl
Trial-Index: 512
[2026-11-03T03:43:27.017] error: *** JOB 40553700 ON n1114 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553658 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553658/40553658_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553658/40553658_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:11,667) - Starting with JobEnvironment(job_id=40553658, hostname=n1297, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:11,668) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553658/40553658_submitted.pkl
Trial-Index: 510
[2026-11-03T03:43:27.010] error: *** JOB 40553658 ON n1297 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553668 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553668/40553668_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553668/40553668_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:11,666) - Starting with JobEnvironment(job_id=40553668, hostname=n1297, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:11,667) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553668/40553668_submitted.pkl
Trial-Index: 537
[2026-11-03T03:43:27.011] error: *** JOB 40553668 ON n1297 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553714 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553714/40553714_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553714/40553714_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:34,354) - Starting with JobEnvironment(job_id=40553714, hostname=n1114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:34,355) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553714/40553714_submitted.pkl
Trial-Index: 519
[2026-11-03T03:43:27.018] error: *** JOB 40553714 ON n1114 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553703 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553703/40553703_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553703/40553703_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,311) - Starting with JobEnvironment(job_id=40553703, hostname=n1114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,314) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553703/40553703_submitted.pkl
Trial-Index: 515
[2026-11-03T03:43:27.018] error: *** JOB 40553703 ON n1114 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553705 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553705/40553705_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553705/40553705_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:34,354) - Starting with JobEnvironment(job_id=40553705, hostname=n1114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:34,355) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553705/40553705_submitted.pkl
Trial-Index: 521
[2026-11-03T03:43:27.018] error: *** JOB 40553705 ON n1114 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553657 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553657/40553657_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553657/40553657_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,249) - Starting with JobEnvironment(job_id=40553657, hostname=n1538, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,249) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553657/40553657_submitted.pkl
Trial-Index: 536
[2026-11-03T03:43:27.011] error: *** JOB 40553657 ON n1538 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553664 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553664/40553664_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553664/40553664_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:11,671) - Starting with JobEnvironment(job_id=40553664, hostname=n1297, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:11,672) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553664/40553664_submitted.pkl
Trial-Index: 541
[2026-11-03T03:43:27.011] error: *** JOB 40553664 ON n1297 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553676 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553676/40553676_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553676/40553676_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:15,727) - Starting with JobEnvironment(job_id=40553676, hostname=n1261, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:15,729) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553676/40553676_submitted.pkl
Trial-Index: 534
[2026-11-03T03:43:27.015] error: *** JOB 40553676 ON n1261 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553707 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553707/40553707_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553707/40553707_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:34,354) - Starting with JobEnvironment(job_id=40553707, hostname=n1114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:34,355) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553707/40553707_submitted.pkl
Trial-Index: 523
[2026-11-03T03:43:27.018] error: *** JOB 40553707 ON n1114 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553662 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553662/40553662_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553662/40553662_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:11,673) - Starting with JobEnvironment(job_id=40553662, hostname=n1297, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:11,673) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553662/40553662_submitted.pkl
Trial-Index: 505
[2026-11-03T03:43:27.010] error: *** JOB 40553662 ON n1297 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553656 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553656/40553656_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553656/40553656_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,249) - Starting with JobEnvironment(job_id=40553656, hostname=n1538, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,250) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553656/40553656_submitted.pkl
Trial-Index: 514
[2026-11-03T03:43:28.041] error: *** JOB 40553656 ON n1538 CANCELLED AT 2026-11-03T03:43:28 DUE TO TIME LIMIT ***
âš Job 40553672 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553672/40553672_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553672/40553672_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:15,748) - Starting with JobEnvironment(job_id=40553672, hostname=n1261, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:15,749) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553672/40553672_submitted.pkl
Trial-Index: 539
[2026-11-03T03:43:27.014] error: *** JOB 40553672 ON n1261 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553687 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553687/40553687_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553687/40553687_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,224) - Starting with JobEnvironment(job_id=40553687, hostname=n1165, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,224) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553687/40553687_submitted.pkl
Trial-Index: 527
[2026-11-03T03:43:27.017] error: *** JOB 40553687 ON n1165 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553670 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553670/40553670_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553670/40553670_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:15,743) - Starting with JobEnvironment(job_id=40553670, hostname=n1261, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:15,745) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553670/40553670_submitted.pkl
Trial-Index: 509
[2026-11-03T03:43:27.015] error: *** JOB 40553670 ON n1261 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553683 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553683/40553683_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553683/40553683_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:15,752) - Starting with JobEnvironment(job_id=40553683, hostname=n1261, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:15,753) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553683/40553683_submitted.pkl
Trial-Index: 502
[2026-11-03T03:43:27.015] error: *** JOB 40553683 ON n1261 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553692 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553692/40553692_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553692/40553692_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,225) - Starting with JobEnvironment(job_id=40553692, hostname=n1165, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,226) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553692/40553692_submitted.pkl
Trial-Index: 550
[2026-11-03T03:43:27.017] error: *** JOB 40553692 ON n1165 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553702 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553702/40553702_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553702/40553702_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,314) - Starting with JobEnvironment(job_id=40553702, hostname=n1114, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,315) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553702/40553702_submitted.pkl
Trial-Index: 528
[2026-11-03T03:43:27.018] error: *** JOB 40553702 ON n1114 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553690 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553690/40553690_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553690/40553690_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,217) - Starting with JobEnvironment(job_id=40553690, hostname=n1165, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,218) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553690/40553690_submitted.pkl
Trial-Index: 544
[2026-11-03T03:43:27.017] error: *** JOB 40553690 ON n1165 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553655 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553655/40553655_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553655/40553655_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:12,248) - Starting with JobEnvironment(job_id=40553655, hostname=n1538, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:12,249) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553655/40553655_submitted.pkl
Trial-Index: 503
[2026-11-03T03:43:27.011] error: *** JOB 40553655 ON n1538 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553675 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553675/40553675_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553675/40553675_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:15,750) - Starting with JobEnvironment(job_id=40553675, hostname=n1261, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:15,752) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553675/40553675_submitted.pkl
Trial-Index: 506
[2026-11-03T03:43:28.041] error: *** JOB 40553675 ON n1261 CANCELLED AT 2026-11-03T03:43:28 DUE TO TIME LIMIT ***
âš Job 40553659 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553659/40553659_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553659/40553659_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:11,664) - Starting with JobEnvironment(job_id=40553659, hostname=n1297, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:11,665) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553659/40553659_submitted.pkl
Trial-Index: 518
[2026-11-03T03:43:27.011] error: *** JOB 40553659 ON n1297 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553660 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553660/40553660_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553660/40553660_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:11,669) - Starting with JobEnvironment(job_id=40553660, hostname=n1297, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:11,670) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553660/40553660_submitted.pkl
Trial-Index: 533
[2026-11-03T03:43:27.011] error: *** JOB 40553660 ON n1297 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40553665 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553665/40553665_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553665/40553665_0_log.out
----------------------------------------
submitit INFO (2026-11-03 02:43:11,671) - Starting with JobEnvironment(job_id=40553665, hostname=n1297, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 02:43:11,671) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40553665/40553665_submitted.pkl
Trial-Index: 508
[2026-11-03T03:43:27.011] error: *** JOB 40553665 ON n1297 CANCELLED AT 2026-11-03T03:43:27 DUE TO TIME LIMIT ***
âš Job 40554364 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554364/40554364_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554364/40554364_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:31,287) - Starting with JobEnvironment(job_id=40554364, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:31,287) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554364/40554364_submitted.pkl
Trial-Index: 591
[2026-11-03T05:02:28.020] error: *** JOB 40554364 ON n1713 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554351 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554351/40554351_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554351/40554351_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:17,404) - Starting with JobEnvironment(job_id=40554351, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:17,404) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554351/40554351_submitted.pkl
Trial-Index: 552
[2026-11-03T05:02:28.018] error: *** JOB 40554351 ON n1714 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554392 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554392/40554392_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554392/40554392_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:03:11,534) - Starting with JobEnvironment(job_id=40554392, hostname=n1701, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:03:11,534) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554392/40554392_submitted.pkl
Trial-Index: 572
[2026-11-03T05:03:28.004] error: *** JOB 40554392 ON n1701 CANCELLED AT 2026-11-03T05:03:28 DUE TO TIME LIMIT ***
âš Job 40554345 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554345/40554345_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554345/40554345_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:17,404) - Starting with JobEnvironment(job_id=40554345, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:17,404) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554345/40554345_submitted.pkl
Trial-Index: 554
[2026-11-03T05:02:28.019] error: *** JOB 40554345 ON n1714 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554369 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554369/40554369_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554369/40554369_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:31,286) - Starting with JobEnvironment(job_id=40554369, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:31,290) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554369/40554369_submitted.pkl
Trial-Index: 582
[2026-11-03T05:02:28.020] error: *** JOB 40554369 ON n1713 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554342 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554342/40554342_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554342/40554342_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:12,260) - Starting with JobEnvironment(job_id=40554342, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:12,260) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554342/40554342_submitted.pkl
Trial-Index: 587
[2026-11-03T05:02:28.019] error: *** JOB 40554342 ON n1714 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554366 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554366/40554366_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554366/40554366_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:31,286) - Starting with JobEnvironment(job_id=40554366, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:31,287) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554366/40554366_submitted.pkl
Trial-Index: 585
[2026-11-03T05:02:28.020] error: *** JOB 40554366 ON n1713 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554377 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554377/40554377_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554377/40554377_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:45,247) - Starting with JobEnvironment(job_id=40554377, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:45,248) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554377/40554377_submitted.pkl
Trial-Index: 574
[2026-11-03T05:02:58.007] error: *** JOB 40554377 ON n1705 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554343 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554343/40554343_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554343/40554343_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:12,260) - Starting with JobEnvironment(job_id=40554343, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:12,260) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554343/40554343_submitted.pkl
Trial-Index: 557
[2026-11-03T05:02:28.019] error: *** JOB 40554343 ON n1714 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554391 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554391/40554391_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554391/40554391_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:51,236) - Starting with JobEnvironment(job_id=40554391, hostname=n1701, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:51,237) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554391/40554391_submitted.pkl
Trial-Index: 594
[2026-11-03T05:02:58.009] error: *** JOB 40554391 ON n1701 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554383 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554383/40554383_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554383/40554383_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:52,429) - Starting with JobEnvironment(job_id=40554383, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:52,430) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554383/40554383_submitted.pkl
Trial-Index: 568
[2026-11-03T05:02:58.008] error: *** JOB 40554383 ON n1702 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554394 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554394/40554394_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554394/40554394_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:03:11,439) - Starting with JobEnvironment(job_id=40554394, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:03:11,440) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554394/40554394_submitted.pkl
Trial-Index: 578
[2026-11-03T05:03:28.005] error: *** JOB 40554394 ON n1700 CANCELLED AT 2026-11-03T05:03:28 DUE TO TIME LIMIT ***
âš Job 40554393 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554393/40554393_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554393/40554393_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:03:11,533) - Starting with JobEnvironment(job_id=40554393, hostname=n1701, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:03:11,534) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554393/40554393_submitted.pkl
Trial-Index: 593
[2026-11-03T05:03:28.004] error: *** JOB 40554393 ON n1701 CANCELLED AT 2026-11-03T05:03:28 DUE TO TIME LIMIT ***
âš Job 40554370 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554370/40554370_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554370/40554370_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:31,286) - Starting with JobEnvironment(job_id=40554370, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:31,287) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554370/40554370_submitted.pkl
Trial-Index: 577
[2026-11-03T05:02:28.020] error: *** JOB 40554370 ON n1713 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554386 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554386/40554386_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554386/40554386_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:52,429) - Starting with JobEnvironment(job_id=40554386, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:52,431) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554386/40554386_submitted.pkl
Trial-Index: 573
[2026-11-03T05:02:58.009] error: *** JOB 40554386 ON n1702 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554371 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554371/40554371_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554371/40554371_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:44,046) - Starting with JobEnvironment(job_id=40554371, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:44,047) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554371/40554371_submitted.pkl
Trial-Index: 553
[2026-11-03T05:02:58.006] error: *** JOB 40554371 ON n1711 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554376 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554376/40554376_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554376/40554376_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:45,247) - Starting with JobEnvironment(job_id=40554376, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:45,248) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554376/40554376_submitted.pkl
Trial-Index: 584
[2026-11-03T05:02:58.007] error: *** JOB 40554376 ON n1705 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554387 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554387/40554387_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554387/40554387_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:52,429) - Starting with JobEnvironment(job_id=40554387, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:52,431) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554387/40554387_submitted.pkl
Trial-Index: 563
[2026-11-03T05:02:58.008] error: *** JOB 40554387 ON n1702 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554357 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554357/40554357_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554357/40554357_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:31,198) - Starting with JobEnvironment(job_id=40554357, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:31,198) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554357/40554357_submitted.pkl
Trial-Index: 567
[2026-11-03T05:02:28.019] error: *** JOB 40554357 ON n1714 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554385 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554385/40554385_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554385/40554385_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:52,429) - Starting with JobEnvironment(job_id=40554385, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:52,431) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554385/40554385_submitted.pkl
Trial-Index: 596
[2026-11-03T05:02:58.008] error: *** JOB 40554385 ON n1702 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554353 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554353/40554353_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554353/40554353_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:17,404) - Starting with JobEnvironment(job_id=40554353, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:17,404) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554353/40554353_submitted.pkl
Trial-Index: 551
[2026-11-03T05:02:28.019] error: *** JOB 40554353 ON n1714 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554399 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554399/40554399_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554399/40554399_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:03:11,508) - Starting with JobEnvironment(job_id=40554399, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:03:11,510) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554399/40554399_submitted.pkl
Trial-Index: 583
[2026-11-03T05:03:28.005] error: *** JOB 40554399 ON n1700 CANCELLED AT 2026-11-03T05:03:28 DUE TO TIME LIMIT ***
âš Job 40554381 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554381/40554381_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554381/40554381_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:49,329) - Starting with JobEnvironment(job_id=40554381, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:49,333) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554381/40554381_submitted.pkl
Trial-Index: 599
[2026-11-03T05:02:58.007] error: *** JOB 40554381 ON n1714 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554384 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554384/40554384_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554384/40554384_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:52,429) - Starting with JobEnvironment(job_id=40554384, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:52,430) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554384/40554384_submitted.pkl
Trial-Index: 555
[2026-11-03T05:02:58.008] error: *** JOB 40554384 ON n1702 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554356 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554356/40554356_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554356/40554356_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:31,198) - Starting with JobEnvironment(job_id=40554356, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:31,198) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554356/40554356_submitted.pkl
Trial-Index: 556
[2026-11-03T05:02:28.019] error: *** JOB 40554356 ON n1714 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554374 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554374/40554374_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554374/40554374_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:45,247) - Starting with JobEnvironment(job_id=40554374, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:45,248) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554374/40554374_submitted.pkl
Trial-Index: 575
[2026-11-03T05:02:58.007] error: *** JOB 40554374 ON n1705 CANCELLED AT 2026-11-03T05:02:58 DUE TO TIME LIMIT ***
âš Job 40554365 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554365/40554365_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554365/40554365_0_log.out
----------------------------------------
submitit INFO (2026-11-03 04:02:31,286) - Starting with JobEnvironment(job_id=40554365, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 04:02:31,287) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554365/40554365_submitted.pkl
Trial-Index: 562
[2026-11-03T05:02:28.020] error: *** JOB 40554365 ON n1713 CANCELLED AT 2026-11-03T05:02:28 DUE TO TIME LIMIT ***
âš Job 40554967 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554967/40554967_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554967/40554967_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:44,384) - Starting with JobEnvironment(job_id=40554967, hostname=n1678, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:44,385) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554967/40554967_submitted.pkl
Trial-Index: 615
[2026-11-03T06:29:58.005] error: *** JOB 40554967 ON n1678 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40555014 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555014/40555014_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555014/40555014_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:16,860) - Starting with JobEnvironment(job_id=40555014, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:16,861) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555014/40555014_submitted.pkl
Trial-Index: 619
[2026-11-03T06:30:28.007] error: *** JOB 40555014 ON n1700 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40554998 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554998/40554998_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554998/40554998_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40554998, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,200) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554998/40554998_submitted.pkl
Trial-Index: 613
[2026-11-03T06:30:28.005] error: *** JOB 40554998 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40555004 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555004/40555004_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555004/40555004_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40555004, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,201) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555004/40555004_submitted.pkl
Trial-Index: 643
[2026-11-03T06:30:28.005] error: *** JOB 40555004 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40555009 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555009/40555009_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555009/40555009_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40555009, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,201) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555009/40555009_submitted.pkl
Trial-Index: 634
[2026-11-03T06:30:28.006] error: *** JOB 40555009 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40554974 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554974/40554974_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554974/40554974_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:37,697) - Starting with JobEnvironment(job_id=40554974, hostname=n1677, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:37,698) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554974/40554974_submitted.pkl
Trial-Index: 631
[2026-11-03T06:29:58.005] error: *** JOB 40554974 ON n1677 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40554971 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554971/40554971_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554971/40554971_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:44,379) - Starting with JobEnvironment(job_id=40554971, hostname=n1678, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:44,380) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554971/40554971_submitted.pkl
Trial-Index: 607
[2026-11-03T06:29:58.005] error: *** JOB 40554971 ON n1678 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40554991 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554991/40554991_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554991/40554991_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:01,393) - Starting with JobEnvironment(job_id=40554991, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:01,394) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554991/40554991_submitted.pkl
Trial-Index: 622
[2026-11-03T06:29:58.006] error: *** JOB 40554991 ON n1627 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40555012 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555012/40555012_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555012/40555012_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:16,860) - Starting with JobEnvironment(job_id=40555012, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:16,861) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555012/40555012_submitted.pkl
Trial-Index: 645
[2026-11-03T06:30:28.007] error: *** JOB 40555012 ON n1700 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40554981 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554981/40554981_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554981/40554981_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:52,510) - Starting with JobEnvironment(job_id=40554981, hostname=n1666, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:52,511) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554981/40554981_submitted.pkl
Trial-Index: 610
[2026-11-03T06:29:58.005] error: *** JOB 40554981 ON n1666 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40555006 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555006/40555006_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555006/40555006_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40555006, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,200) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555006/40555006_submitted.pkl
Trial-Index: 601
[2026-11-03T06:30:28.005] error: *** JOB 40555006 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40555003 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555003/40555003_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555003/40555003_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40555003, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,204) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555003/40555003_submitted.pkl
Trial-Index: 632
[2026-11-03T06:30:28.006] error: *** JOB 40555003 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40555010 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555010/40555010_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555010/40555010_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40555010, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,201) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555010/40555010_submitted.pkl
Trial-Index: 630
[2026-11-03T06:30:28.005] error: *** JOB 40555010 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40554970 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554970/40554970_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554970/40554970_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:44,384) - Starting with JobEnvironment(job_id=40554970, hostname=n1678, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:44,384) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554970/40554970_submitted.pkl
Trial-Index: 649
[2026-11-03T06:29:58.005] error: *** JOB 40554970 ON n1678 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40554968 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554968/40554968_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554968/40554968_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:44,383) - Starting with JobEnvironment(job_id=40554968, hostname=n1678, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:44,383) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554968/40554968_submitted.pkl
Trial-Index: 602
[2026-11-03T06:29:58.005] error: *** JOB 40554968 ON n1678 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40555002 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555002/40555002_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555002/40555002_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40555002, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,202) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555002/40555002_submitted.pkl
Trial-Index: 642
[2026-11-03T06:30:28.005] error: *** JOB 40555002 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40554972 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554972/40554972_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554972/40554972_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:37,697) - Starting with JobEnvironment(job_id=40554972, hostname=n1677, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:37,698) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554972/40554972_submitted.pkl
Trial-Index: 605
[2026-11-03T06:29:58.005] error: *** JOB 40554972 ON n1677 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40555001 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555001/40555001_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555001/40555001_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40555001, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,201) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555001/40555001_submitted.pkl
Trial-Index: 609
[2026-11-03T06:30:28.005] error: *** JOB 40555001 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40554965 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554965/40554965_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554965/40554965_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:44,382) - Starting with JobEnvironment(job_id=40554965, hostname=n1678, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:44,383) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554965/40554965_submitted.pkl
Trial-Index: 640
[2026-11-03T06:29:58.005] error: *** JOB 40554965 ON n1678 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40555005 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555005/40555005_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555005/40555005_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40555005, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,201) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555005/40555005_submitted.pkl
Trial-Index: 603
[2026-11-03T06:30:28.006] error: *** JOB 40555005 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40554969 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554969/40554969_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554969/40554969_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:44,384) - Starting with JobEnvironment(job_id=40554969, hostname=n1678, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:44,385) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554969/40554969_submitted.pkl
Trial-Index: 604
[2026-11-03T06:29:58.005] error: *** JOB 40554969 ON n1678 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40554978 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554978/40554978_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554978/40554978_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:49,250) - Starting with JobEnvironment(job_id=40554978, hostname=n1677, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:49,251) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554978/40554978_submitted.pkl
Trial-Index: 606
[2026-11-03T06:29:58.005] error: *** JOB 40554978 ON n1677 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40554997 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554997/40554997_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554997/40554997_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40554997, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,201) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554997/40554997_submitted.pkl
Trial-Index: 611
[2026-11-03T06:30:29.037] error: *** JOB 40554997 ON n1714 CANCELLED AT 2026-11-03T06:30:29 DUE TO TIME LIMIT ***
âš Job 40554999 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554999/40554999_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554999/40554999_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:20,200) - Starting with JobEnvironment(job_id=40554999, hostname=n1714, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:20,202) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554999/40554999_submitted.pkl
Trial-Index: 612
[2026-11-03T06:30:28.006] error: *** JOB 40554999 ON n1714 CANCELLED AT 2026-11-03T06:30:28 DUE TO TIME LIMIT ***
âš Job 40554990 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554990/40554990_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554990/40554990_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:30:01,393) - Starting with JobEnvironment(job_id=40554990, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:30:01,394) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554990/40554990_submitted.pkl
Trial-Index: 614
[2026-11-03T06:29:58.006] error: *** JOB 40554990 ON n1627 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40554985 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554985/40554985_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554985/40554985_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:52,511) - Starting with JobEnvironment(job_id=40554985, hostname=n1666, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:52,512) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554985/40554985_submitted.pkl
Trial-Index: 608
[2026-11-03T06:29:58.006] error: *** JOB 40554985 ON n1666 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40554983 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554983/40554983_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554983/40554983_0_log.out
----------------------------------------
submitit INFO (2026-11-03 05:29:52,512) - Starting with JobEnvironment(job_id=40554983, hostname=n1666, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 05:29:52,512) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40554983/40554983_submitted.pkl
Trial-Index: 616
[2026-11-03T06:29:58.005] error: *** JOB 40554983 ON n1666 CANCELLED AT 2026-11-03T06:29:58 DUE TO TIME LIMIT ***
âš Job 40555919 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555919/40555919_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555919/40555919_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:49,234) - Starting with JobEnvironment(job_id=40555919, hostname=n1572, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:49,235) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555919/40555919_submitted.pkl
Trial-Index: 669
[2026-11-03T07:49:58.061] error: *** JOB 40555919 ON n1572 CANCELLED AT 2026-11-03T07:49:58 DUE TO TIME LIMIT ***
âš Job 40555941 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555941/40555941_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555941/40555941_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,944) - Starting with JobEnvironment(job_id=40555941, hostname=n1649, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,945) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555941/40555941_submitted.pkl
Trial-Index: 686
[2026-11-03T07:50:27.005] error: *** JOB 40555941 ON n1649 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555956 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555956/40555956_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555956/40555956_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:04,835) - Starting with JobEnvironment(job_id=40555956, hostname=n1572, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:04,836) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555956/40555956_submitted.pkl
Trial-Index: 693
[2026-11-03T07:50:27.006] error: *** JOB 40555956 ON n1572 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555949 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555949/40555949_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555949/40555949_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,944) - Starting with JobEnvironment(job_id=40555949, hostname=n1649, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,945) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555949/40555949_submitted.pkl
Trial-Index: 688
[2026-11-03T07:50:27.005] error: *** JOB 40555949 ON n1649 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555924 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555924/40555924_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555924/40555924_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:49,242) - Starting with JobEnvironment(job_id=40555924, hostname=n1572, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:49,242) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555924/40555924_submitted.pkl
Trial-Index: 652
[2026-11-03T07:49:57.010] error: *** JOB 40555924 ON n1572 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555960 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555960/40555960_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555960/40555960_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,449) - Starting with JobEnvironment(job_id=40555960, hostname=n1496, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,450) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555960/40555960_submitted.pkl
Trial-Index: 681
[2026-11-03T07:50:27.006] error: *** JOB 40555960 ON n1496 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555942 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555942/40555942_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555942/40555942_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,944) - Starting with JobEnvironment(job_id=40555942, hostname=n1649, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,945) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555942/40555942_submitted.pkl
Trial-Index: 659
[2026-11-03T07:50:27.005] error: *** JOB 40555942 ON n1649 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555928 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555928/40555928_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555928/40555928_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:59,708) - Starting with JobEnvironment(job_id=40555928, hostname=n1672, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:59,709) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555928/40555928_submitted.pkl
Trial-Index: 663
[2026-11-03T07:49:57.010] error: *** JOB 40555928 ON n1672 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555921 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555921/40555921_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555921/40555921_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:49,231) - Starting with JobEnvironment(job_id=40555921, hostname=n1572, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:49,232) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555921/40555921_submitted.pkl
Trial-Index: 687
[2026-11-03T07:49:57.009] error: *** JOB 40555921 ON n1572 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555935 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555935/40555935_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555935/40555935_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:59,706) - Starting with JobEnvironment(job_id=40555935, hostname=n1672, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:59,707) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555935/40555935_submitted.pkl
Trial-Index: 683
[2026-11-03T07:49:57.010] error: *** JOB 40555935 ON n1672 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555939 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555939/40555939_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555939/40555939_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,944) - Starting with JobEnvironment(job_id=40555939, hostname=n1649, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,945) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555939/40555939_submitted.pkl
Trial-Index: 664
[2026-11-03T07:50:27.005] error: *** JOB 40555939 ON n1649 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555936 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555936/40555936_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555936/40555936_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:59,710) - Starting with JobEnvironment(job_id=40555936, hostname=n1672, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:59,710) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555936/40555936_submitted.pkl
Trial-Index: 660
[2026-11-03T07:49:57.010] error: *** JOB 40555936 ON n1672 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555937 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555937/40555937_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555937/40555937_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:59,714) - Starting with JobEnvironment(job_id=40555937, hostname=n1672, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:59,714) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555937/40555937_submitted.pkl
Trial-Index: 656
[2026-11-03T07:49:57.011] error: *** JOB 40555937 ON n1672 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555957 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555957/40555957_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555957/40555957_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,446) - Starting with JobEnvironment(job_id=40555957, hostname=n1496, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,447) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555957/40555957_submitted.pkl
Trial-Index: 695
[2026-11-03T07:50:27.006] error: *** JOB 40555957 ON n1496 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555961 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555961/40555961_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555961/40555961_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,443) - Starting with JobEnvironment(job_id=40555961, hostname=n1496, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,444) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555961/40555961_submitted.pkl
Trial-Index: 682
[2026-11-03T07:50:27.006] error: *** JOB 40555961 ON n1496 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555930 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555930/40555930_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555930/40555930_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:59,711) - Starting with JobEnvironment(job_id=40555930, hostname=n1672, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:59,712) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555930/40555930_submitted.pkl
Trial-Index: 654
[2026-11-03T07:49:57.010] error: *** JOB 40555930 ON n1672 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555920 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555920/40555920_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555920/40555920_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:49,245) - Starting with JobEnvironment(job_id=40555920, hostname=n1572, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:49,246) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555920/40555920_submitted.pkl
Trial-Index: 668
[2026-11-03T07:49:57.009] error: *** JOB 40555920 ON n1572 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555918 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555918/40555918_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555918/40555918_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:49,240) - Starting with JobEnvironment(job_id=40555918, hostname=n1572, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:49,240) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555918/40555918_submitted.pkl
Trial-Index: 666
[2026-11-03T07:49:57.010] error: *** JOB 40555918 ON n1572 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555948 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555948/40555948_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555948/40555948_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,944) - Starting with JobEnvironment(job_id=40555948, hostname=n1649, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,945) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555948/40555948_submitted.pkl
Trial-Index: 667
[2026-11-03T07:50:27.005] error: *** JOB 40555948 ON n1649 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555931 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555931/40555931_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555931/40555931_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:59,716) - Starting with JobEnvironment(job_id=40555931, hostname=n1672, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:59,718) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555931/40555931_submitted.pkl
Trial-Index: 655
[2026-11-03T07:49:57.010] error: *** JOB 40555931 ON n1672 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555965 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555965/40555965_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555965/40555965_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:28,577) - Starting with JobEnvironment(job_id=40555965, hostname=n1675, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:28,578) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555965/40555965_submitted.pkl
Trial-Index: 678
[2026-11-03T07:50:27.007] error: *** JOB 40555965 ON n1675 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555958 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555958/40555958_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555958/40555958_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,448) - Starting with JobEnvironment(job_id=40555958, hostname=n1496, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,449) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555958/40555958_submitted.pkl
Trial-Index: 690
[2026-11-03T07:50:27.006] error: *** JOB 40555958 ON n1496 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555929 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555929/40555929_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555929/40555929_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:59,715) - Starting with JobEnvironment(job_id=40555929, hostname=n1672, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:59,716) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555929/40555929_submitted.pkl
Trial-Index: 658
[2026-11-03T07:49:57.011] error: *** JOB 40555929 ON n1672 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555932 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555932/40555932_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555932/40555932_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:49:59,714) - Starting with JobEnvironment(job_id=40555932, hostname=n1672, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:49:59,715) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555932/40555932_submitted.pkl
Trial-Index: 700
[2026-11-03T07:49:57.011] error: *** JOB 40555932 ON n1672 CANCELLED AT 2026-11-03T07:49:57 DUE TO TIME LIMIT ***
âš Job 40555966 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555966/40555966_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555966/40555966_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:28,577) - Starting with JobEnvironment(job_id=40555966, hostname=n1675, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:28,578) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555966/40555966_submitted.pkl
Trial-Index: 680
[2026-11-03T07:50:28.010] error: *** JOB 40555966 ON n1675 CANCELLED AT 2026-11-03T07:50:28 DUE TO TIME LIMIT ***
âš Job 40555943 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555943/40555943_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555943/40555943_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,944) - Starting with JobEnvironment(job_id=40555943, hostname=n1649, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,945) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555943/40555943_submitted.pkl
Trial-Index: 657
[2026-11-03T07:50:27.005] error: *** JOB 40555943 ON n1649 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555954 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555954/40555954_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555954/40555954_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,944) - Starting with JobEnvironment(job_id=40555954, hostname=n1649, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,945) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555954/40555954_submitted.pkl
Trial-Index: 676
[2026-11-03T07:50:27.005] error: *** JOB 40555954 ON n1649 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40555945 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555945/40555945_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555945/40555945_0_log.out
----------------------------------------
submitit INFO (2026-11-03 06:50:07,944) - Starting with JobEnvironment(job_id=40555945, hostname=n1649, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 06:50:07,945) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40555945/40555945_submitted.pkl
Trial-Index: 653
[2026-11-03T07:50:27.005] error: *** JOB 40555945 ON n1649 CANCELLED AT 2026-11-03T07:50:27 DUE TO TIME LIMIT ***
âš Job 40557077 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557077/40557077_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557077/40557077_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:28,330) - Starting with JobEnvironment(job_id=40557077, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:28,331) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557077/40557077_submitted.pkl
Trial-Index: 704
[2026-11-03T09:21:29.065] error: *** JOB 40557077 ON n1705 CANCELLED AT 2026-11-03T09:21:29 DUE TO TIME LIMIT ***
âš Job 40557090 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557090/40557090_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557090/40557090_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:37,344) - Starting with JobEnvironment(job_id=40557090, hostname=n1362, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:37,345) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557090/40557090_submitted.pkl
Trial-Index: 721
[2026-11-03T09:21:59.005] error: *** JOB 40557090 ON n1362 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557071 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557071/40557071_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557071/40557071_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:28,330) - Starting with JobEnvironment(job_id=40557071, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:28,331) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557071/40557071_submitted.pkl
Trial-Index: 702
[2026-11-03T09:21:28.009] error: *** JOB 40557071 ON n1705 CANCELLED AT 2026-11-03T09:21:28 DUE TO TIME LIMIT ***
âš Job 40557070 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557070/40557070_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557070/40557070_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:14,496) - Starting with JobEnvironment(job_id=40557070, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:14,498) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557070/40557070_submitted.pkl
Trial-Index: 731
[2026-11-03T09:21:28.009] error: *** JOB 40557070 ON n1705 CANCELLED AT 2026-11-03T09:21:28 DUE TO TIME LIMIT ***
âš Job 40557087 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557087/40557087_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557087/40557087_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:37,344) - Starting with JobEnvironment(job_id=40557087, hostname=n1362, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:37,345) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557087/40557087_submitted.pkl
Trial-Index: 709
[2026-11-03T09:21:59.004] error: *** JOB 40557087 ON n1362 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557069 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557069/40557069_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557069/40557069_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:14,496) - Starting with JobEnvironment(job_id=40557069, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:14,497) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557069/40557069_submitted.pkl
Trial-Index: 713
[2026-11-03T09:21:28.010] error: *** JOB 40557069 ON n1705 CANCELLED AT 2026-11-03T09:21:28 DUE TO TIME LIMIT ***
âš Job 40557084 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557084/40557084_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557084/40557084_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:34,298) - Starting with JobEnvironment(job_id=40557084, hostname=n1373, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:34,300) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557084/40557084_submitted.pkl
Trial-Index: 711
[2026-11-03T09:21:59.006] error: *** JOB 40557084 ON n1373 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557089 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557089/40557089_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557089/40557089_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:37,340) - Starting with JobEnvironment(job_id=40557089, hostname=n1362, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:37,341) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557089/40557089_submitted.pkl
Trial-Index: 743
[2026-11-03T09:21:59.005] error: *** JOB 40557089 ON n1362 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557118 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557118/40557118_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557118/40557118_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:22:15,640) - Starting with JobEnvironment(job_id=40557118, hostname=n1574, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:22:15,641) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557118/40557118_submitted.pkl
Trial-Index: 738
[2026-11-03T09:22:29.033] error: *** JOB 40557118 ON n1574 CANCELLED AT 2026-11-03T09:22:29 DUE TO TIME LIMIT ***
âš Job 40557074 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557074/40557074_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557074/40557074_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:28,330) - Starting with JobEnvironment(job_id=40557074, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:28,331) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557074/40557074_submitted.pkl
Trial-Index: 735
[2026-11-03T09:21:28.009] error: *** JOB 40557074 ON n1705 CANCELLED AT 2026-11-03T09:21:28 DUE TO TIME LIMIT ***
âš Job 40557075 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557075/40557075_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557075/40557075_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:28,330) - Starting with JobEnvironment(job_id=40557075, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:28,331) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557075/40557075_submitted.pkl
Trial-Index: 730
[2026-11-03T09:21:28.012] error: *** JOB 40557075 ON n1705 CANCELLED AT 2026-11-03T09:21:28 DUE TO TIME LIMIT ***
âš Job 40557097 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557097/40557097_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557097/40557097_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:55,106) - Starting with JobEnvironment(job_id=40557097, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:55,107) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557097/40557097_submitted.pkl
Trial-Index: 722
[2026-11-03T09:21:59.005] error: *** JOB 40557097 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557086 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557086/40557086_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557086/40557086_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:37,343) - Starting with JobEnvironment(job_id=40557086, hostname=n1362, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:37,344) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557086/40557086_submitted.pkl
Trial-Index: 733
[2026-11-03T09:21:59.004] error: *** JOB 40557086 ON n1362 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557108 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557108/40557108_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557108/40557108_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:59,757) - Starting with JobEnvironment(job_id=40557108, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:59,757) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557108/40557108_submitted.pkl
Trial-Index: 729
[2026-11-03T09:21:59.006] error: *** JOB 40557108 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557088 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557088/40557088_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557088/40557088_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:37,345) - Starting with JobEnvironment(job_id=40557088, hostname=n1362, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:37,346) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557088/40557088_submitted.pkl
Trial-Index: 740
[2026-11-03T09:21:59.005] error: *** JOB 40557088 ON n1362 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557109 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557109/40557109_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557109/40557109_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:59,757) - Starting with JobEnvironment(job_id=40557109, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:59,757) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557109/40557109_submitted.pkl
Trial-Index: 706
[2026-11-03T09:21:59.006] error: *** JOB 40557109 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557099 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557099/40557099_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557099/40557099_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:55,106) - Starting with JobEnvironment(job_id=40557099, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:55,107) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557099/40557099_submitted.pkl
Trial-Index: 741
[2026-11-03T09:21:59.007] error: *** JOB 40557099 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557080 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557080/40557080_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557080/40557080_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:28,358) - Starting with JobEnvironment(job_id=40557080, hostname=n1615, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:28,359) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557080/40557080_submitted.pkl
Trial-Index: 746
[2026-11-03T09:21:28.010] error: *** JOB 40557080 ON n1615 CANCELLED AT 2026-11-03T09:21:28 DUE TO TIME LIMIT ***
âš Job 40557104 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557104/40557104_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557104/40557104_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:55,106) - Starting with JobEnvironment(job_id=40557104, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:55,107) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557104/40557104_submitted.pkl
Trial-Index: 744
[2026-11-03T09:21:59.007] error: *** JOB 40557104 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557076 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557076/40557076_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557076/40557076_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:28,330) - Starting with JobEnvironment(job_id=40557076, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:28,331) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557076/40557076_submitted.pkl
Trial-Index: 719
[2026-11-03T09:21:28.009] error: *** JOB 40557076 ON n1705 CANCELLED AT 2026-11-03T09:21:28 DUE TO TIME LIMIT ***
âš Job 40557102 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557102/40557102_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557102/40557102_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:55,107) - Starting with JobEnvironment(job_id=40557102, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:55,107) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557102/40557102_submitted.pkl
Trial-Index: 728
[2026-11-03T09:21:59.006] error: *** JOB 40557102 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557112 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557112/40557112_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557112/40557112_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:59,757) - Starting with JobEnvironment(job_id=40557112, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:59,757) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557112/40557112_submitted.pkl
Trial-Index: 752
[2026-11-03T09:21:59.007] error: *** JOB 40557112 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557073 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557073/40557073_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557073/40557073_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:28,330) - Starting with JobEnvironment(job_id=40557073, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:28,331) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557073/40557073_submitted.pkl
Trial-Index: 734
[2026-11-03T09:21:28.009] error: *** JOB 40557073 ON n1705 CANCELLED AT 2026-11-03T09:21:28 DUE TO TIME LIMIT ***
âš Job 40557111 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557111/40557111_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557111/40557111_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:59,757) - Starting with JobEnvironment(job_id=40557111, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:59,760) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557111/40557111_submitted.pkl
Trial-Index: 708
[2026-11-03T09:21:59.007] error: *** JOB 40557111 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557094 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557094/40557094_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557094/40557094_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:55,106) - Starting with JobEnvironment(job_id=40557094, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:55,110) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557094/40557094_submitted.pkl
Trial-Index: 707
[2026-11-03T09:21:59.005] error: *** JOB 40557094 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557096 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557096/40557096_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557096/40557096_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:55,106) - Starting with JobEnvironment(job_id=40557096, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:55,107) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557096/40557096_submitted.pkl
Trial-Index: 742
[2026-11-03T09:21:59.005] error: *** JOB 40557096 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557128 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557128/40557128_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557128/40557128_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:22:11,653) - Starting with JobEnvironment(job_id=40557128, hostname=n1362, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:22:11,654) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557128/40557128_submitted.pkl
Trial-Index: 747
[2026-11-03T09:22:29.033] error: *** JOB 40557128 ON n1362 CANCELLED AT 2026-11-03T09:22:29 DUE TO TIME LIMIT ***
âš Job 40557098 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557098/40557098_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557098/40557098_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:55,107) - Starting with JobEnvironment(job_id=40557098, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:55,107) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557098/40557098_submitted.pkl
Trial-Index: 716
[2026-11-03T09:21:59.005] error: *** JOB 40557098 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40557116 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557116/40557116_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557116/40557116_0_log.out
----------------------------------------
submitit INFO (2026-11-03 08:21:59,757) - Starting with JobEnvironment(job_id=40557116, hostname=n1702, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 08:21:59,757) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40557116/40557116_submitted.pkl
Trial-Index: 726
[2026-11-03T09:21:59.007] error: *** JOB 40557116 ON n1702 CANCELLED AT 2026-11-03T09:21:59 DUE TO TIME LIMIT ***
âš Job 40558258 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558258/40558258_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558258/40558258_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:34,747) - Starting with JobEnvironment(job_id=40558258, hostname=n1698, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:34,749) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558258/40558258_submitted.pkl
Trial-Index: 787
[2026-11-03T10:48:33.200] error: *** JOB 40558258 ON n1698 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558243 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558243/40558243_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558243/40558243_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:33,634) - Starting with JobEnvironment(job_id=40558243, hostname=n1707, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:33,635) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558243/40558243_submitted.pkl
Trial-Index: 799
[2026-11-03T10:48:33.198] error: *** JOB 40558243 ON n1707 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558255 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558255/40558255_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558255/40558255_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:34,739) - Starting with JobEnvironment(job_id=40558255, hostname=n1698, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:34,741) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558255/40558255_submitted.pkl
Trial-Index: 795
[2026-11-03T10:48:33.200] error: *** JOB 40558255 ON n1698 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558241 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558241/40558241_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558241/40558241_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:33,511) - Starting with JobEnvironment(job_id=40558241, hostname=n1710, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:33,513) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558241/40558241_submitted.pkl
Trial-Index: 789
[2026-11-03T10:48:33.198] error: *** JOB 40558241 ON n1710 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558247 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558247/40558247_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558247/40558247_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:28,904) - Starting with JobEnvironment(job_id=40558247, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:28,905) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558247/40558247_submitted.pkl
Trial-Index: 761
[2026-11-03T10:48:33.198] error: *** JOB 40558247 ON n1627 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558256 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558256/40558256_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558256/40558256_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:34,741) - Starting with JobEnvironment(job_id=40558256, hostname=n1698, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:34,742) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558256/40558256_submitted.pkl
Trial-Index: 794
[2026-11-03T10:48:33.200] error: *** JOB 40558256 ON n1698 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558252 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558252/40558252_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558252/40558252_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:34,747) - Starting with JobEnvironment(job_id=40558252, hostname=n1698, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:34,748) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558252/40558252_submitted.pkl
Trial-Index: 773
[2026-11-03T10:48:33.199] error: *** JOB 40558252 ON n1698 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558248 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558248/40558248_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558248/40558248_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:28,904) - Starting with JobEnvironment(job_id=40558248, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:28,905) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558248/40558248_submitted.pkl
Trial-Index: 800
[2026-11-03T10:48:33.199] error: *** JOB 40558248 ON n1627 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558242 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558242/40558242_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558242/40558242_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:33,511) - Starting with JobEnvironment(job_id=40558242, hostname=n1710, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:33,513) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558242/40558242_submitted.pkl
Trial-Index: 788
[2026-11-03T10:48:33.199] error: *** JOB 40558242 ON n1710 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558246 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558246/40558246_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558246/40558246_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:28,904) - Starting with JobEnvironment(job_id=40558246, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:28,905) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558246/40558246_submitted.pkl
Trial-Index: 784
[2026-11-03T10:48:33.199] error: *** JOB 40558246 ON n1627 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558250 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558250/40558250_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558250/40558250_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:28,904) - Starting with JobEnvironment(job_id=40558250, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:28,905) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558250/40558250_submitted.pkl
Trial-Index: 758
[2026-11-03T10:48:33.199] error: *** JOB 40558250 ON n1627 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558249 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558249/40558249_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558249/40558249_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:28,904) - Starting with JobEnvironment(job_id=40558249, hostname=n1627, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:28,907) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558249/40558249_submitted.pkl
Trial-Index: 759
[2026-11-03T10:48:33.198] error: *** JOB 40558249 ON n1627 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***
âš Job 40558259 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558259/40558259_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558259/40558259_0_log.out
----------------------------------------
submitit INFO (2026-11-03 09:48:34,744) - Starting with JobEnvironment(job_id=40558259, hostname=n1698, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 09:48:34,745) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/3/single_runs/40558259/40558259_submitted.pkl
Trial-Index: 791
[2026-11-03T10:48:33.200] error: *** JOB 40558259 ON n1698 CANCELLED AT 2026-11-03T10:48:33 DUE TO TIME LIMIT ***