Parameters ranked by absolute Pearson correlation with each result. Stronger correlations (|r| ≥ 0.7) suggest the parameter has more influence on the outcome.
Convergence Analysis
Parameter statistics
Parameter
Min
Max
Mean
Std Dev
Count
run_time
212
5373
2087.9872
1430.5686
467
OO_Info_ZEIT
168.9594
5323.3248
2028.0974
1434.5712
467
RESULT
0.0009
0.0058
0.0018
0.0007
467
epochs
200
380
345.6187
50.6307
653
filters
30
100
77.8423
23.3501
653
window
5
15
10.3936
3.9927
653
stride
1
5
2.196
1.3907
653
dense
120
350
286.6753
63.2601
653
batchsz
4
16
7.5191
3.9131
653
lr
0
0.0002
0.0001
0.0001
653
âš Job 40573435 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40573435/40573435_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40573435/40573435_0_log.out
----------------------------------------
submitit INFO (2026-11-03 12:49:21,547) - Starting with JobEnvironment(job_id=40573435, hostname=n1636, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 12:49:21,549) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40573435/40573435_submitted.pkl
Trial-Index: 81
[2026-11-03T14:19:29.005] error: *** JOB 40573435 ON n1636 CANCELLED AT 2026-11-03T14:19:29 DUE TO TIME LIMIT ***
âš Job 40573420 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40573420/40573420_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40573420/40573420_0_log.out
----------------------------------------
submitit INFO (2026-11-03 12:49:10,140) - Starting with JobEnvironment(job_id=40573420, hostname=n1705, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 12:49:10,141) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40573420/40573420_submitted.pkl
Trial-Index: 90
[2026-11-03T14:18:59.012] error: *** JOB 40573420 ON n1705 CANCELLED AT 2026-11-03T14:18:59 DUE TO TIME LIMIT ***
âš Job 40585647 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585647/40585647_0_result.pkl
has not produced any output (state: NODE_FAIL)
No output/error stream produced ! Check: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585647/40585647_0_log.out
âš Job 40585645 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585645/40585645_0_result.pkl
has not produced any output (state: NODE_FAIL)
No output/error stream produced ! Check: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585645/40585645_0_log.out
âš Job 40585648 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585648/40585648_0_result.pkl
has not produced any output (state: NODE_FAIL)
No output/error stream produced ! Check: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585648/40585648_0_log.out
âš Job 40585642 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585642/40585642_0_result.pkl
has not produced any output (state: NODE_FAIL)
No output/error stream produced ! Check: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585642/40585642_0_log.out
âš Job 40585650 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585650/40585650_0_result.pkl
has not produced any output (state: NODE_FAIL)
No output/error stream produced ! Check: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585650/40585650_0_log.out
âš Job 40585646 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585646/40585646_0_result.pkl
has not produced any output (state: NODE_FAIL)
No output/error stream produced ! Check: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585646/40585646_0_log.out
âš Job 40585643 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585643/40585643_0_result.pkl
has not produced any output (state: NODE_FAIL)
No output/error stream produced ! Check: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585643/40585643_0_log.out
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585646/40585646_0_log.out not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585646/40585646_0_log.err not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585645/40585645_0_log.out not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585647/40585647_0_log.out not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585645/40585645_0_log.err not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585647/40585647_0_log.err not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585643/40585643_0_log.out not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585643/40585643_0_log.err not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585650/40585650_0_log.out not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585642/40585642_0_log.out not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585648/40585648_0_log.out not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585650/40585650_0_log.err not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585648/40585648_0_log.err not found
get_program_code_from_out_file: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40585642/40585642_0_log.err not found
âš Job 40586195 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586195/40586195_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586195/40586195_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:50:30,238) - Starting with JobEnvironment(job_id=40586195, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:50:30,239) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586195/40586195_submitted.pkl
Trial-Index: 224
[2026-11-03T18:20:29.548] error: *** JOB 40586195 ON n1713 CANCELLED AT 2026-11-03T18:20:29 DUE TO TIME LIMIT ***
âš Job 40586199 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586199/40586199_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586199/40586199_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:50:30,238) - Starting with JobEnvironment(job_id=40586199, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:50:30,239) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586199/40586199_submitted.pkl
Trial-Index: 229
[2026-11-03T18:20:29.548] error: *** JOB 40586199 ON n1713 CANCELLED AT 2026-11-03T18:20:29 DUE TO TIME LIMIT ***
âš Job 40586172 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586172/40586172_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586172/40586172_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:49:44,744) - Starting with JobEnvironment(job_id=40586172, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:49:44,745) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586172/40586172_submitted.pkl
Trial-Index: 205
[2026-11-03T18:19:57.486] error: *** JOB 40586172 ON n1713 CANCELLED AT 2026-11-03T18:19:57 DUE TO TIME LIMIT ***
âš Job 40586208 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586208/40586208_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586208/40586208_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:50:34,448) - Starting with JobEnvironment(job_id=40586208, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:50:34,448) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586208/40586208_submitted.pkl
Trial-Index: 241
[2026-11-03T18:20:29.548] error: *** JOB 40586208 ON n1713 CANCELLED AT 2026-11-03T18:20:29 DUE TO TIME LIMIT ***
âš Job 40586182 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586182/40586182_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586182/40586182_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:49:54,259) - Starting with JobEnvironment(job_id=40586182, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:49:54,259) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586182/40586182_submitted.pkl
Trial-Index: 233
[2026-11-03T18:19:57.486] error: *** JOB 40586182 ON n1713 CANCELLED AT 2026-11-03T18:19:57 DUE TO TIME LIMIT ***
âš Job 40586205 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586205/40586205_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586205/40586205_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:50:34,448) - Starting with JobEnvironment(job_id=40586205, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:50:34,449) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586205/40586205_submitted.pkl
Trial-Index: 211
[2026-11-03T18:20:29.548] error: *** JOB 40586205 ON n1713 CANCELLED AT 2026-11-03T18:20:29 DUE TO TIME LIMIT ***
âš Job 40586192 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586192/40586192_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586192/40586192_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:50:05,056) - Starting with JobEnvironment(job_id=40586192, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:50:05,057) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586192/40586192_submitted.pkl
Trial-Index: 212
[2026-11-03T18:19:57.487] error: *** JOB 40586192 ON n1713 CANCELLED AT 2026-11-03T18:19:57 DUE TO TIME LIMIT ***
âš Job 40586193 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586193/40586193_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586193/40586193_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:50:05,056) - Starting with JobEnvironment(job_id=40586193, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:50:05,057) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586193/40586193_submitted.pkl
Trial-Index: 223
[2026-11-03T18:19:57.487] error: *** JOB 40586193 ON n1713 CANCELLED AT 2026-11-03T18:19:57 DUE TO TIME LIMIT ***
âš Job 40586181 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586181/40586181_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586181/40586181_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:49:54,259) - Starting with JobEnvironment(job_id=40586181, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:49:54,259) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586181/40586181_submitted.pkl
Trial-Index: 201
[2026-11-03T18:19:57.486] error: *** JOB 40586181 ON n1713 CANCELLED AT 2026-11-03T18:19:57 DUE TO TIME LIMIT ***
âš Job 40586188 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586188/40586188_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586188/40586188_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:50:05,056) - Starting with JobEnvironment(job_id=40586188, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:50:05,058) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586188/40586188_submitted.pkl
Trial-Index: 210
[2026-11-03T18:19:57.488] error: *** JOB 40586188 ON n1713 CANCELLED AT 2026-11-03T18:19:57 DUE TO TIME LIMIT ***
âš Job 40586183 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586183/40586183_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586183/40586183_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:49:59,452) - Starting with JobEnvironment(job_id=40586183, hostname=n1720, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:49:59,453) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586183/40586183_submitted.pkl
Trial-Index: 203
[2026-11-03T18:19:57.485] error: *** JOB 40586183 ON n1720 CANCELLED AT 2026-11-03T18:19:57 DUE TO TIME LIMIT ***
âš Job 40586201 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586201/40586201_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586201/40586201_0_log.out
----------------------------------------
submitit INFO (2026-11-03 16:50:31,961) - Starting with JobEnvironment(job_id=40586201, hostname=n1713, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 16:50:31,961) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40586201/40586201_submitted.pkl
Trial-Index: 217
[2026-11-03T18:20:29.548] error: *** JOB 40586201 ON n1713 CANCELLED AT 2026-11-03T18:20:29 DUE TO TIME LIMIT ***
While saving all trials as a pandas-dataframe-csv, an error occurred: dictionary changed size during iteration
âš Job 40587942 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587942/40587942_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587942/40587942_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:38,263) - Starting with JobEnvironment(job_id=40587942, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:38,264) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587942/40587942_submitted.pkl
Trial-Index: 316
[2026-11-03T21:08:49.008] error: *** JOB 40587942 ON n1696 CANCELLED AT 2026-11-03T21:08:49 DUE TO TIME LIMIT ***
âš Job 40587956 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587956/40587956_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587956/40587956_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:59,711) - Starting with JobEnvironment(job_id=40587956, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:59,712) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587956/40587956_submitted.pkl
Trial-Index: 311
[2026-11-03T21:09:20.377] error: *** JOB 40587956 ON n1696 CANCELLED AT 2026-11-03T21:09:20 DUE TO TIME LIMIT ***
âš Job 40587926 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587926/40587926_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587926/40587926_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:38,359) - Starting with JobEnvironment(job_id=40587926, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:38,359) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587926/40587926_submitted.pkl
Trial-Index: 302
[2026-11-03T21:08:49.008] error: *** JOB 40587926 ON n1696 CANCELLED AT 2026-11-03T21:08:49 DUE TO TIME LIMIT ***
âš Job 40587949 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587949/40587949_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587949/40587949_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:38,303) - Starting with JobEnvironment(job_id=40587949, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:38,304) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587949/40587949_submitted.pkl
Trial-Index: 303
[2026-11-03T21:08:49.009] error: *** JOB 40587949 ON n1696 CANCELLED AT 2026-11-03T21:08:49 DUE TO TIME LIMIT ***
âš Job 40587954 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587954/40587954_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587954/40587954_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:59,699) - Starting with JobEnvironment(job_id=40587954, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:59,699) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587954/40587954_submitted.pkl
Trial-Index: 337
[2026-11-03T21:09:20.377] error: *** JOB 40587954 ON n1696 CANCELLED AT 2026-11-03T21:09:20 DUE TO TIME LIMIT ***
âš Job 40587958 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587958/40587958_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587958/40587958_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:59,699) - Starting with JobEnvironment(job_id=40587958, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:59,699) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587958/40587958_submitted.pkl
Trial-Index: 348
[2026-11-03T21:09:20.377] error: *** JOB 40587958 ON n1696 CANCELLED AT 2026-11-03T21:09:20 DUE TO TIME LIMIT ***
âš Job 40587939 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587939/40587939_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587939/40587939_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:38,406) - Starting with JobEnvironment(job_id=40587939, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:38,406) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587939/40587939_submitted.pkl
Trial-Index: 315
[2026-11-03T21:08:50.025] error: *** JOB 40587939 ON n1696 CANCELLED AT 2026-11-03T21:08:50 DUE TO TIME LIMIT ***
âš Job 40587945 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587945/40587945_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587945/40587945_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:38,607) - Starting with JobEnvironment(job_id=40587945, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:38,608) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587945/40587945_submitted.pkl
Trial-Index: 336
[2026-11-03T21:08:49.009] error: *** JOB 40587945 ON n1696 CANCELLED AT 2026-11-03T21:08:49 DUE TO TIME LIMIT ***
âš Job 40587947 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587947/40587947_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587947/40587947_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:38,579) - Starting with JobEnvironment(job_id=40587947, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:38,579) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587947/40587947_submitted.pkl
Trial-Index: 333
[2026-11-03T21:08:49.008] error: *** JOB 40587947 ON n1696 CANCELLED AT 2026-11-03T21:08:49 DUE TO TIME LIMIT ***
âš Job 40587957 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587957/40587957_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587957/40587957_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:59,711) - Starting with JobEnvironment(job_id=40587957, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:59,712) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587957/40587957_submitted.pkl
Trial-Index: 338
[2026-11-03T21:09:20.378] error: *** JOB 40587957 ON n1696 CANCELLED AT 2026-11-03T21:09:20 DUE TO TIME LIMIT ***
âš Job 40587955 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587955/40587955_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587955/40587955_0_log.out
----------------------------------------
submitit INFO (2026-11-03 19:38:59,699) - Starting with JobEnvironment(job_id=40587955, hostname=n1696, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 19:38:59,699) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40587955/40587955_submitted.pkl
Trial-Index: 321
[2026-11-03T21:09:20.378] error: *** JOB 40587955 ON n1696 CANCELLED AT 2026-11-03T21:09:20 DUE TO TIME LIMIT ***
âš Job 40588657 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588657/40588657_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588657/40588657_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,316) - Starting with JobEnvironment(job_id=40588657, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,318) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588657/40588657_submitted.pkl
Trial-Index: 353
[2026-11-03T22:57:39.061] error: *** JOB 40588657 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588641 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588641/40588641_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588641/40588641_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:38,932) - Starting with JobEnvironment(job_id=40588641, hostname=n1672, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:38,933) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588641/40588641_submitted.pkl
Trial-Index: 371
[2026-11-03T22:57:39.059] error: *** JOB 40588641 ON n1672 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588678 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588678/40588678_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588678/40588678_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:41,117) - Starting with JobEnvironment(job_id=40588678, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:41,117) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588678/40588678_submitted.pkl
Trial-Index: 365
[2026-11-03T22:57:39.061] error: *** JOB 40588678 ON n1450 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588654 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588654/40588654_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588654/40588654_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,316) - Starting with JobEnvironment(job_id=40588654, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,317) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588654/40588654_submitted.pkl
Trial-Index: 350
[2026-11-03T22:57:39.060] error: *** JOB 40588654 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588667 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588667/40588667_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588667/40588667_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:41,117) - Starting with JobEnvironment(job_id=40588667, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:41,117) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588667/40588667_submitted.pkl
Trial-Index: 356
[2026-11-03T22:57:39.060] error: *** JOB 40588667 ON n1450 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588650 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588650/40588650_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588650/40588650_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,316) - Starting with JobEnvironment(job_id=40588650, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,317) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588650/40588650_submitted.pkl
Trial-Index: 372
[2026-11-03T22:57:39.059] error: *** JOB 40588650 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588664 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588664/40588664_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588664/40588664_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,316) - Starting with JobEnvironment(job_id=40588664, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,317) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588664/40588664_submitted.pkl
Trial-Index: 398
[2026-11-03T22:57:39.061] error: *** JOB 40588664 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588663 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588663/40588663_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588663/40588663_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,319) - Starting with JobEnvironment(job_id=40588663, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,320) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588663/40588663_submitted.pkl
Trial-Index: 363
[2026-11-03T22:57:39.061] error: *** JOB 40588663 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588665 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588665/40588665_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588665/40588665_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:41,117) - Starting with JobEnvironment(job_id=40588665, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:41,117) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588665/40588665_submitted.pkl
Trial-Index: 364
[2026-11-03T22:57:39.060] error: *** JOB 40588665 ON n1450 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588649 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588649/40588649_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588649/40588649_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,319) - Starting with JobEnvironment(job_id=40588649, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,320) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588649/40588649_submitted.pkl
Trial-Index: 352
[2026-11-03T22:57:39.060] error: *** JOB 40588649 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588647 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588647/40588647_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588647/40588647_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,316) - Starting with JobEnvironment(job_id=40588647, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,317) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588647/40588647_submitted.pkl
Trial-Index: 391
[2026-11-03T22:57:39.060] error: *** JOB 40588647 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588658 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588658/40588658_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588658/40588658_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,319) - Starting with JobEnvironment(job_id=40588658, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,320) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588658/40588658_submitted.pkl
Trial-Index: 374
[2026-11-03T22:57:39.061] error: *** JOB 40588658 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588645 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588645/40588645_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588645/40588645_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,317) - Starting with JobEnvironment(job_id=40588645, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,318) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588645/40588645_submitted.pkl
Trial-Index: 386
[2026-11-03T22:57:39.059] error: *** JOB 40588645 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588648 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588648/40588648_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588648/40588648_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,319) - Starting with JobEnvironment(job_id=40588648, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,320) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588648/40588648_submitted.pkl
Trial-Index: 385
[2026-11-03T22:57:39.060] error: *** JOB 40588648 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588651 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588651/40588651_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588651/40588651_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,319) - Starting with JobEnvironment(job_id=40588651, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,319) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588651/40588651_submitted.pkl
Trial-Index: 373
[2026-11-03T22:57:39.061] error: *** JOB 40588651 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588674 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588674/40588674_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588674/40588674_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:41,117) - Starting with JobEnvironment(job_id=40588674, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:41,117) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588674/40588674_submitted.pkl
Trial-Index: 370
[2026-11-03T22:57:39.060] error: *** JOB 40588674 ON n1450 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588662 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588662/40588662_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588662/40588662_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,316) - Starting with JobEnvironment(job_id=40588662, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,317) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588662/40588662_submitted.pkl
Trial-Index: 376
[2026-11-03T22:57:39.061] error: *** JOB 40588662 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588652 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588652/40588652_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588652/40588652_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,319) - Starting with JobEnvironment(job_id=40588652, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,319) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588652/40588652_submitted.pkl
Trial-Index: 380
[2026-11-03T22:57:39.061] error: *** JOB 40588652 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588646 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588646/40588646_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588646/40588646_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,319) - Starting with JobEnvironment(job_id=40588646, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,320) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588646/40588646_submitted.pkl
Trial-Index: 388
[2026-11-03T22:57:39.059] error: *** JOB 40588646 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588683 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588683/40588683_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588683/40588683_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:54,436) - Starting with JobEnvironment(job_id=40588683, hostname=n1448, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:54,437) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588683/40588683_submitted.pkl
Trial-Index: 360
[2026-11-03T22:58:10.005] error: *** JOB 40588683 ON n1448 CANCELLED AT 2026-11-03T22:58:10 DUE TO TIME LIMIT ***
âš Job 40588660 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588660/40588660_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588660/40588660_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,316) - Starting with JobEnvironment(job_id=40588660, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,319) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588660/40588660_submitted.pkl
Trial-Index: 368
[2026-11-03T22:57:39.060] error: *** JOB 40588660 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588680 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588680/40588680_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588680/40588680_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:41,117) - Starting with JobEnvironment(job_id=40588680, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:41,117) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588680/40588680_submitted.pkl
Trial-Index: 369
[2026-11-03T22:57:39.061] error: *** JOB 40588680 ON n1450 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588669 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588669/40588669_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588669/40588669_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:41,117) - Starting with JobEnvironment(job_id=40588669, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:41,120) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588669/40588669_submitted.pkl
Trial-Index: 351
[2026-11-03T22:57:39.060] error: *** JOB 40588669 ON n1450 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588655 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588655/40588655_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588655/40588655_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,319) - Starting with JobEnvironment(job_id=40588655, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,319) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588655/40588655_submitted.pkl
Trial-Index: 355
[2026-11-03T22:57:39.060] error: *** JOB 40588655 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588685 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588685/40588685_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588685/40588685_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:54,436) - Starting with JobEnvironment(job_id=40588685, hostname=n1448, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:54,437) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588685/40588685_submitted.pkl
Trial-Index: 362
[2026-11-03T22:58:10.005] error: *** JOB 40588685 ON n1448 CANCELLED AT 2026-11-03T22:58:10 DUE TO TIME LIMIT ***
âš Job 40588661 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588661/40588661_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588661/40588661_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:40,316) - Starting with JobEnvironment(job_id=40588661, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:40,317) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588661/40588661_submitted.pkl
Trial-Index: 397
[2026-11-03T22:57:39.061] error: *** JOB 40588661 ON n1653 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40588679 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588679/40588679_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588679/40588679_0_log.out
----------------------------------------
submitit INFO (2026-11-03 21:27:41,117) - Starting with JobEnvironment(job_id=40588679, hostname=n1450, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 21:27:41,117) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40588679/40588679_submitted.pkl
Trial-Index: 366
[2026-11-03T22:57:39.060] error: *** JOB 40588679 ON n1450 CANCELLED AT 2026-11-03T22:57:39 DUE TO TIME LIMIT ***
âš Job 40589178 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589178/40589178_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589178/40589178_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:14,273) - Starting with JobEnvironment(job_id=40589178, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:14,274) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589178/40589178_submitted.pkl
Trial-Index: 416
[2026-11-04T00:43:36.004] error: *** JOB 40589178 ON n1709 CANCELLED AT 2026-11-04T00:43:36 DUE TO TIME LIMIT ***
âš Job 40589160 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589160/40589160_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589160/40589160_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:54,626) - Starting with JobEnvironment(job_id=40589160, hostname=n1675, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:54,629) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589160/40589160_submitted.pkl
Trial-Index: 447
[2026-11-04T00:43:05.010] error: *** JOB 40589160 ON n1675 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589157 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589157/40589157_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589157/40589157_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:55,799) - Starting with JobEnvironment(job_id=40589157, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:55,800) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589157/40589157_submitted.pkl
Trial-Index: 409
[2026-11-04T00:43:05.009] error: *** JOB 40589157 ON n1709 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589183 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589183/40589183_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589183/40589183_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:16,141) - Starting with JobEnvironment(job_id=40589183, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:16,144) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589183/40589183_submitted.pkl
Trial-Index: 419
[2026-11-04T00:43:36.005] error: *** JOB 40589183 ON n1700 CANCELLED AT 2026-11-04T00:43:36 DUE TO TIME LIMIT ***
âš Job 40589144 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589144/40589144_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589144/40589144_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:03,432) - Starting with JobEnvironment(job_id=40589144, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:03,432) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589144/40589144_submitted.pkl
Trial-Index: 449
[2026-11-04T00:42:06.005] error: *** JOB 40589144 ON n1709 CANCELLED AT 2026-11-04T00:42:06 DUE TO TIME LIMIT ***
âš Job 40589147 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589147/40589147_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589147/40589147_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:34,451) - Starting with JobEnvironment(job_id=40589147, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:34,451) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589147/40589147_submitted.pkl
Trial-Index: 439
[2026-11-04T00:42:35.006] error: *** JOB 40589147 ON n1709 CANCELLED AT 2026-11-04T00:42:35 DUE TO TIME LIMIT ***
âš Job 40589181 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589181/40589181_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589181/40589181_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:16,141) - Starting with JobEnvironment(job_id=40589181, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:16,143) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589181/40589181_submitted.pkl
Trial-Index: 420
[2026-11-04T00:43:36.005] error: *** JOB 40589181 ON n1700 CANCELLED AT 2026-11-04T00:43:36 DUE TO TIME LIMIT ***
âš Job 40589169 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589169/40589169_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589169/40589169_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:11,507) - Starting with JobEnvironment(job_id=40589169, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:11,508) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589169/40589169_submitted.pkl
Trial-Index: 411
[2026-11-04T00:43:05.010] error: *** JOB 40589169 ON n1663 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589173 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589173/40589173_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589173/40589173_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:11,512) - Starting with JobEnvironment(job_id=40589173, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:11,512) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589173/40589173_submitted.pkl
Trial-Index: 422
[2026-11-04T00:43:05.011] error: *** JOB 40589173 ON n1663 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589155 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589155/40589155_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589155/40589155_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:34,451) - Starting with JobEnvironment(job_id=40589155, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:34,451) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589155/40589155_submitted.pkl
Trial-Index: 401
[2026-11-04T00:42:35.006] error: *** JOB 40589155 ON n1709 CANCELLED AT 2026-11-04T00:42:35 DUE TO TIME LIMIT ***
âš Job 40589148 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589148/40589148_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589148/40589148_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:34,451) - Starting with JobEnvironment(job_id=40589148, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:34,451) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589148/40589148_submitted.pkl
Trial-Index: 404
[2026-11-04T00:42:35.006] error: *** JOB 40589148 ON n1709 CANCELLED AT 2026-11-04T00:42:35 DUE TO TIME LIMIT ***
âš Job 40589167 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589167/40589167_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589167/40589167_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:09,865) - Starting with JobEnvironment(job_id=40589167, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:09,866) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589167/40589167_submitted.pkl
Trial-Index: 400
[2026-11-04T00:43:05.010] error: *** JOB 40589167 ON n1663 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589152 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589152/40589152_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589152/40589152_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:34,451) - Starting with JobEnvironment(job_id=40589152, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:34,453) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589152/40589152_submitted.pkl
Trial-Index: 403
[2026-11-04T00:42:35.006] error: *** JOB 40589152 ON n1709 CANCELLED AT 2026-11-04T00:42:35 DUE TO TIME LIMIT ***
âš Job 40589149 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589149/40589149_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589149/40589149_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:34,451) - Starting with JobEnvironment(job_id=40589149, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:34,451) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589149/40589149_submitted.pkl
Trial-Index: 413
[2026-11-04T00:42:35.006] error: *** JOB 40589149 ON n1709 CANCELLED AT 2026-11-04T00:42:35 DUE TO TIME LIMIT ***
âš Job 40589151 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589151/40589151_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589151/40589151_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:34,451) - Starting with JobEnvironment(job_id=40589151, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:34,451) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589151/40589151_submitted.pkl
Trial-Index: 436
[2026-11-04T00:42:35.006] error: *** JOB 40589151 ON n1709 CANCELLED AT 2026-11-04T00:42:35 DUE TO TIME LIMIT ***
âš Job 40589143 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589143/40589143_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589143/40589143_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:03,434) - Starting with JobEnvironment(job_id=40589143, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:03,435) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589143/40589143_submitted.pkl
Trial-Index: 405
[2026-11-04T00:42:07.017] error: *** JOB 40589143 ON n1709 CANCELLED AT 2026-11-04T00:42:07 DUE TO TIME LIMIT ***
âš Job 40589176 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589176/40589176_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589176/40589176_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:11,512) - Starting with JobEnvironment(job_id=40589176, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:11,512) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589176/40589176_submitted.pkl
Trial-Index: 418
[2026-11-04T00:43:05.011] error: *** JOB 40589176 ON n1663 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589180 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589180/40589180_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589180/40589180_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:16,141) - Starting with JobEnvironment(job_id=40589180, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:16,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589180/40589180_submitted.pkl
Trial-Index: 428
[2026-11-04T00:43:36.005] error: *** JOB 40589180 ON n1700 CANCELLED AT 2026-11-04T00:43:36 DUE TO TIME LIMIT ***
âš Job 40589174 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589174/40589174_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589174/40589174_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:11,512) - Starting with JobEnvironment(job_id=40589174, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:11,512) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589174/40589174_submitted.pkl
Trial-Index: 426
[2026-11-04T00:43:05.010] error: *** JOB 40589174 ON n1663 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589166 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589166/40589166_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589166/40589166_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:56,651) - Starting with JobEnvironment(job_id=40589166, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:56,651) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589166/40589166_submitted.pkl
Trial-Index: 423
[2026-11-04T00:43:05.011] error: *** JOB 40589166 ON n1663 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589171 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589171/40589171_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589171/40589171_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:11,512) - Starting with JobEnvironment(job_id=40589171, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:11,513) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589171/40589171_submitted.pkl
Trial-Index: 431
[2026-11-04T00:43:05.010] error: *** JOB 40589171 ON n1663 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589145 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589145/40589145_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589145/40589145_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:03,429) - Starting with JobEnvironment(job_id=40589145, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:03,431) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589145/40589145_submitted.pkl
Trial-Index: 408
[2026-11-04T00:42:06.005] error: *** JOB 40589145 ON n1709 CANCELLED AT 2026-11-04T00:42:06 DUE TO TIME LIMIT ***
âš Job 40589170 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589170/40589170_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589170/40589170_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:11,507) - Starting with JobEnvironment(job_id=40589170, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:11,508) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589170/40589170_submitted.pkl
Trial-Index: 414
[2026-11-04T00:43:05.010] error: *** JOB 40589170 ON n1663 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589172 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589172/40589172_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589172/40589172_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:11,512) - Starting with JobEnvironment(job_id=40589172, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:11,512) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589172/40589172_submitted.pkl
Trial-Index: 444
[2026-11-04T00:43:05.010] error: *** JOB 40589172 ON n1663 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589146 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589146/40589146_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589146/40589146_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:03,433) - Starting with JobEnvironment(job_id=40589146, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:03,434) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589146/40589146_submitted.pkl
Trial-Index: 417
[2026-11-04T00:42:06.005] error: *** JOB 40589146 ON n1709 CANCELLED AT 2026-11-04T00:42:06 DUE TO TIME LIMIT ***
âš Job 40589158 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589158/40589158_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589158/40589158_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:55,799) - Starting with JobEnvironment(job_id=40589158, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:55,800) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589158/40589158_submitted.pkl
Trial-Index: 407
[2026-11-04T00:43:05.009] error: *** JOB 40589158 ON n1709 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40589156 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589156/40589156_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589156/40589156_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:34,451) - Starting with JobEnvironment(job_id=40589156, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:34,451) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589156/40589156_submitted.pkl
Trial-Index: 415
[2026-11-04T00:42:35.006] error: *** JOB 40589156 ON n1709 CANCELLED AT 2026-11-04T00:42:35 DUE TO TIME LIMIT ***
âš Job 40589185 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589185/40589185_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589185/40589185_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:13:16,141) - Starting with JobEnvironment(job_id=40589185, hostname=n1700, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:13:16,141) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589185/40589185_submitted.pkl
Trial-Index: 424
[2026-11-04T00:43:36.005] error: *** JOB 40589185 ON n1700 CANCELLED AT 2026-11-04T00:43:36 DUE TO TIME LIMIT ***
âš Job 40589150 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589150/40589150_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589150/40589150_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:34,451) - Starting with JobEnvironment(job_id=40589150, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:34,451) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589150/40589150_submitted.pkl
Trial-Index: 432
[2026-11-04T00:42:35.006] error: *** JOB 40589150 ON n1709 CANCELLED AT 2026-11-04T00:42:35 DUE TO TIME LIMIT ***
âš Job 40589142 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589142/40589142_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589142/40589142_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:03,431) - Starting with JobEnvironment(job_id=40589142, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:03,432) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589142/40589142_submitted.pkl
Trial-Index: 446
[2026-11-04T00:42:06.005] error: *** JOB 40589142 ON n1709 CANCELLED AT 2026-11-04T00:42:06 DUE TO TIME LIMIT ***
âš Job 40589159 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589159/40589159_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589159/40589159_0_log.out
----------------------------------------
submitit INFO (2026-11-03 23:12:55,799) - Starting with JobEnvironment(job_id=40589159, hostname=n1709, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-03 23:12:55,800) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40589159/40589159_submitted.pkl
Trial-Index: 429
[2026-11-04T00:43:05.009] error: *** JOB 40589159 ON n1709 CANCELLED AT 2026-11-04T00:43:05 DUE TO TIME LIMIT ***
âš Job 40593291 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593291/40593291_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593291/40593291_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:00:37,536) - Starting with JobEnvironment(job_id=40593291, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:00:37,537) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593291/40593291_submitted.pkl
Trial-Index: 473
[2026-11-04T02:30:35.909] error: *** JOB 40593291 ON n1663 CANCELLED AT 2026-11-04T02:30:35 DUE TO TIME LIMIT ***
âš Job 40593307 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593307/40593307_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593307/40593307_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,140) - Starting with JobEnvironment(job_id=40593307, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593307/40593307_submitted.pkl
Trial-Index: 463
[2026-11-04T02:31:05.039] error: *** JOB 40593307 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593298 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593298/40593298_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593298/40593298_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:00:41,763) - Starting with JobEnvironment(job_id=40593298, hostname=n1384, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:00:41,764) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593298/40593298_submitted.pkl
Trial-Index: 474
[2026-11-04T02:30:35.909] error: *** JOB 40593298 ON n1384 CANCELLED AT 2026-11-04T02:30:35 DUE TO TIME LIMIT ***
âš Job 40593295 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593295/40593295_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593295/40593295_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:00:38,608) - Starting with JobEnvironment(job_id=40593295, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:00:38,611) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593295/40593295_submitted.pkl
Trial-Index: 451
[2026-11-04T02:30:35.909] error: *** JOB 40593295 ON n1663 CANCELLED AT 2026-11-04T02:30:35 DUE TO TIME LIMIT ***
âš Job 40593297 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593297/40593297_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593297/40593297_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:00:41,764) - Starting with JobEnvironment(job_id=40593297, hostname=n1384, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:00:41,765) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593297/40593297_submitted.pkl
Trial-Index: 450
[2026-11-04T02:30:35.909] error: *** JOB 40593297 ON n1384 CANCELLED AT 2026-11-04T02:30:35 DUE TO TIME LIMIT ***
âš Job 40593287 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593287/40593287_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593287/40593287_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:00:37,536) - Starting with JobEnvironment(job_id=40593287, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:00:37,537) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593287/40593287_submitted.pkl
Trial-Index: 456
[2026-11-04T02:30:35.908] error: *** JOB 40593287 ON n1663 CANCELLED AT 2026-11-04T02:30:35 DUE TO TIME LIMIT ***
âš Job 40593312 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593312/40593312_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593312/40593312_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,141) - Starting with JobEnvironment(job_id=40593312, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593312/40593312_submitted.pkl
Trial-Index: 495
[2026-11-04T02:31:05.039] error: *** JOB 40593312 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593317 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593317/40593317_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593317/40593317_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,141) - Starting with JobEnvironment(job_id=40593317, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593317/40593317_submitted.pkl
Trial-Index: 466
[2026-11-04T02:31:05.039] error: *** JOB 40593317 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593322 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593322/40593322_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593322/40593322_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,140) - Starting with JobEnvironment(job_id=40593322, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,141) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593322/40593322_submitted.pkl
Trial-Index: 497
[2026-11-04T02:31:05.040] error: *** JOB 40593322 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593325 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593325/40593325_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593325/40593325_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,140) - Starting with JobEnvironment(job_id=40593325, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593325/40593325_submitted.pkl
Trial-Index: 492
[2026-11-04T02:31:05.040] error: *** JOB 40593325 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593299 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593299/40593299_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593299/40593299_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:00:41,763) - Starting with JobEnvironment(job_id=40593299, hostname=n1384, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:00:41,765) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593299/40593299_submitted.pkl
Trial-Index: 457
[2026-11-04T02:30:35.909] error: *** JOB 40593299 ON n1384 CANCELLED AT 2026-11-04T02:30:35 DUE TO TIME LIMIT ***
âš Job 40593290 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593290/40593290_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593290/40593290_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:00:37,536) - Starting with JobEnvironment(job_id=40593290, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:00:37,537) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593290/40593290_submitted.pkl
Trial-Index: 454
[2026-11-04T02:30:35.909] error: *** JOB 40593290 ON n1663 CANCELLED AT 2026-11-04T02:30:35 DUE TO TIME LIMIT ***
âš Job 40593323 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593323/40593323_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593323/40593323_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,141) - Starting with JobEnvironment(job_id=40593323, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593323/40593323_submitted.pkl
Trial-Index: 478
[2026-11-04T02:31:05.040] error: *** JOB 40593323 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593303 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593303/40593303_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593303/40593303_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:00:41,433) - Starting with JobEnvironment(job_id=40593303, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:00:41,433) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593303/40593303_submitted.pkl
Trial-Index: 476
[2026-11-04T02:30:35.910] error: *** JOB 40593303 ON n1322 CANCELLED AT 2026-11-04T02:30:35 DUE TO TIME LIMIT ***
âš Job 40593321 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593321/40593321_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593321/40593321_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,141) - Starting with JobEnvironment(job_id=40593321, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593321/40593321_submitted.pkl
Trial-Index: 453
[2026-11-04T02:31:05.040] error: *** JOB 40593321 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593315 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593315/40593315_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593315/40593315_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,141) - Starting with JobEnvironment(job_id=40593315, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,143) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593315/40593315_submitted.pkl
Trial-Index: 489
[2026-11-04T02:31:05.039] error: *** JOB 40593315 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593316 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593316/40593316_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593316/40593316_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,140) - Starting with JobEnvironment(job_id=40593316, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593316/40593316_submitted.pkl
Trial-Index: 499
[2026-11-04T02:31:05.039] error: *** JOB 40593316 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593319 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593319/40593319_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593319/40593319_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,141) - Starting with JobEnvironment(job_id=40593319, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593319/40593319_submitted.pkl
Trial-Index: 480
[2026-11-04T02:31:05.039] error: *** JOB 40593319 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40593305 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593305/40593305_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593305/40593305_0_log.out
----------------------------------------
submitit INFO (2026-11-04 01:01:14,140) - Starting with JobEnvironment(job_id=40593305, hostname=n1322, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 01:01:14,142) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40593305/40593305_submitted.pkl
Trial-Index: 482
[2026-11-04T02:31:05.040] error: *** JOB 40593305 ON n1322 CANCELLED AT 2026-11-04T02:31:05 DUE TO TIME LIMIT ***
âš Job 40595289 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595289/40595289_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595289/40595289_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:34,467) - Starting with JobEnvironment(job_id=40595289, hostname=n1640, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:34,468) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595289/40595289_submitted.pkl
Trial-Index: 513
[2026-11-04T04:22:25.075] error: *** JOB 40595289 ON n1640 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595308 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595308/40595308_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595308/40595308_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,954) - Starting with JobEnvironment(job_id=40595308, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,955) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595308/40595308_submitted.pkl
Trial-Index: 532
[2026-11-04T04:22:57.005] error: *** JOB 40595308 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595271 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595271/40595271_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595271/40595271_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:28,742) - Starting with JobEnvironment(job_id=40595271, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:28,743) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595271/40595271_submitted.pkl
Trial-Index: 549
[2026-11-04T04:22:25.074] error: *** JOB 40595271 ON n1663 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595272 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595272/40595272_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595272/40595272_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:28,742) - Starting with JobEnvironment(job_id=40595272, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:28,743) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595272/40595272_submitted.pkl
Trial-Index: 510
[2026-11-04T04:22:25.074] error: *** JOB 40595272 ON n1663 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595309 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595309/40595309_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595309/40595309_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,949) - Starting with JobEnvironment(job_id=40595309, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,950) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595309/40595309_submitted.pkl
Trial-Index: 533
[2026-11-04T04:22:57.006] error: *** JOB 40595309 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595288 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595288/40595288_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595288/40595288_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:34,467) - Starting with JobEnvironment(job_id=40595288, hostname=n1640, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:34,468) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595288/40595288_submitted.pkl
Trial-Index: 502
[2026-11-04T04:22:25.075] error: *** JOB 40595288 ON n1640 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595302 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595302/40595302_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595302/40595302_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,954) - Starting with JobEnvironment(job_id=40595302, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,955) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595302/40595302_submitted.pkl
Trial-Index: 524
[2026-11-04T04:22:57.004] error: *** JOB 40595302 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595297 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595297/40595297_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595297/40595297_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,954) - Starting with JobEnvironment(job_id=40595297, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,955) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595297/40595297_submitted.pkl
Trial-Index: 528
[2026-11-04T04:22:57.005] error: *** JOB 40595297 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595274 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595274/40595274_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595274/40595274_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:28,742) - Starting with JobEnvironment(job_id=40595274, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:28,743) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595274/40595274_submitted.pkl
Trial-Index: 515
[2026-11-04T04:22:25.074] error: *** JOB 40595274 ON n1663 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595281 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595281/40595281_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595281/40595281_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:34,467) - Starting with JobEnvironment(job_id=40595281, hostname=n1640, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:34,468) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595281/40595281_submitted.pkl
Trial-Index: 522
[2026-11-04T04:22:25.075] error: *** JOB 40595281 ON n1640 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595286 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595286/40595286_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595286/40595286_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:34,467) - Starting with JobEnvironment(job_id=40595286, hostname=n1640, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:34,468) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595286/40595286_submitted.pkl
Trial-Index: 512
[2026-11-04T04:22:25.076] error: *** JOB 40595286 ON n1640 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595273 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595273/40595273_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595273/40595273_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:28,742) - Starting with JobEnvironment(job_id=40595273, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:28,743) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595273/40595273_submitted.pkl
Trial-Index: 511
[2026-11-04T04:22:25.073] error: *** JOB 40595273 ON n1663 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595295 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595295/40595295_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595295/40595295_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,954) - Starting with JobEnvironment(job_id=40595295, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,955) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595295/40595295_submitted.pkl
Trial-Index: 506
[2026-11-04T04:22:57.004] error: *** JOB 40595295 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595279 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595279/40595279_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595279/40595279_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:28,742) - Starting with JobEnvironment(job_id=40595279, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:28,743) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595279/40595279_submitted.pkl
Trial-Index: 501
[2026-11-04T04:22:25.074] error: *** JOB 40595279 ON n1663 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595285 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595285/40595285_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595285/40595285_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:34,467) - Starting with JobEnvironment(job_id=40595285, hostname=n1640, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:34,468) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595285/40595285_submitted.pkl
Trial-Index: 527
[2026-11-04T04:22:25.075] error: *** JOB 40595285 ON n1640 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595314 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595314/40595314_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595314/40595314_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,954) - Starting with JobEnvironment(job_id=40595314, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,954) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595314/40595314_submitted.pkl
Trial-Index: 545
[2026-11-04T04:22:57.006] error: *** JOB 40595314 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595276 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595276/40595276_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595276/40595276_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:28,742) - Starting with JobEnvironment(job_id=40595276, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:28,743) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595276/40595276_submitted.pkl
Trial-Index: 509
[2026-11-04T04:22:25.074] error: *** JOB 40595276 ON n1663 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595291 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595291/40595291_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595291/40595291_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:34,467) - Starting with JobEnvironment(job_id=40595291, hostname=n1640, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:34,470) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595291/40595291_submitted.pkl
Trial-Index: 504
[2026-11-04T04:22:25.075] error: *** JOB 40595291 ON n1640 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595310 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595310/40595310_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595310/40595310_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,954) - Starting with JobEnvironment(job_id=40595310, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,955) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595310/40595310_submitted.pkl
Trial-Index: 537
[2026-11-04T04:22:57.006] error: *** JOB 40595310 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595311 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595311/40595311_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595311/40595311_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,949) - Starting with JobEnvironment(job_id=40595311, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,951) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595311/40595311_submitted.pkl
Trial-Index: 517
[2026-11-04T04:22:57.006] error: *** JOB 40595311 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595315 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595315/40595315_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595315/40595315_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,949) - Starting with JobEnvironment(job_id=40595315, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,950) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595315/40595315_submitted.pkl
Trial-Index: 518
[2026-11-04T04:22:57.006] error: *** JOB 40595315 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595296 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595296/40595296_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595296/40595296_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,954) - Starting with JobEnvironment(job_id=40595296, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,955) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595296/40595296_submitted.pkl
Trial-Index: 514
[2026-11-04T04:22:57.005] error: *** JOB 40595296 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595303 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595303/40595303_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595303/40595303_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,954) - Starting with JobEnvironment(job_id=40595303, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,955) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595303/40595303_submitted.pkl
Trial-Index: 531
[2026-11-04T04:22:57.005] error: *** JOB 40595303 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40595290 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595290/40595290_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595290/40595290_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:34,467) - Starting with JobEnvironment(job_id=40595290, hostname=n1640, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:34,468) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595290/40595290_submitted.pkl
Trial-Index: 516
[2026-11-04T04:22:25.075] error: *** JOB 40595290 ON n1640 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595280 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595280/40595280_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595280/40595280_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:28,742) - Starting with JobEnvironment(job_id=40595280, hostname=n1663, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:28,743) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595280/40595280_submitted.pkl
Trial-Index: 507
[2026-11-04T04:22:25.074] error: *** JOB 40595280 ON n1663 CANCELLED AT 2026-11-04T04:22:25 DUE TO TIME LIMIT ***
âš Job 40595304 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595304/40595304_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595304/40595304_0_log.out
----------------------------------------
submitit INFO (2026-11-04 02:52:53,952) - Starting with JobEnvironment(job_id=40595304, hostname=n1711, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 02:52:53,952) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40595304/40595304_submitted.pkl
Trial-Index: 525
[2026-11-04T04:22:57.005] error: *** JOB 40595304 ON n1711 CANCELLED AT 2026-11-04T04:22:57 DUE TO TIME LIMIT ***
âš Job 40596490 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596490/40596490_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596490/40596490_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:19,519) - Starting with JobEnvironment(job_id=40596490, hostname=n1677, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:19,520) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596490/40596490_submitted.pkl
Trial-Index: 557
[2026-11-04T06:19:18.349] error: *** JOB 40596490 ON n1677 CANCELLED AT 2026-11-04T06:19:18 DUE TO TIME LIMIT ***
âš Job 40596494 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596494/40596494_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596494/40596494_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:24,265) - Starting with JobEnvironment(job_id=40596494, hostname=n1667, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:24,267) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596494/40596494_submitted.pkl
Trial-Index: 563
[2026-11-04T06:19:18.350] error: *** JOB 40596494 ON n1667 CANCELLED AT 2026-11-04T06:19:18 DUE TO TIME LIMIT ***
âš Job 40596489 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596489/40596489_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596489/40596489_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:19,519) - Starting with JobEnvironment(job_id=40596489, hostname=n1677, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:19,520) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596489/40596489_submitted.pkl
Trial-Index: 553
[2026-11-04T06:19:18.349] error: *** JOB 40596489 ON n1677 CANCELLED AT 2026-11-04T06:19:18 DUE TO TIME LIMIT ***
âš Job 40596513 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596513/40596513_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596513/40596513_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:48,085) - Starting with JobEnvironment(job_id=40596513, hostname=n1667, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:48,085) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596513/40596513_submitted.pkl
Trial-Index: 598
[2026-11-04T06:19:48.005] error: *** JOB 40596513 ON n1667 CANCELLED AT 2026-11-04T06:19:48 DUE TO TIME LIMIT ***
âš Job 40596529 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596529/40596529_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596529/40596529_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:53,248) - Starting with JobEnvironment(job_id=40596529, hostname=n1615, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:53,249) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596529/40596529_submitted.pkl
Trial-Index: 568
[2026-11-04T06:19:48.005] error: *** JOB 40596529 ON n1615 CANCELLED AT 2026-11-04T06:19:48 DUE TO TIME LIMIT ***
âš Job 40596493 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596493/40596493_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596493/40596493_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:24,268) - Starting with JobEnvironment(job_id=40596493, hostname=n1667, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:24,269) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596493/40596493_submitted.pkl
Trial-Index: 560
[2026-11-04T06:19:18.350] error: *** JOB 40596493 ON n1667 CANCELLED AT 2026-11-04T06:19:18 DUE TO TIME LIMIT ***
âš Job 40596535 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596535/40596535_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596535/40596535_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:50:12,140) - Starting with JobEnvironment(job_id=40596535, hostname=n1675, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:50:12,140) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596535/40596535_submitted.pkl
Trial-Index: 577
[2026-11-04T06:20:18.005] error: *** JOB 40596535 ON n1675 CANCELLED AT 2026-11-04T06:20:18 DUE TO TIME LIMIT ***
âš Job 40596517 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596517/40596517_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596517/40596517_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:48,085) - Starting with JobEnvironment(job_id=40596517, hostname=n1667, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:48,085) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596517/40596517_submitted.pkl
Trial-Index: 570
[2026-11-04T06:19:48.005] error: *** JOB 40596517 ON n1667 CANCELLED AT 2026-11-04T06:19:48 DUE TO TIME LIMIT ***
âš Job 40596491 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596491/40596491_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596491/40596491_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:24,267) - Starting with JobEnvironment(job_id=40596491, hostname=n1667, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:24,269) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596491/40596491_submitted.pkl
Trial-Index: 556
[2026-11-04T06:19:18.350] error: *** JOB 40596491 ON n1667 CANCELLED AT 2026-11-04T06:19:18 DUE TO TIME LIMIT ***
âš Job 40596512 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596512/40596512_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596512/40596512_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:47,974) - Starting with JobEnvironment(job_id=40596512, hostname=n1673, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:47,975) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596512/40596512_submitted.pkl
Trial-Index: 584
[2026-11-04T06:19:48.005] error: *** JOB 40596512 ON n1673 CANCELLED AT 2026-11-04T06:19:48 DUE TO TIME LIMIT ***
âš Job 40596492 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596492/40596492_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596492/40596492_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:24,267) - Starting with JobEnvironment(job_id=40596492, hostname=n1667, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:24,269) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596492/40596492_submitted.pkl
Trial-Index: 551
[2026-11-04T06:19:18.350] error: *** JOB 40596492 ON n1667 CANCELLED AT 2026-11-04T06:19:18 DUE TO TIME LIMIT ***
âš Job 40596525 (task: 0) with path /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596525/40596525_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596525/40596525_0_log.out
----------------------------------------
submitit INFO (2026-11-04 04:49:55,018) - Starting with JobEnvironment(job_id=40596525, hostname=n1653, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-11-04 04:49:55,019) - Loading pickle: /data/horse/ws/pwinkler-timeseries/omniopt/runs/timeseries/4/single_runs/40596525/40596525_submitted.pkl
Trial-Index: 600
[2026-11-04T06:19:48.005] error: *** JOB 40596525 ON n1653 CANCELLED AT 2026-11-04T06:19:48 DUE TO TIME LIMIT ***