GUI page with all the settings of this job

Experiment overview

Job Summary per Generation Node

Experiment parameters

Constraints

Number of evaluations

Result names and types

Last progressbar status

Git-Version

Copy raw data to clipboard

Download »results.csv« as file

trial_index,submit_time,queue_time,worker_generator_uuid,start_time,end_time,run_time,program_string,exit_code,signal,hostname,OO_Info_SLURM_JOB_ID,arm_name,trial_status,generation_node,VAL_LOSS,PARAMS,batch_size,ffn,max_turnstiles,epochs,n_layers,d_model,max_sequence_length,lr,n_heads,batches_per_epoch,val_batches
0,1777984079,19,cef43004-eb21-4c98-bf01-6617884ae783,1777984098,1777997992,13894,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=250 --topo --topo-every 1 --kelp-every=1 --n-layers=7 --d-model=3 --n-heads=110 --tokenizer_initial_nr=1000 --lr=0.04539815329792453585 --batches-per-epoch=19 --val-batches=128 --epochs=51 --task=turnstile --max-seq-len 37 --max-turnstiles 13 --scheduler=cosine --ffn=126,0,,c150,3491044,0_0,COMPLETED,SOBOL,0.000004999999999999999561982322,1760444,250,126,13,51,7,3,37,0.045398153297924535853891114812,110,19,128
1,,,,,,,,,,,,1_0,FAILED,SOBOL,,,496,240,29,149,11,8,58,0.073062541823745150160185346522,9,5,128
2,,,,,,,,,,,,2_0,FAILED,SOBOL,,,391,467,9,230,14,5,22,0.00046727962098233404540648106,82,17,128
3,,,,,,,,,,,,3_0,FAILED,SOBOL,,,349,33,39,283,18,7,45,0.035828632492153074384777511341,128,15,128
4,1777984079,10,cef43004-eb21-4c98-bf01-6617884ae783,1777984089,1777988682,4593,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=368 --topo --topo-every 1 --kelp-every=1 --n-layers=7 --d-model=8 --n-heads=90 --tokenizer_initial_nr=1000 --lr=0.00908666021894002224 --batches-per-epoch=4 --val-batches=128 --epochs=385 --task=turnstile --max-seq-len 25 --max-turnstiles 20 --scheduler=cosine --ffn=406,0,,c154,3491035,4_0,FAILED,SOBOL,,,368,406,20,385,7,8,25,0.009086660218940022237799070126,90,4,128
5,,,,,,,,,,,,5_0,FAILED,SOBOL,,,173,203,36,201,20,4,54,0.051163389289718030905618206816,19,10,128
6,,,,,,,,,,,,6_0,FAILED,SOBOL,,,132,263,5,314,13,3,17,0.084547247357930441213902383879,62,19,128
7,1777984079,19,cef43004-eb21-4c98-bf01-6617884ae783,1777984098,1777993208,9110,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=43 --topo --topo-every 1 --kelp-every=1 --n-layers=15 --d-model=3 --n-heads=101 --tokenizer_initial_nr=1000 --lr=0.03839238840223792998 --batches-per-epoch=1 --val-batches=128 --epochs=326 --task=turnstile --max-seq-len 41 --max-turnstiles 32 --scheduler=cosine --ffn=59,0,,c149,3491046,7_0,COMPLETED,SOBOL,0.692280999999999924199300949113,2836159,43,59,32,326,15,3,41,0.038392388402237929978522146257,101,1,128
8,,,,,,,,,,,,8_0,FAILED,SOBOL,,,270,170,16,486,8,6,62,0.067765729978045918802109781609,2,17,128
9,1777984079,19,cef43004-eb21-4c98-bf01-6617884ae783,1777984098,1778017649,33551,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=27 --topo --topo-every 1 --kelp-every=1 --n-layers=6 --d-model=6 --n-heads=114 --tokenizer_initial_nr=1000 --lr=0.09994489770304942899 --batches-per-epoch=15 --val-batches=128 --epochs=244 --task=turnstile --max-seq-len 59 --max-turnstiles 53 --scheduler=cosine --ffn=378,0,,c150,3491043,9_0,COMPLETED,SOBOL,0.693278999999999978598452798906,2298914,27,378,53,244,6,6,59,0.099944897703049428994148684069,114,15,128
10,,,,,,,,,,,,10_0,FAILED,SOBOL,,,454,39,7,428,18,2,27,0.041558050784867170412972825488,59,3,128
11,,,,,,,,,,,,11_0,FAILED,SOBOL,,,433,283,33,443,17,5,53,0.08138948165151467517564043419,100,20,128
12,1777984079,19,cef43004-eb21-4c98-bf01-6617884ae783,1777984098,1777985453,1355,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=190 --topo --topo-every 1 --kelp-every=1 --n-layers=2 --d-model=6 --n-heads=7 --tokenizer_initial_nr=1000 --lr=0.01227006670031025905 --batches-per-epoch=5 --val-batches=128 --epochs=372 --task=turnstile --max-seq-len 42 --max-turnstiles 19 --scheduler=cosine --ffn=393,0,,c150,3491042,12_0,FAILED,SOBOL,,,190,393,19,372,2,6,42,0.012270066700310259047435046398,7,5,128
13,1777984080,28,cef43004-eb21-4c98-bf01-6617884ae783,1777984108,1777987169,3061,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=333 --topo --topo-every 1 --kelp-every=1 --n-layers=4 --d-model=2 --n-heads=108 --tokenizer_initial_nr=1000 --lr=0.07741046143263262402 --batches-per-epoch=1 --val-batches=128 --epochs=165 --task=turnstile --max-seq-len 57 --max-turnstiles 15 --scheduler=cosine --ffn=351,0,,c148,3491051,13_0,COMPLETED,SOBOL,0.618102999999999958014029743936,1364582,333,351,15,165,4,2,57,0.077410461432632624023675305125,108,1,128
14,1777984079,29,cef43004-eb21-4c98-bf01-6617884ae783,1777984108,1777998575,14467,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=104 --topo --topo-every 1 --kelp-every=1 --n-layers=20 --d-model=7 --n-heads=16 --tokenizer_initial_nr=1000 --lr=0.00355474116039620688 --batches-per-epoch=16 --val-batches=128 --epochs=33 --task=turnstile --max-seq-len 46 --max-turnstiles 30 --scheduler=cosine --ffn=463,0,,c148,3491050,14_0,COMPLETED,SOBOL,0.078158999999999992369659196356,689934,104,463,30,33,20,7,46,0.003554741160396206883786973663,16,16,128
15,1777984079,29,cef43004-eb21-4c98-bf01-6617884ae783,1777984108,1777984368,260,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=68 --topo --topo-every 1 --kelp-every=1 --n-layers=16 --d-model=7 --n-heads=87 --tokenizer_initial_nr=1000 --lr=0.06997305950262966423 --batches-per-epoch=3 --val-batches=128 --epochs=116 --task=turnstile --max-seq-len 12 --max-turnstiles 11 --scheduler=cosine --ffn=244,0,,c148,3491049,15_0,FAILED,SOBOL,,,68,244,11,116,16,7,12,0.069973059502629664230255457369,87,3,128
16,,,,,,,,,,,,16_0,FAILED,SOBOL,,,236,320,41,398,13,8,63,0.091276855921758340284277721821,121,6,128
17,1777984079,28,cef43004-eb21-4c98-bf01-6617884ae783,1777984107,1778004061,19954,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=306 --topo --topo-every 1 --kelp-every=1 --n-layers=1 --d-model=4 --n-heads=52 --tokenizer_initial_nr=1000 --lr=0.04846750081764068796 --batches-per-epoch=18 --val-batches=128 --epochs=214 --task=turnstile --max-seq-len 31 --max-turnstiles 25 --scheduler=cosine --ffn=99,0,,c147,3491054,17_0,COMPLETED,SOBOL,0.000018000000000000000456015434,66037,306,99,25,214,1,4,31,0.048467500817640687960086154362,52,18,128
18,1777984080,28,cef43004-eb21-4c98-bf01-6617884ae783,1777984108,1777995702,11594,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=301 --topo --topo-every 1 --kelp-every=1 --n-layers=18 --d-model=3 --n-heads=95 --tokenizer_initial_nr=1000 --lr=0.03397932908425713161 --batches-per-epoch=8 --val-batches=128 --epochs=59 --task=turnstile --max-seq-len 60 --max-turnstiles 6 --scheduler=cosine --ffn=449,0,,c148,3491052,18_0,COMPLETED,SOBOL,0.000025999999999999998399934434,5711314,301,449,6,59,18,3,60,0.033979329084257131610158353396,95,8,128
19,1777984080,27,cef43004-eb21-4c98-bf01-6617884ae783,1777984107,1777990647,6540,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=72 --topo --topo-every 1 --kelp-every=1 --n-layers=3 --d-model=6 --n-heads=57 --tokenizer_initial_nr=1000 --lr=0.06008076319750119132 --batches-per-epoch=12 --val-batches=128 --epochs=142 --task=turnstile --max-seq-len 35 --max-turnstiles 20 --scheduler=cosine --ffn=302,0,,c147,3491053,19_0,COMPLETED,SOBOL,0.117175000000000001376676550535,367760,72,302,20,142,3,6,35,0.060080763197501191319460644991,57,12,128
20,,,,,,,,,,,,20_0,RUNNING,SOBOL,,,143,477,46,259,6,7,52,0.047455095238721281547533692446,80,3,128
21,,,,,,,,,,,,21_0,RUNNING,SOBOL,,,379,266,43,224,8,2,47,0.062595925910732624908305865574,36,4,128
22,,,,,,,,,,,,22_0,RUNNING,SOBOL,,,422,197,13,291,2,3,24,0.097931488064164295970215334819,14,7,128
23,,,,,,,,,,,,23_0,RUNNING,SOBOL,,,465,505,24,334,5,3,63,0.025214394089312478636211167782,85,14,128
24,,,,,,,,,,,,24_0,RUNNING,SOBOL,,,221,364,10,477,20,8,39,0.056149825811212049353482456127,49,9,128
25,,,,,,,,,,,,25_0,RUNNING,SOBOL,,,205,42,45,92,3,2,58,0.007100740214735642812093896481,37,17,128
26,,,,,,,,,,,,26_0,RUNNING,SOBOL,,,136,217,27,465,5,2,45,0.094775165161238336675175730761,83,8,128
27,,,,,,,,,,,,27_0,RUNNING,SOBOL,,,364,77,11,349,12,7,51,0.024093701787151956816845910225,55,13,128
28,,,,,,,,,,,,28_0,RUNNING,SOBOL,,,157,161,6,188,13,4,48,0.089427265666796276755157180105,91,13,128
29,1778027458,2649,cef43004-eb21-4c98-bf01-6617884ae783,1778030107,1778035963,5856,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=403 --topo --topo-every 1 --kelp-every=1 --n-layers=9 --d-model=7 --n-heads=64 --tokenizer_initial_nr=1000 --lr=0.01713444616300975956 --batches-per-epoch=8 --val-batches=128 --epochs=10 --task=turnstile --max-seq-len 55 --max-turnstiles 22 --scheduler=cosine --ffn=14,0,,c53,3493379,29_0,COMPLETED,SOBOL,0.644831000000000043037573504989,633984,403,14,22,10,9,7,55,0.01713444616300975956169061476,64,8,128
30,,,,,,,,,,,,30_0,RUNNING,SOBOL,,,485,305,18,124,7,8,32,0.056990434118747622382894491011,103,12,128
31,,,,,,,,,,,,31_0,RUNNING,SOBOL,,,312,474,32,192,10,7,42,0.086990926944091731543018397588,115,7,128
32,1778027458,2680,cef43004-eb21-4c98-bf01-6617884ae783,1778030138,1778032168,2030,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=60 --topo --topo-every 1 --kelp-every=1 --n-layers=15 --d-model=2 --n-heads=24 --tokenizer_initial_nr=1000 --lr=0.00683676631502853731 --batches-per-epoch=12 --val-batches=128 --epochs=14 --task=turnstile --max-seq-len 53 --max-turnstiles 16 --scheduler=cosine --ffn=331,0,,c127,3493381,32_0,COMPLETED,SOBOL,0.02212199999999999930344607435,626855,60,331,16,14,15,2,53,0.006836766315028537308040590403,24,12,128
33,,,,,,,,,,,,33_0,RUNNING,SOBOL,,,396,299,36,304,3,8,60,0.024220340948431472172241996077,6,15,128
34,,,,,,,,,,,,34_0,RUNNING,SOBOL,,,79,120,22,121,11,5,27,0.067078333021519803702759077169,78,9,128
35,,,,,,,,,,,,35_0,RUNNING,SOBOL,,,354,230,5,202,6,6,7,0.039273571041013866089741668475,44,13,128
36,,,,,,,,,,,,36_0,RUNNING,SOBOL,,,359,367,54,89,16,5,64,0.015796249185855874147543786989,14,5,128
37,,,,,,,,,,,,37_0,RUNNING,SOBOL,,,179,51,8,272,9,2,31,0.074134457644952841959273825978,72,12,128
38,1778027468,2755,cef43004-eb21-4c98-bf01-6617884ae783,1778030223,1778030248,25,python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=136 --topo --topo-every 1 --kelp-every=1 --n-layers=1 --d-model=4 --n-heads=106 --tokenizer_initial_nr=1000 --lr=0.0903351544193544409 --batches-per-epoch=20 --val-batches=128 --epochs=233 --task=turnstile --max-seq-len 40 --max-turnstiles 40 --scheduler=cosine --ffn=510,0,,c70,3493388,38_0,FAILED,SOBOL,,,136,510,40,233,1,4,40,0.090335154419354440902267810998,106,20,128
39,,,,,,,,,,,,39_0,RUNNING,SOBOL,,,475,415,12,470,13,7,46,0.07842189452549210015863678791,125,15,128

Copy raw data to clipboard

Download »results.csv« as file

Copy raw data to clipboard

Download »oo_errors.txt« as file

⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.


⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.


⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.


⚠ Job 3491048 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491048/3491048_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491048/3491048_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:20,638) - Starting with JobEnvironment(job_id=3491048, hostname=c149, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:20,640) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491048/3491048_submitted.pkl
Trial-Index: 11
slurmstepd: error: *** JOB 3491048 ON c149 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***



⚠ Job 3491037 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491037/3491037_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491037/3491037_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:11,021) - Starting with JobEnvironment(job_id=3491037, hostname=c152, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:11,023) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491037/3491037_submitted.pkl
Trial-Index: 8
slurmstepd: error: *** JOB 3491037 ON c152 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***



⚠ Job 3491038 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491038/3491038_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491038/3491038_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:11,021) - Starting with JobEnvironment(job_id=3491038, hostname=c152, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:11,031) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491038/3491038_submitted.pkl
Trial-Index: 1
slurmstepd: error: *** JOB 3491038 ON c152 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***



⚠ Job 3491047 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491047/3491047_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491047/3491047_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:20,638) - Starting with JobEnvironment(job_id=3491047, hostname=c149, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:20,640) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491047/3491047_submitted.pkl
Trial-Index: 2
slurmstepd: error: *** JOB 3491047 ON c149 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***



⚠ Job 3491039 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491039/3491039_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491039/3491039_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:11,021) - Starting with JobEnvironment(job_id=3491039, hostname=c152, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:11,023) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491039/3491039_submitted.pkl
Trial-Index: 6
slurmstepd: error: *** JOB 3491039 ON c152 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***



⚠ Job 3491036 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491036/3491036_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491036/3491036_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:01,228) - Starting with JobEnvironment(job_id=3491036, hostname=c153, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:01,230) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491036/3491036_submitted.pkl
Trial-Index: 10
slurmstepd: error: *** JOB 3491036 ON c153 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***



⚠ Job 3491040 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491040/3491040_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491040/3491040_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:11,020) - Starting with JobEnvironment(job_id=3491040, hostname=c152, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:11,022) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491040/3491040_submitted.pkl
Trial-Index: 5
slurmstepd: error: *** JOB 3491040 ON c152 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***



⚠ Job 3491045 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491045/3491045_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491045/3491045_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:10,926) - Starting with JobEnvironment(job_id=3491045, hostname=c149, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:10,929) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491045/3491045_submitted.pkl
Trial-Index: 3
slurmstepd: error: *** JOB 3491045 ON c149 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***



⚠ Job 3491041 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491041/3491041_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491041/3491041_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:10,787) - Starting with JobEnvironment(job_id=3491041, hostname=c150, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:10,789) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491041/3491041_submitted.pkl
Trial-Index: 16
slurmstepd: error: *** JOB 3491041 ON c150 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***



⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.

Copy raw data to clipboard

Download »oo_errors.txt« as file

Copy raw data to clipboard

Download »outfile« as file

To cancel, press CTRL c, then run 'scancel 3489359'
⠋ Importing logging...
⠋ Importing warnings...
⠋ Importing argparse...
⠋ Importing datetime...
⠋ Importing dataclass...
⠋ Importing socket...
⠋ Importing stat...
⠋ Importing pwd...
⠋ Importing base64...
⠋ Importing json...
⠋ Importing yaml...
⠋ Importing toml...
⠋ Importing csv...
⠋ Importing ast...
⠋ Importing rich.table...
⠋ Importing rich print...
⠋ Importing rich.pretty...
⠋ Importing pformat...
⠋ Importing rich.prompt...
⠋ Importing types.FunctionType...
⠋ Importing typing...
⠋ Importing ThreadPoolExecutor...
⠋ Importing submitit.LocalExecutor...
⠋ Importing submitit.Job...
⠋ Importing importlib.util...
⠋ Importing platform...
⠋ Importing inspect frame info...
⠋ Importing pathlib.Path...
⠋ Importing uuid...
⠋ Importing cowsay...
⠋ Importing shutil...
⠋ Importing itertools.combinations...
⠋ Importing os.listdir...
⠋ Importing os.path...
⠋ Importing PIL.Image...
⠋ Importing sixel...
⠋ Importing subprocess...
⠋ Importing tqdm...
⠼ Importing beartype...
[WARNING 05-05 14:27:47] ax.storage.sqa_store.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
⠙ Importing rendering stuff...
⠋ Importing statistics...
⠋ Trying to import pyfiglet...
⠙ Importing helpers...
⠋ Importing pareto...
⠋ Parsing arguments...
⠋ Importing torch...
⠋ Importing numpy...
⠋ Importing ax...
⠋ Importing ax.core.generator_run...
⠋ Importing Cont_X_trans and Y_trans from ax.adapter.registry...
⠋ Importing ax.core.arm...
⠋ Importing ax.core.objective...
⠋ Importing ax.core.Metric...
⠋ Importing ax.exceptions.core...
⠋ Importing ax.exceptions.generation_strategy...
⠋ Importing CORE_DECODER_REGISTRY...
⠋ Trying ax.generation_strategy.generation_node...
⠋ Importing GenerationStep, GenerationStrategy from generation_strategy...
⠋ Importing GenerationNode from generation_node...
⠋ Importing ExternalGenerationNode...
⠋ Importing MinTrials...
⠋ Importing GeneratorSpec...
⠋ Importing Generators from ax.generation_strategy.registry...
⠋ Importing load_experiment...
⠋ Importing save_experiment...
⠋ Importing save_experiment_to_db...
⠋ Importing TrialStatus...
⠋ Importing Data...
⠋ Importing Experiment...
⠋ Importing parameter types...
⠋ Importing TParameterization...
⠋ Importing pandas...
⠋ Importing AxClient and ObjectiveProperties...
⠋ Importing RandomForestRegressor...
⠋ Importing botorch...
⠋ Importing submitit...
⠋ Importing ax logger...
⠋ Importing SQL-Storage-Stuff...
Run-UUID: 0d941129-a9ab-4a46-a911-66df26e23f94
 _____                 _ _____       _   _____ 
|  _  |               (_)  _  |     | | / __  \
| | | |_ __ ___  _ __  _| | | |_ __ | |_`' / /'
| | | | '_ ` _ \| '_ \| | | | | '_ \| __| / /  
\ \_/ / | | | | | | | | \ \_/ / |_) | |_./ /___
 \___/|_| |_| |_|_| |_|_|\___/| .__/ \__\_____/
                              | |              
                              |_|              

⠋ Writing worker creation log...
omniopt --partition=alpha --experiment_name=tinyLogicLLM --mem_gb=32 --time=7200 --worker_timeout=720 --max_eval=1000 --num_parallel_jobs=20 --gpus=1 --num_random_steps=20 --follow --live_share --send_anonymized_usage_stats --result_names 'VAL_LOSS=min PARAMS=min' --run_program='cHl0aG9uMyAvZGF0YS9ob3JzZS93cy9zMzgxMTE0MS1ncm9rL2xsbV9ncm9ra2luZ19sbHZtX3Rlc3QvdHJhaW4ucHkgLS1iYXRjaC1zaXplPSUoYmF0Y2hfc2l6ZSkgLS10b3BvIC0tdG9wby1ldmVyeSAxIC0ta2VscC1ldmVyeT0xIC0tbi1sYXllcnM9JShuX2xheWVycykgLS1kLW1vZGVsPSUoZF9tb2RlbCkgLS1uLWhlYWRzPSUobl9oZWFkcykgLS10b2tlbml6ZXJfaW5pdGlhbF9ucj0xMDAwIC0tbHI9JShscikgLS1iYXRjaGVzLXBlci1lcG9jaD0lKGJhdGNoZXNfcGVyX2Vwb2NoKSAtLXZhbC1iYXRjaGVzPSUodmFsX2JhdGNoZXMpIC0tZXBvY2hzPSUoZXBvY2hzKSAtLXRhc2s9dHVybnN0aWxlIC0tbWF4LXNlcS1sZW4gJShtYXhfc2VxdWVuY2VfbGVuZ3RoKSAtLW1heC10dXJuc3RpbGVzICUobWF4X3R1cm5zdGlsZXMpIC0tc2NoZWR1bGVyPWNvc2luZSAtLWZmbj0lKGZmbikK' --cpus_per_task=1 --nodes_per_job=1 --revert_to_random_when_seemingly_exhausted --model=BOTORCH_MODULAR --n_estimators_randomforest=100 --run_mode=local --show_sixel_general --show_sixel_trial_index_result --show_sixel_scatter --occ_type=euclid --main_process_gb=8 --nr_evals_per_arm=1 --max_nr_of_zero_results=50 --slurm_signal_delay_s=0 --max_failed_jobs=0 --max_attempts_for_generation=20 --num_restarts=20 --raw_samples=1024 --max_abandoned_retrial=20 --max_num_of_parallel_sruns=16 --number_of_generators=1 --generate_all_jobs_at_once --parameter batch_size range 16 512 int false --parameter ffn range 2 512 int false --parameter max_turnstiles range 5 64 int false --parameter epochs range 10 500 int false --parameter n_layers range 1 20 int false --parameter d_model range 2 8 int false --parameter max_sequence_length range 5 64 int false --parameter val_batches fixed 128 --parameter lr range 0.0000001 0.1 float false --parameter n_heads range 1 128 int false --parameter batches_per_epoch range 1 20 int false --experiment_constraints 'bWF4X3NlcXVlbmNlX2xlbmd0aCA+PSBtYXhfdHVybnN0aWxlcw=='
⠋ Disabling logging...
⠋ Setting run folder...
⠋ Creating folder /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0...
⠋ Writing revert_to_random_when_seemingly_exhausted file ...
⠋ Writing username state file...
⠋ Writing result names file...
⠋ Writing result min/max file...
⠋ Saving state files...
Run-folder: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0
⠋ Writing live_share file if it is present...
⠋ Writing job_start_time file...
⠋ Writing git information
⠋ Checking max_eval...
⠋ Calculating number of steps...
⠋ Adding excluded nodes...
⠋ Initializing ax_client...
⠋ Setting orchestrator...
See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=tinyLogicLLM&run_nr=0 for live-results.

You have 1 CPUs available for the main process. Using CUDA device NVIDIA H100.
Generation strategy: SOBOL for 20 steps and then BOTORCH_MODULAR for 980 steps.
Run-Program: python3 /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/train.py --batch-size=%(batch_size) --topo --topo-every 1 --kelp-every=1 --n-layers=%(n_layers) --d-model=%(d_model) --n-heads=%(n_heads) --tokenizer_initial_nr=1000 --lr=%(lr) --batches-per-epoch=%(batches_per_epoch) --val-batches=%(val_batches) --epochs=%(epochs) --task=turnstile --max-seq-len %(max_sequence_length) --max-turnstiles %(max_turnstiles) --scheduler=cosine --ffn=%(ffn)

Experiment parameters
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃ Name                ┃ Type  ┃ Lower bound ┃ Upper bound ┃ Values ┃ Type  ┃ Log Scale? ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│ batch_size          │ range │ 16          │ 512         │        │ int   │ No         │
│ ffn                 │ range │ 2           │ 512         │        │ int   │ No         │
│ max_turnstiles      │ range │ 5           │ 64          │        │ int   │ No         │
│ epochs              │ range │ 10          │ 500         │        │ int   │ No         │
│ n_layers            │ range │ 1           │ 20          │        │ int   │ No         │
│ d_model             │ range │ 2           │ 8           │        │ int   │ No         │
│ max_sequence_length │ range │ 5           │ 64          │        │ int   │ No         │
│ val_batches         │ fixed │             │             │ 128    │       │            │
│ lr                  │ range │ 1e-07       │ 0.1         │        │ float │ No         │
│ n_heads             │ range │ 1           │ 128         │        │ int   │ No         │
│ batches_per_epoch   │ range │ 1           │ 20          │        │ int   │ No         │
└─────────────────────┴───────┴─────────────┴─────────────┴────────┴───────┴────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Constraints                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ max_sequence_length >= max_turnstiles │
└───────────────────────────────────────┘
Result-Names
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Result-Name ┃ Min or max? ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ VAL_LOSS    │         min │
│ PARAMS      │         min │
└─────────────┴─────────────┘
⠋ Write files and show overview
SOBOL, running 20 = ∑20/20, waiting for 20 jobs                    :   0%|░░░░░░░░░░| 0/1000 [00:30
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 1 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), running 19 = ∑19/20, waiting for 19 jobs                :   0%|░░░░░░░░░░| 0/1000 [04:57
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 2 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 1 done, running 17 = ∑17/20, waiting for 17 jobs                           :   0%|░░░░░░░░░░| 1/1000 [51:40<859:13:39, 3096.32s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 3 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 8 done, timeout 9 = ∑9/20, waiting for 9 jobs                                      :   1%|░░░░░░░░░░| 8/1000 [12:02:04<1771:30:01, 6428.83s/it]
⚠ Job 3491048 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491048/3491048_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491048/3491048_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:20,638) - Starting with JobEnvironment(job_id=3491048, hostname=c149, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:20,640) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491048/3491048_submitted.pkl
Trial-Index: 11
slurmstepd: error: *** JOB 3491048 ON c149 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***


⚠ Job 3491036 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491036/3491036_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491036/3491036_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:01,228) - Starting with JobEnvironment(job_id=3491036, hostname=c153, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:01,230) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491036/3491036_submitted.pkl
Trial-Index: 10
slurmstepd: error: *** JOB 3491036 ON c153 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***


⚠ Job 3491039 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491039/3491039_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491039/3491039_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:11,021) - Starting with JobEnvironment(job_id=3491039, hostname=c152, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:11,023) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491039/3491039_submitted.pkl
Trial-Index: 6
slurmstepd: error: *** JOB 3491039 ON c152 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***


⚠ Job 3491040 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491040/3491040_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491040/3491040_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:11,020) - Starting with JobEnvironment(job_id=3491040, hostname=c152, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:11,022) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491040/3491040_submitted.pkl
Trial-Index: 5
slurmstepd: error: *** JOB 3491040 ON c152 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***


⚠ Job 3491037 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491037/3491037_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491037/3491037_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:11,021) - Starting with JobEnvironment(job_id=3491037, hostname=c152, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:11,023) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491037/3491037_submitted.pkl
Trial-Index: 8
slurmstepd: error: *** JOB 3491037 ON c152 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***


⚠ Job 3491047 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491047/3491047_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491047/3491047_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:20,638) - Starting with JobEnvironment(job_id=3491047, hostname=c149, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:20,640) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491047/3491047_submitted.pkl
Trial-Index: 2
slurmstepd: error: *** JOB 3491047 ON c149 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***


⚠ Job 3491038 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491038/3491038_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491038/3491038_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:11,021) - Starting with JobEnvironment(job_id=3491038, hostname=c152, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:11,031) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491038/3491038_submitted.pkl
Trial-Index: 1
slurmstepd: error: *** JOB 3491038 ON c152 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***


⚠ Job 3491045 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491045/3491045_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491045/3491045_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:10,926) - Starting with JobEnvironment(job_id=3491045, hostname=c149, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:10,929) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491045/3491045_submitted.pkl
Trial-Index: 3
slurmstepd: error: *** JOB 3491045 ON c149 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***


⚠ Job 3491041 (task: 0) with path /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491041/3491041_0_result.pkl
has not produced any output (state: TIMEOUT)
No error stream produced. Look at stdout: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491041/3491041_0_log.out
----------------------------------------
submitit INFO (2026-05-05 14:28:10,787) - Starting with JobEnvironment(job_id=3491041, hostname=c150, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2026-05-05 14:28:10,789) - Loading pickle: /data/horse/ws/s3811141-grok/llm_grokking_llvm_test/omniopt/runs/tinyLogicLLM/0/single_runs/3491041/3491041_submitted.pkl
Trial-Index: 16
slurmstepd: error: *** JOB 3491041 ON c150 CANCELLED AT 2026-05-06T02:28:26 DUE TO TIME LIMIT ***

SOBOL, failed: 12 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 8 done, running/pending 14/6 = ∑20/20, waiting for 20 jobs                        :   1%|░░░░░░░░░░| 8/1000 [12:48:17<1771:30:01, 6428.83s/it]
⚠ Raw data does not conform to the expected structure. Expected either a tuple of (mean, SEM) or a float, but got None.
SOBOL, failed: 13 ('VAL_LOSS: ' not found, 'PARAMS: ' not found (found ignoring case)), 9 done, running 18 = ∑18/20, new result: VAL_LOSS: 0.644831, PARAMS: 633984.000000:   1%|░░░░░░░░░░| 10/1000 [14:24:52<2025:08:59, 7364.18s/it]

Copy raw data to clipboard

Download »outfile« as file

Copy raw data to clipboard

Download »progressbar« as file

2026-05-05 14:27:53 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, Started OmniOpt2 run...
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #1/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #2/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #3/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #4/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #5/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #6/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #7/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #8/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #9/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #10/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #11/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #12/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #13/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #14/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #15/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #16/20
2026-05-05 14:27:54 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #17/20
2026-05-05 14:27:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #18/20
2026-05-05 14:27:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #19/20
2026-05-05 14:27:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, getting new HP set #20/20
2026-05-05 14:27:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, requested 20 jobs, got 20, 0.07 s/job
2026-05-05 14:27:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #1/20 start                     
2026-05-05 14:27:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #2/20 start                     
2026-05-05 14:27:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #3/20 start                     
2026-05-05 14:27:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #4/20 start                     
2026-05-05 14:27:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #5/20 start                     
2026-05-05 14:27:56 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #6/20 start                     
2026-05-05 14:27:56 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #7/20 start                     
2026-05-05 14:27:57 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #8/20 start                     
2026-05-05 14:27:57 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #9/20 start                     
2026-05-05 14:27:57 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #10/20 start                    
2026-05-05 14:27:57 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #11/20 start                    
2026-05-05 14:27:57 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #12/20 start                    
2026-05-05 14:27:57 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #13/20 start                    
2026-05-05 14:27:58 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #14/20 start                    
2026-05-05 14:27:58 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #15/20 start                    
2026-05-05 14:27:58 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #16/20 start                    
2026-05-05 14:27:58 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #17/20 start                    
2026-05-05 14:27:58 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #18/20 start                    
2026-05-05 14:27:58 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #19/20 start                    
2026-05-05 14:27:58 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, eval #20/20 start                    
2026-05-05 14:27:59 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, starting new job                     
2026-05-05 14:27:59 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, unknown 1 = ∑1/20, started new job   
2026-05-05 14:27:59 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, unknown 2 = ∑2/20, started new job   
2026-05-05 14:27:59 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, unknown 2 = ∑2/20, starting new job  
2026-05-05 14:28:00 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, unknown 1 = ∑1/20, starting new job  
2026-05-05 14:28:00 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, pending/unknown 2/1 = ∑3/20, started new job
2026-05-05 14:28:00 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, pending/unknown 2/1 = ∑3/20, starting new job
2026-05-05 14:28:00 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, pending/unknown 2/2 = ∑4/20, started new job 
2026-05-05 14:28:00 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, pending/unknown 2/2 = ∑4/20, starting new job
2026-05-05 14:28:00 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, pending/unknown 2/3 = ∑5/20, started new job 
2026-05-05 14:28:00 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, pending/unknown 2/4 = ∑6/20, started new job 
2026-05-05 14:28:00 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, pending/unknown 2/5 = ∑7/20, started new job 
2026-05-05 14:28:01 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, pending/unknown 2/6 = ∑8/20, started new job 
2026-05-05 14:28:01 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, pending/unknown 2/7 = ∑9/20, started new job 
2026-05-05 14:28:01 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/1 = ∑10/20, started new job
2026-05-05 14:28:01 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/2 = ∑11/20, started new job
2026-05-05 14:28:01 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/3 = ∑12/20, started new job
2026-05-05 14:28:02 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/4 = ∑13/20, started new job
2026-05-05 14:28:02 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/5 = ∑14/20, started new job
2026-05-05 14:28:02 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/6 = ∑15/20, started new job
2026-05-05 14:28:02 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/7 = ∑16/20, started new job
2026-05-05 14:28:02 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/8 = ∑17/20, started new job
2026-05-05 14:28:03 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/9 = ∑18/20, started new job
2026-05-05 14:28:03 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/10 = ∑19/20, started new job
2026-05-05 14:28:03 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/11 = ∑20/20, started new job
2026-05-05 14:28:04 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending/unknown 2/7/11 = ∑20/20, waiting for 20 jobs
2026-05-05 14:28:08 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending 2/18 = ∑20/20, waiting for 20 jobs          
2026-05-05 14:28:13 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running/pending 12/8 = ∑20/20, waiting for 20 jobs          
2026-05-05 14:28:24 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running 20 = ∑20/20, waiting for 20 jobs                    
2026-05-05 14:32:49 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, running 20 = ∑20/20, job_failed                             
2026-05-05 14:32:49 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 1 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 19 = ∑19/20, waiting for 20 jobs, finished 1 job
2026-05-05 14:32:49 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 1 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 19 = ∑19/20, waiting for 19 jobs                
2026-05-05 14:50:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 1 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 19 = ∑19/20, job_failed                         
2026-05-05 14:50:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 2 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 18 = ∑18/20, waiting for 19 jobs, finished 1 job
2026-05-05 14:50:55 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 2 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 18 = ∑18/20, waiting for 18 jobs                
2026-05-05 15:19:29 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 2 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), running 18 = ∑18/20, new result: VAL_LOSS: 0.618103, PARAMS: 1364582.000000
2026-05-05 15:19:32 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 2 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 1 done, running 17 = ∑17/20, waiting for 18 jobs, finished 1 job           
2026-05-05 15:19:32 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 2 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 1 done, running 17 = ∑17/20, waiting for 17 jobs                           
2026-05-05 15:44:43 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 2 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 1 done, running 17 = ∑17/20, job_failed                                    
2026-05-05 15:44:43 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 1 done, running 16 = ∑16/20, waiting for 17 jobs, finished 1 job           
2026-05-05 15:44:44 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 1 done, running 16 = ∑16/20, waiting for 16 jobs                           
2026-05-05 16:17:28 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 1 done, running 16 = ∑16/20, new result: VAL_LOSS: 0.117175, PARAMS: 367760.000000
2026-05-05 16:17:33 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, running 15 = ∑15/20, waiting for 16 jobs, finished 1 job                  
2026-05-05 16:17:33 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, running 15 = ∑15/20, waiting for 15 jobs                                  
2026-05-05 17:00:09 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 2 done, running 15 = ∑15/20, new result: VAL_LOSS: 0.692281, PARAMS: 2836159.000000
2026-05-05 17:00:14 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 3 done, running 14 = ∑14/20, waiting for 15 jobs, finished 1 job                   
2026-05-05 17:00:14 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 3 done, running 14 = ∑14/20, waiting for 14 jobs                                   
2026-05-05 17:41:42 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 3 done, running 14 = ∑14/20, new result: VAL_LOSS: 0.000026, PARAMS: 5711314.000000
2026-05-05 17:41:48 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 4 done, running 13 = ∑13/20, waiting for 14 jobs, finished 1 job                   
2026-05-05 17:41:48 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 4 done, running 13 = ∑13/20, waiting for 13 jobs                                   
2026-05-05 18:19:52 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 4 done, running 13 = ∑13/20, new result: VAL_LOSS: 0.000005, PARAMS: 1760444.000000
2026-05-05 18:19:58 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 5 done, running 12 = ∑12/20, waiting for 13 jobs, finished 1 job                   
2026-05-05 18:19:59 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 5 done, running 12 = ∑12/20, waiting for 12 jobs                                   
2026-05-05 18:29:35 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 5 done, running 12 = ∑12/20, new result: VAL_LOSS: 0.078159, PARAMS: 689934.000000 
2026-05-05 18:29:43 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 6 done, running 11 = ∑11/20, waiting for 12 jobs, finished 1 job                   
2026-05-05 18:29:43 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 6 done, running 11 = ∑11/20, waiting for 11 jobs                                   
2026-05-05 20:01:02 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 6 done, running 11 = ∑11/20, new result: VAL_LOSS: 0.000018, PARAMS: 66037.000000  
2026-05-05 20:01:10 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running 10 = ∑10/20, waiting for 11 jobs, finished 1 job                   
2026-05-05 20:01:11 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running 10 = ∑10/20, waiting for 10 jobs                                   
2026-05-05 23:47:31 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 7 done, running 10 = ∑10/20, new result: VAL_LOSS: 0.693279, PARAMS: 2298914.000000
2026-05-05 23:47:39 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running 9 = ∑9/20, waiting for 10 jobs, finished 1 job                     
2026-05-05 23:47:40 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running 9 = ∑9/20, waiting for 9 jobs                                      
2026-05-06 02:29:58 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, timeout 9 = ∑9/20, waiting for 9 jobs                                      
2026-05-06 02:30:16 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 3 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, timeout 9 = ∑9/20, job_failed                                              
2026-05-06 02:30:19 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, waiting for 9 jobs, finished 9 jobs                                       
2026-05-06 02:30:22 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #1/20                                                  
2026-05-06 02:30:22 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #2/20                                                  
2026-05-06 02:30:22 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #3/20                                                  
2026-05-06 02:30:22 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #4/20                                                  
2026-05-06 02:30:23 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #5/20                                                  
2026-05-06 02:30:23 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #6/20                                                  
2026-05-06 02:30:23 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #7/20                                                  
2026-05-06 02:30:24 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #8/20                                                  
2026-05-06 02:30:24 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #9/20                                                  
2026-05-06 02:30:24 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #10/20                                                 
2026-05-06 02:30:25 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #11/20                                                 
2026-05-06 02:30:25 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #12/20                                                 
2026-05-06 02:30:25 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #13/20                                                 
2026-05-06 02:30:25 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #14/20                                                 
2026-05-06 02:30:26 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #15/20                                                 
2026-05-06 02:30:26 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #16/20                                                 
2026-05-06 02:30:26 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #17/20                                                 
2026-05-06 02:30:27 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #18/20                                                 
2026-05-06 02:30:27 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #19/20                                                 
2026-05-06 02:30:27 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, getting new HP set #20/20                                                 
2026-05-06 02:30:28 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, requested 20 jobs, got 20, 0.34 s/job                                     
2026-05-06 02:30:29 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #1/20 start                                                          
2026-05-06 02:30:30 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #2/20 start                                                          
2026-05-06 02:30:31 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #3/20 start                                                          
2026-05-06 02:30:32 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #4/20 start                                                          
2026-05-06 02:30:34 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #5/20 start                                                          
2026-05-06 02:30:34 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #6/20 start                                                          
2026-05-06 02:30:36 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #7/20 start                                                          
2026-05-06 02:30:37 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #8/20 start                                                          
2026-05-06 02:30:38 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #9/20 start                                                          
2026-05-06 02:30:39 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #10/20 start                                                         
2026-05-06 02:30:41 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #11/20 start                                                         
2026-05-06 02:30:43 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #12/20 start                                                         
2026-05-06 02:30:44 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #13/20 start                                                         
2026-05-06 02:30:45 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #14/20 start                                                         
2026-05-06 02:30:45 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #15/20 start                                                         
2026-05-06 02:30:46 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #16/20 start                                                         
2026-05-06 02:30:47 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #17/20 start                                                         
2026-05-06 02:30:48 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #18/20 start                                                         
2026-05-06 02:30:49 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #19/20 start                                                         
2026-05-06 02:30:50 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, eval #20/20 start                                                         
2026-05-06 02:30:57 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, starting new job                                                          
2026-05-06 02:31:04 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, pending 13 = ∑13/20, started new job                                      
2026-05-06 02:31:05 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, pending 15 = ∑15/20, started new job                                      
2026-05-06 02:31:05 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, pending 13 = ∑13/20, started new job                                      
2026-05-06 02:31:05 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, pending 16 = ∑16/20, started new job                                      
2026-05-06 02:31:08 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, pending 16 = ∑16/20, starting new job                                     
2026-05-06 02:31:10 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, pending/unknown 16/4 = ∑20/20, started new job                            
2026-05-06 02:31:12 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, pending/unknown 16/4 = ∑20/20, waiting for 20 jobs                        
2026-05-06 02:31:16 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, pending 20 = ∑20/20, waiting for 20 jobs                                  
2026-05-06 03:16:11 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running/pending 14/6 = ∑20/20, waiting for 20 jobs                        
2026-05-06 03:17:29 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 12 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running/pending 14/6 = ∑20/20, job_failed                                 
2026-05-06 03:17:30 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 13 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running/pending 14/5 = ∑19/20, waiting for 20 jobs, finished 1 job        
2026-05-06 03:17:30 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 13 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running/pending 14/5 = ∑19/20, waiting for 19 jobs                        
2026-05-06 03:26:10 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 13 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running 19 = ∑19/20, waiting for 19 jobs                                  
2026-05-06 03:49:29 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 13 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 8 done, running 19 = ∑19/20, new result: VAL_LOSS: 0.022122, PARAMS: 626855.000000
2026-05-06 03:49:40 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 13 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 9 done, running 18 = ∑18/20, waiting for 19 jobs, finished 1 job                  
2026-05-06 03:49:40 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 13 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 9 done, running 18 = ∑18/20, waiting for 18 jobs                                  
2026-05-06 04:52:45 (cef43004-eb21-4c98-bf01-6617884ae783): SOBOL, failed: 13 ('VAL_LOSS: <FLOAT>' not found, 'PARAMS: <FLOAT>' not found (found ignoring case)), 9 done, running 18 = ∑18/20, new result: VAL_LOSS: 0.644831, PARAMS: 633984.000000

Copy raw data to clipboard

Download »progressbar« as file

Arguments Overview

Copy raw data to clipboard

Download »worker_usage.csv« as file

1777984070.4429,20,0,0
1777984079.3644,20,0,0
1777984079.7093,20,1,5
1777984080.1252,20,1,5
1777984080.2418,20,3,15
1777984083.6676,20,3,15
1777984084.7045,20,20,100
1777984373.5022,20,20,100
1777984374.051,20,19,95
1777985458.5987,20,19,95
1777985459.1479,20,18,90
1777987173.5505,20,18,90
1777987174.1189,20,17,85
1777988689.7052,20,17,85
1777988690.3019,20,16,80
1777990655.7213,20,16,80
1777990656.315,20,15,75
1777993216.9062,20,15,75
1777993217.5175,20,14,70
1777995706.2121,20,14,70
1777995708.3597,20,13,65
1777997996.3825,20,13,65
1777997998.4964,20,12,60
1777998576.1474,20,12,60
1777998579.9675,20,11,55
1778004062.902,20,11,55
1778004068.0621,20,10,50
1778017652.0069,20,10,50
1778017656.4698,20,9,45
1778027310.8593,20,9,45
1778027311.6435,20,7,35
1778027315.4325,20,7,35
1778027316.2156,20,0,0
1778027456.4672,20,0,0
1778027458.1431,20,1,5
1778027458.5496,20,2,10
1778027462.8003,20,2,10
1778027463.2315,20,16,80
1778027472.3907,20,16,80
1778027473.2322,20,20,100
1778030252.5669,20,20,100
1778030253.3689,20,19,95
1778032170.0762,20,19,95
1778032175.3294,20,18,90
1778035966.19,20,18,90
1778035970.0749,20,17,85
1778035970.3395,20,17,85

Copy raw data to clipboard

Download »worker_usage.csv« as file

Copy raw data to clipboard

Download »cpu_ram_usage.csv« as file

timestamp,ram_usage_mb,cpu_usage_percent
1777984070,794.33203125,7.9
1777984130,833.265625,7.1
1777984190,833.265625,7.1
1777984250,840.58203125,7.1
1777984310,840.58203125,7.1
1777984370,847.56640625,7.1
1777984430,847.58984375,7.2
1777984490,847.58984375,7.8
1777984550,847.5859375,9.4
1777984610,847.5859375,9.6
1777984670,847.59765625,9.8
1777984730,847.59765625,7.1
1777984790,847.62890625,7.1
1777984850,847.59765625,8.6
1777984910,847.59765625,8.6
1777984970,847.59765625,8.4
1777985030,847.6015625,7.2
1777985090,847.6015625,7.2
1777985150,847.59765625,5.6
1777985210,847.59765625,5.6
1777985270,847.609375,5.6
1777985330,847.61328125,5.6
1777985390,847.62109375,5.8
1777985450,847.6171875,7.2
1777985510,848.1328125,7.2
1777985570,848.12890625,7.2
1777985630,848.125,7.2
1777985690,848.12109375,6
1777985750,848.12890625,5.6
1777985810,848.12890625,5.6
1777985870,848.125,5.6
1777985930,848.12890625,5.8
1777985990,848.19921875,5.6
1777986050,848.12890625,5.6
1777986112,848.1328125,6
1777986173,848.20703125,7.2
1777986233,848.12890625,7.3
1777986293,848.12890625,7.2
1777986353,848.12890625,7.6
1777986429,848.12890625,6
1777986489,848.125,6
1777986549,848.12890625,6.2
1777986609,848.1328125,6.1
1777986669,848.125,6.3
1777986729,848.14453125,6.5
1777986789,848.12890625,6.4
1777986849,848.17578125,6.8
1777986909,848.17578125,7
1777986969,848.1328125,8.8
1777987029,848.1328125,8.7
1777987090,840.81640625,9
1777987150,840.83203125,8.9
1777987210,840.83984375,9.1
1777987270,840.83984375,10.2
1777987330,840.84375,9.1
1777987390,840.85546875,8.9
1777987450,840.87109375,10.5
1777987510,840.859375,10.5
1777987570,840.859375,10.6
1777987630,840.85546875,10.5
1777987690,840.86328125,10.4
1777987750,840.87109375,10.5
1777987810,840.86328125,10.4
1777987870,840.859375,10.6
1777987930,840.85546875,12
1777987990,840.859375,11.9
1777988050,840.96875,12
1777988110,840.84765625,12.1
1777988171,840.91015625,11.3
1777988231,840.8984375,10.6
1777988291,840.9296875,10.4
1777988351,840.98046875,10.6
1777988411,840.9921875,10.5
1777988471,840.9375,10.5
1777988531,840.921875,10.5
1777988591,840.9375,10.4
1777988651,840.8828125,11.9
1777988711,844.1484375,12.1
1777988771,844.1796875,12
1777988831,844.13671875,12
1777988891,844.13671875,11.8
1777988951,844.11328125,10.8
1777989011,844.21875,10.7
1777989071,844.15234375,10.7
1777989131,844.13671875,10.6
1777989191,844.14453125,10.7
1777989251,844.1328125,10.5
1777989311,844.2109375,11.9
1777989371,844.734375,12.2
1777989431,845.31640625,12.3
1777989491,866.890625,12.1
1777989551,867.45703125,11.8
1777989611,868.5234375,10.7
1777989671,870.59375,10.7
1777989731,870.64453125,10.8
1777989791,870.68359375,10.6
1777989851,870.75390625,10.7
1777989911,870.82421875,10.6
1777989971,870.87109375,10.7
1777990031,870.91796875,10.7
1777990091,870.99609375,11.4
1777990151,871.0390625,12.3
1777990211,871.140625,12.2
1777990271,871.19921875,12.2
1777990331,876.59765625,12
1777990391,876.6640625,10.8
1777990451,876.74609375,10.7
1777990511,876.80859375,10.6
1777990571,876.859375,10.7
1777990631,876.94140625,10.6
1777990691,877.7421875,12.3
1777990752,877.74609375,12.1
1777990812,877.73828125,12.1
1777990872,877.73046875,12.3
1777990932,877.73828125,12.2
1777990992,877.71484375,10.7
1777991052,877.73828125,10.7
1777991112,877.8046875,10.7
1777991172,877.87109375,10.8
1777991232,877.91796875,10.6
1777991292,878,10.7
1777991352,878.0390625,12.1
1777991412,878.09765625,12.3
1777991472,878.171875,12.2
1777991532,878.2265625,12.2
1777991592,878.30078125,11.7
1777991652,878.34375,10.9
1777991712,878.41796875,10.8
1777991772,878.49609375,10.7
1777991832,878.5390625,10.5
1777991892,878.63671875,10.6
1777991952,878.6796875,10.8
1777992012,878.734375,10.4
1777992072,878.80078125,10.4
1777992132,878.87890625,10.4
1777992192,878.921875,10.4
1777992252,879.00390625,11
1777992312,879.03515625,11.9
1777992372,879.10546875,12
1777992436,879.1953125,11
1777992496,879.24609375,10.6
1777992556,879.328125,9.3
1777992616,879.3828125,9.1
1777992676,879.44140625,9
1777992736,879.4765625,9
1777992796,879.57421875,9
1777992856,879.625,9
1777992916,879.67578125,9.1
1777992976,879.76171875,9
1777993036,879.828125,10.6
1777993097,879.87890625,10.6
1777993157,879.94140625,10.6
1777993217,881.4140625,10.7
1777993277,881.4140625,10.6
1777993337,881.4140625,9.1
1777993398,881.4140625,9
1777993458,881.41796875,9
1777993518,881.40234375,10.6
1777993578,881.4375,10.6
1777993638,881.4921875,10.6
1777993698,881.51953125,10.6
1777993758,881.6640625,10.6
1777993818,881.62890625,9.1
1777993878,881.7109375,9
1777993938,881.765625,9.1
1777993998,881.80078125,9
1777994058,883.86328125,9.9
1777994118,883.92578125,10.6
1777994178,884.015625,10.6
1777994238,884.1015625,10.6
1777994298,884.1484375,10.6
1777994358,884.22265625,10.6
1777994418,884.2890625,9
1777994478,884.36328125,9
1777994538,884.41796875,9.2
1777994598,884.453125,9.1
1777994658,884.54296875,9.2
1777994718,884.62890625,9.1
1777994778,884.68359375,9.2
1777994838,884.73046875,9.2
1777994898,884.79296875,8.9
1777994958,884.84765625,8.9
1777995018,884.921875,8.9
1777995078,884.98828125,8.9
1777995138,885.05078125,8.9
1777995198,885.15625,8.9
1777995258,885.1640625,8.9
1777995318,885.25390625,9.6
1777995378,885.296875,10.5
1777995438,885.3671875,10.5
1777995498,885.4140625,10.4
1777995558,885.47265625,10.2
1777995618,885.53125,9
1777995678,885.58984375,8.9
1777995738,886.05859375,9
1777995798,886.05859375,10.5
1777995858,886.05859375,10.5
1777995918,886.04296875,10.5
1777995978,886.0625,10.5
1777996038,886.07421875,10.5
1777996099,886.0859375,8.9
1777996159,886.12890625,9
1777996219,886.17578125,9
1777996279,886.24609375,8.9
1777996339,886.30078125,9
1777996399,886.390625,8.9
1777996459,886.4140625,8.9
1777996519,886.48828125,8.9
1777996579,886.5703125,9
1777996639,886.6328125,8.9
1777996699,886.69140625,9
1777996759,886.75,8.9
1777996819,886.80078125,9.1
1777996879,886.8671875,8.9
1777996939,886.91796875,9
1777996999,886.98828125,8.9
1777997059,887.06640625,10.5
1777997119,887.1015625,10.4
1777997179,887.1875,10.5
1777997239,887.25,10.5
1777997299,887.3046875,10.5
1777997359,887.359375,10.5
1777997419,887.41796875,9.1
1777997479,887.45703125,9
1777997539,887.5390625,8.9
1777997599,887.578125,8.9
1777997659,887.6640625,8.8
1777997719,887.6875,9
1777997779,887.76953125,8.9
1777997839,887.86328125,8.9
1777997899,887.90625,8.9
1777997959,887.95703125,10.5
1777998019,888.41015625,10.7
1777998079,888.41796875,10.5
1777998139,888.42578125,10.5
1777998199,888.4296875,9.6
1777998259,888.42578125,9.1
1777998319,888.41796875,9
1777998379,893.203125,9
1777998439,893.25390625,9
1777998499,893.296875,9
1777998559,893.34765625,9
1777998619,894.08984375,8.9
1777998679,894.0859375,9.2
1777998739,894.09765625,9.3
1777998799,894.09375,9.2
1777998859,894.0859375,9.2
1777998919,894.10546875,9.2
1777998979,894.109375,9.2
1777999039,894.140625,10.8
1777999099,894.2109375,10.8
1777999159,894.2578125,10.8
1777999219,894.30859375,10.8
1777999279,894.375,10.7
1777999339,894.421875,9.8
1777999399,894.48828125,10
1777999459,894.53515625,10
1777999519,894.58203125,10.1
1777999579,894.66796875,10.4
1777999639,894.7265625,10.2
1777999699,894.76171875,10.1
1777999759,894.90234375,10.3
1777999819,894.90234375,10.2
1777999879,894.95703125,10.2
1777999939,895.109375,10.3
1777999999,895.07421875,10.2
1778000059,895.1328125,10.3
1778000119,895.18359375,10.5
1778000179,895.2421875,10.4
1778000239,895.3046875,10.6
1778000299,895.36328125,10.4
1778000359,895.4140625,10.3
1778000419,895.48046875,10.4
1778000479,895.52734375,10.4
1778000539,895.56640625,11.9
1778000599,895.6328125,11.9
1778000659,895.7109375,11.9
1778000719,895.72265625,12
1778000779,895.828125,12
1778000839,895.8828125,10.3
1778000899,895.94921875,10.4
1778000959,895.99609375,10.5
1778001019,896.0390625,10.5
1778001079,896.109375,10.4
1778001139,896.16796875,10.4
1778001199,896.2265625,10.4
1778001259,896.29296875,10.4
1778001319,896.34765625,10.4
1778001379,896.421875,10.4
1778001439,896.46484375,10.4
1778001499,896.52734375,10.5
1778001559,896.609375,10.5
1778001619,896.65234375,10.5
1778001679,896.74609375,10.4
1778001739,896.7890625,10.5
1778001799,896.8203125,11.9
1778001859,896.87890625,12
1778001919,896.94140625,12
1778001979,897.015625,12.3
1778002039,897.08203125,11.2
1778002099,897.140625,10.7
1778002159,897.2109375,10.5
1778002219,897.2421875,10.7
1778002279,897.3359375,12.3
1778002339,897.3515625,12.2
1778002399,897.4453125,12.3
1778002459,897.4921875,12.2
1778002519,897.54296875,11
1778002579,897.56640625,10.7
1778002639,897.62890625,10.7
1778002699,897.6953125,12.2
1778002759,897.7578125,12.2
1778002819,897.796875,12.3
1778002879,897.86328125,12.2
1778002939,897.9375,12.3
1778002999,897.98828125,10.7
1778003059,898.03515625,10.5
1778003119,898.09765625,10.7
1778003179,898.16015625,10.6
1778003239,898.20703125,10.8
1778003299,898.26953125,10.9
1778003359,898.34375,10.9
1778003419,898.3828125,10.8
1778003479,898.4296875,10.8
1778003539,898.4921875,11
1778003599,898.5859375,10.8
1778003659,898.6640625,11
1778003719,898.69921875,10.5
1778003779,898.75390625,10.6
1778003839,898.82421875,10.5
1778003899,903.65625,10.6
1778003959,903.72265625,11.9
1778004019,903.765625,12.1
1778004079,905.0390625,12.3
1778004139,905.01953125,11.9
1778004199,905.01953125,10.4
1778004259,905.0078125,10.3
1778004319,905.01953125,10.4
1778004379,905.015625,10.2
1778004439,905.0078125,10.3
1778004499,905.0625,10.3
1778004559,905.1015625,10.5
1778004619,905.15234375,10.3
1778004679,905.23046875,10.2
1778004739,905.25,10.3
1778004799,905.3046875,10.3
1778004859,905.3828125,10.5
1778004919,905.421875,10.3
1778004980,905.46875,10.2
1778005040,905.53515625,10.3
1778005100,905.5703125,10.3
1778005160,905.640625,10.3
1778005220,905.703125,10.4
1778005280,905.765625,10.4
1778005340,905.8046875,10.3
1778005400,905.88671875,10.3
1778005460,905.90234375,10.3
1778005520,905.98046875,10.3
1778005580,906.0234375,12
1778005640,906.08203125,11.9
1778005700,906.14453125,11.9
1778005760,906.203125,11.9
1778005820,906.24609375,10.4
1778005880,906.3046875,10.3
1778005940,906.375,10.4
1778006000,906.40625,10.3
1778006060,906.46875,10.3
1778006120,906.54296875,10.9
1778006180,906.58984375,11.8
1778006240,906.63671875,13.9
1778006300,906.70703125,13.8
1778006360,906.75390625,13.7
1778006420,906.80078125,13.8
1778006480,906.8671875,13.9
1778006540,906.9609375,13.6
1778006600,906.96484375,13.7
1778006660,907.0390625,13.8
1778006720,907.09375,13.8
1778006780,907.15625,13.7
1778006840,907.19921875,13.8
1778006900,907.25390625,13.8
1778006960,907.328125,13.7
1778007020,907.3671875,13.7
1778007080,907.43359375,13.7
1778007140,907.484375,13.7
1778007200,907.55078125,13.8
1778007260,907.59375,13.6
1778007320,907.66015625,13.9
1778007380,907.734375,13.7
1778007440,907.76171875,13.8
1778007500,907.8203125,13.7
1778007560,907.87890625,13.7
1778007620,907.92578125,13.9
1778007680,907.98828125,13.7
1778007740,908.046875,13.8
1778007800,908.14453125,13.7
1778007860,908.16015625,14
1778007920,908.22265625,13.9
1778007980,908.30078125,13.8
1778008040,908.31640625,13.9
1778008100,908.37109375,13.8
1778008160,908.4453125,13.9
1778008220,908.49609375,13.8
1778008280,908.546875,13.8
1778008340,908.62890625,13.8
1778008400,908.6640625,13.9
1778008460,908.71875,13.8
1778008520,908.80078125,13.8
1778008580,908.83203125,15
1778008640,908.875,16
1778008700,908.94140625,15.9
1778008760,908.98828125,16.3
1778008820,909.08203125,16.4
1778008880,909.1015625,16.3
1778008940,909.16015625,16.4
1778009000,909.2265625,16.3
1778009060,909.26171875,16.5
1778009120,909.3203125,16.4
1778009180,909.4140625,16.4
1778009240,909.44140625,16.4
1778009300,909.4921875,16.4
1778009360,909.54296875,16.4
1778009420,909.59375,16.4
1778009480,909.65234375,16.4
1778009540,909.703125,16.3
1778009600,909.75390625,16.4
1778009660,909.8125,12.1
1778009720,909.87109375,13
1778009780,909.9296875,13
1778009840,909.9921875,13.1
1778009900,910.04296875,13.2
1778009960,910.09765625,13.2
1778010020,910.1484375,13.2
1778010080,910.203125,13.4
1778010140,910.2578125,13.5
1778010200,910.328125,13.4
1778010260,910.3671875,13.3
1778010320,910.42578125,13.1
1778010380,910.46875,13.3
1778010440,910.53125,13.4
1778010500,910.6015625,13.3
1778010560,910.6328125,13.2
1778010620,910.69140625,13.3
1778010680,910.7734375,13.6
1778010740,910.79296875,13.3
1778010800,910.859375,13.3
1778010860,910.91015625,13.4
1778010920,910.94921875,13.5
1778010980,911.01953125,13.4
1778011040,911.08203125,13.3
1778011100,911.1328125,13.2
1778011160,911.18359375,13.3
1778011220,911.25390625,13.3
1778011280,911.31640625,13.2
1778011340,911.33984375,13.4
1778011400,911.40234375,13.6
1778011460,911.47265625,13.7
1778011520,911.51953125,13.6
1778011580,911.5859375,13.7
1778011640,911.625,13.7
1778011700,911.66015625,13.7
1778011760,911.73828125,13.7
1778011820,911.78515625,13.5
1778011880,911.84375,13.6
1778011940,911.90625,13.5
1778012000,911.93359375,13.5
1778012060,911.98828125,13.5
1778012120,912.05078125,13.6
1778012180,912.12109375,13.7
1778012240,912.16796875,13.7
1778012300,912.2109375,13.7
1778012360,912.26171875,13.7
1778012420,912.3203125,13.7
1778012480,912.375,13.7
1778012540,912.42578125,13.7
1778012600,912.48828125,13.5
1778012660,912.5390625,13.7
1778012720,912.625,13.5
1778012780,912.65625,13.6
1778012840,912.6953125,13.6
1778012900,912.75,13.6
1778012960,912.81640625,13.8
1778013020,912.85546875,13.7
1778013080,912.9140625,13.7
1778013140,912.97265625,13.8
1778013200,913.015625,13.7
1778013260,913.06640625,13.7
1778013320,913.15234375,13.7
1778013380,913.17578125,13.6
1778013440,913.24609375,13.6
1778013500,913.2890625,13.6
1778013560,913.35546875,15.3
1778013621,913.38671875,16.5
1778013681,913.46484375,16.5
1778013741,913.49609375,16.8
1778013801,913.5625,16.9
1778013861,913.6171875,17.1
1778013921,913.65625,16.9
1778013981,913.7109375,16.9
1778014041,913.77734375,17
1778014101,913.82421875,16.9
1778014161,913.875,17.1
1778014221,913.97265625,16.8
1778014281,913.98046875,17
1778014341,914.03125,17
1778014401,914.11328125,17
1778014461,914.13671875,17
1778014521,914.1953125,16.9
1778014581,914.28125,17
1778014641,914.30078125,17
1778014701,914.35546875,16.6
1778014761,914.43359375,16.6
1778014821,914.4765625,16.5
1778014881,914.5234375,17.1
1778014941,914.5859375,16.9
1778015001,914.6328125,17
1778015061,914.703125,17.1
1778015121,914.734375,17
1778015181,914.78125,17.1
1778015241,914.8359375,16.9
1778015301,914.8984375,17
1778015361,914.96484375,17
1778015421,914.98828125,17
1778015481,915.0546875,16.9
1778015541,915.1328125,17
1778015601,915.15234375,17
1778015661,915.2109375,17
1778015721,915.26171875,16.9
1778015781,915.31640625,13.6
1778015841,915.375,13.8
1778015901,915.421875,14
1778015962,915.47265625,14
1778016022,915.53515625,13.9
1778016082,915.59765625,14
1778016142,915.6484375,13.8
1778016202,915.70703125,13.8
1778016262,915.765625,13.6
1778016322,915.796875,13.9
1778016382,915.84765625,13.8
1778016442,915.953125,13.7
1778016502,915.94921875,13.9
1778016562,916.0234375,13.8
1778016622,916.0546875,13.8
1778016682,916.10546875,13.8
1778016742,916.18359375,13.9
1778016802,921,13.7
1778016862,921.0390625,13.7
1778016922,921.1015625,13.8
1778016982,921.1484375,13.5
1778017042,921.19921875,13.5
1778017102,921.265625,13.7
1778017162,921.3125,13.6
1778017222,921.359375,13.6
1778017282,921.39453125,13.7
1778017342,921.484375,13.6
1778017402,921.53515625,13.7
1778017462,921.56640625,13.7
1778017522,921.640625,13.6
1778017582,921.6640625,13.6
1778017642,921.7265625,13.7
1778017702,922.1953125,13.6
1778017762,922.19921875,13.6
1778017822,922.19921875,13.8
1778017882,922.21875,13.6
1778017942,922.1953125,13.5
1778018002,922.203125,13.7
1778018062,922.203125,13.6
1778018122,922.234375,13.6
1778018182,922.25390625,13.7
1778018242,922.32421875,13.6
1778018302,922.3515625,13.6
1778018362,922.41796875,13.6
1778018422,922.4375,13.8
1778018482,922.48828125,13.5
1778018542,922.53515625,13.6
1778018602,922.59375,13.8
1778018662,922.62890625,13.7
1778018722,922.69140625,13.6
1778018782,922.7265625,13.7
1778018842,922.7734375,13.6
1778018902,922.8203125,13.6
1778018962,922.86328125,13.8
1778019022,922.93359375,13.5
1778019082,922.9765625,13.4
1778019142,923.02734375,13.7
1778019202,923.078125,13.6
1778019262,923.140625,13.6
1778019322,923.18359375,13.7
1778019382,923.22265625,13.6
1778019442,923.2734375,13.6
1778019502,923.33203125,13.8
1778019562,923.375,13.5
1778019622,923.421875,13.6
1778019682,923.4921875,13.8
1778019742,923.52734375,13.5
1778019802,923.57421875,13.7
1778019862,923.6328125,13.8
1778019922,923.6953125,13.6
1778019982,923.76171875,13.7
1778020042,923.80078125,13.7
1778020102,923.84375,13.7
1778020162,923.90234375,13.6
1778020222,923.9375,13.7
1778020282,924,16.4
1778020342,924.05078125,16.4
1778020402,924.11328125,16.3
1778020462,924.16015625,16.3
1778020522,924.22265625,16.4
1778020582,924.24609375,16.7
1778020642,924.30859375,16.6
1778020702,924.3359375,16.7
1778020762,924.4140625,16.7
1778020822,924.47265625,16.7
1778020882,924.515625,16.7
1778020942,924.546875,16.6
1778021002,924.59765625,16.7
1778021062,924.68359375,16.7
1778021122,924.69921875,16.7
1778021182,924.75,16.7
1778021242,924.8203125,16.9
1778021302,924.8671875,16.8
1778021362,924.90234375,16.8
1778021422,924.96875,16.8
1778021482,925.01953125,16.8
1778021542,925.0625,16.9
1778021602,925.109375,16.9
1778021662,925.15234375,16.8
1778021722,925.203125,16.8
1778021782,925.25,16.9
1778021842,925.34375,16.7
1778021902,925.359375,16.6
1778021962,925.4140625,16.4
1778022022,925.47265625,16.7
1778022082,925.53125,16.6
1778022142,925.5703125,16.8
1778022202,925.625,16.8
1778022262,925.66015625,17
1778022322,925.72265625,16.9
1778022382,925.7734375,16.8
1778022442,925.8046875,16.9
1778022502,925.890625,16.8
1778022562,925.9453125,16.9
1778022622,925.97265625,16.8
1778022682,926.01171875,16.8
1778022742,926.08203125,16.9
1778022802,926.12109375,16.9
1778022862,926.17578125,17
1778022922,926.2265625,16.8
1778022982,926.28515625,16.8
1778023042,926.3359375,16.9
1778023102,926.390625,16.9
1778023162,926.42578125,16.9
1778023222,926.46875,16.8
1778023282,926.52734375,16.9
1778023342,926.578125,17
1778023402,926.6171875,16.8
1778023462,926.6953125,13.5
1778023522,926.734375,13.6
1778023582,926.76953125,13.6
1778023642,926.828125,12.8
1778023702,926.87890625,14.7
1778023762,926.93359375,14.8
1778023822,926.97265625,14.6
1778023882,927.0234375,14.8
1778023942,927.0703125,14.7
1778024002,927.13671875,14.7
1778024062,927.1875,14.7
1778024122,927.23828125,14.7
1778024182,927.28515625,14.6
1778024242,927.33984375,14.8
1778024302,927.37890625,14.7
1778024362,927.4296875,14.8
1778024422,933.9140625,14.9
1778024482,933.9453125,14.9
1778024542,933.98828125,15.1
1778024602,934.03125,15
1778024662,934.09375,14.8
1778024722,934.16015625,14.9
1778024782,934.1953125,14.9
1778024842,934.2578125,14.9
1778024902,934.31640625,14.8
1778024962,934.3671875,15.1
1778025022,934.3828125,14.9
1778025082,934.4609375,15
1778025142,934.484375,14.9
1778025202,934.53125,14.9
1778025262,934.609375,15
1778025322,934.65625,14.8
1778025382,934.70703125,14.9
1778025442,934.74609375,14.9
1778025502,934.78515625,15.1
1778025562,934.8515625,14.9
1778025622,934.89453125,15
1778025682,934.97265625,14.9
1778025742,934.98046875,14.9
1778025802,935.01953125,15
1778025862,935.09375,14.9
1778025922,935.1484375,14.9
1778025982,935.1796875,14.9
1778026042,935.26171875,15.1
1778026102,935.28125,14.9
1778026162,935.33203125,14.9
1778026222,935.390625,14.9
1778026282,935.43359375,15.1
1778026342,935.48828125,14.3
1778026402,935.55078125,14.8
1778026462,935.6015625,14.8
1778026522,935.63671875,15
1778026582,935.67578125,15.1
1778026642,935.75,14.8
1778026702,935.79296875,15
1778026762,935.84375,15
1778026822,935.87890625,15
1778026882,935.94140625,14.8
1778026942,937.48828125,14.9
1778027002,938.09765625,15
1778027062,938.1484375,15
1778027122,938.1640625,15
1778027182,938.21484375,14.9
1778027242,938.29296875,15
1778027302,938.3203125,14.9
1778027362,938.41796875,15
1778027422,943.71484375,14.8
1778027482,954.421875,15.7
1778027542,954.46484375,15
1778027602,954.45703125,14.9
1778027662,954.48046875,15
1778027722,954.484375,15.1
1778027782,954.52734375,14.8
1778027842,954.5625,14.8
1778027902,954.60546875,15
1778027962,954.64453125,14.9
1778028022,954.70703125,14.6
1778028082,954.76171875,14.7
1778028142,954.80078125,14.6
1778028202,954.84375,14.8
1778028262,954.88671875,14.7
1778028322,954.9296875,14.6
1778028382,954.96875,14.6
1778028442,955.0078125,14.7
1778028502,955.0546875,14.7
1778028562,955.08203125,14.6
1778028622,955.10546875,14.7
1778028682,955.16015625,14.6
1778028742,955.2109375,14.9
1778028802,955.28515625,14.6
1778028862,955.296875,14.7
1778028922,955.375,14.7
1778028982,955.4296875,14.7
1778029042,955.43359375,14.5
1778029102,955.45703125,14.6
1778029162,955.50390625,17.8
1778029222,955.58203125,17.1
1778029282,955.6015625,16.2
1778029342,955.67578125,16.2
1778029402,955.73046875,16.2
1778029462,955.75390625,16.3
1778029522,955.77734375,19.6
1778029582,955.828125,19.5
1778029642,955.87109375,19.5
1778029702,955.8984375,19.6
1778029762,955.9765625,19.6
1778029822,956,19.8
1778029882,956.0546875,19.8
1778029942,956.109375,19.8
1778030002,956.12890625,19.8
1778030062,956.203125,19.8
1778030122,956.23046875,19.9
1778030182,956.28515625,19.9
1778030242,956.31640625,19.9
1778030302,957.09375,19.9
1778030362,957.0859375,19.8
1778030422,957.07421875,19.8
1778030482,957.08984375,19.8
1778030542,957.05859375,19.9
1778030602,957.0625,19.8
1778030662,957.08984375,19.9
1778030722,957.0625,19.9
1778030782,957.0625,19.5
1778030842,957.0625,19.6
1778030902,957.078125,19.7
1778030962,957.09765625,19.6
1778031022,957.09765625,19.6
1778031082,957.09765625,19.9
1778031142,957.0625,19.6
1778031202,957.1015625,19.6
1778031262,957.1328125,19.6
1778031322,957.15625,19.6
1778031382,957.1796875,19.7
1778031442,957.2421875,19.8
1778031502,957.27734375,19.9
1778031562,957.359375,19.9
1778031622,957.36328125,19.8
1778031682,957.43359375,19.9
1778031742,957.4453125,19.9
1778031802,957.49609375,19.9
1778031862,957.55859375,19.9
1778031922,957.6171875,19.9
1778031982,957.67578125,19.9
1778032042,957.6875,19.9
1778032102,957.7578125,19.8
1778032162,957.80859375,19.9
1778032222,960.87109375,19.8
1778032282,960.8671875,19.8
1778032342,960.83203125,19.9
1778032402,960.83203125,14.5
1778032462,960.8671875,13.7
1778032522,960.8359375,13.7
1778032582,960.85546875,13.7
1778032642,960.87109375,13.7
1778032702,960.85546875,13.7
1778032762,960.8671875,12.8
1778032822,960.83984375,7.6
1778032882,960.84375,7.6
1778032942,960.8671875,7.7
1778033002,960.84765625,7.6
1778033062,960.83984375,7.9
1778033122,960.87109375,7.9
1778033182,960.8671875,7.9
1778033242,960.921875,7.9
1778033302,960.98828125,7.9
1778033362,961.0078125,8.2
1778033422,961.04296875,8.2
1778033482,961.078125,8.2
1778033542,961.13671875,8.2
1778033602,961.19921875,8.2
1778033662,963.2109375,8.2
1778033722,963.2890625,8.2
1778033782,963.3359375,8.2
1778033842,963.33984375,8.2
1778033902,963.421875,8.2
1778033962,963.4609375,8.2
1778034022,963.46875,8.2
1778034082,963.515625,8.2
1778034142,963.58203125,8.2
1778034202,963.6015625,8.2
1778034262,963.671875,8.2
1778034322,963.71875,8.2
1778034382,963.76171875,8.2
1778034442,963.7890625,8.2
1778034502,963.8515625,8.5
1778034562,963.90625,8.4
1778034622,963.91796875,8.5
1778034682,963.99609375,8.4
1778034742,964.046875,8.4
1778034802,964.0859375,8.5
1778034862,964.12109375,8.5
1778034922,964.18359375,8.4
1778034982,964.19921875,8.6
1778035042,964.28125,8.6
1778035102,964.296875,8.6
1778035162,964.359375,8.7
1778035222,964.42578125,8.6
1778035282,964.46484375,8.6
1778035342,964.49609375,8.7
1778035402,964.55859375,8.6
1778035462,964.609375,8.7
1778035522,964.62890625,8.8
1778035582,964.671875,8.9
1778035642,964.7265625,8.9
1778035702,964.78125,8.9
1778035762,964.8125,8.8
1778035822,964.87890625,8.9
1778035882,964.9375,8.9
1778035942,965.01171875,8.8

Copy raw data to clipboard

Download »cpu_ram_usage.csv« as file

VAL_LOSS (goal: minimize)

Best value: 5.0E-6
Achieved at:
- run_time = 13894
- PARAMS = 1760444
- batch_size = 250
- ffn = 126
- max_turnstiles = 13
- epochs = 51
- n_layers = 7
- d_model = 3
- max_sequence_length = 37
- lr = 0.045398153297925
- n_heads = 110
- batches_per_epoch = 19
- val_batches = 128

PARAMS (goal: minimize)

Best value: 66037
Achieved at:
- run_time = 19954
- VAL_LOSS = 1.8E-5
- batch_size = 306
- ffn = 99
- max_turnstiles = 25
- epochs = 214
- n_layers = 1
- d_model = 4
- max_sequence_length = 31
- lr = 0.048467500817641
- n_heads = 52
- batches_per_epoch = 18
- val_batches = 128

Parameter statistics