/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683168/683168_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683169/683169_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683170/683170_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683171/683171_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683172/683172_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683173/683173_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683174/683174_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683175/683175_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683177/683177_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683179/683179_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683181/683181_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683183/683183_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683185/683185_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683188/683188_0_log.err not found
Job SlurmJob<job_id=683162, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683163, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683165, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683162, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683163, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683165, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683162/683162_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683163/683163_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683164/683164_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683165/683165_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683166/683166_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683167/683167_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683239/683239_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683241/683241_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683242/683242_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683243/683243_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683245/683245_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683246/683246_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683247/683247_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683248/683248_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683249/683249_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683250/683250_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683251/683251_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683252/683252_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683253/683253_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683254/683254_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683255/683255_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683256/683256_0_log.err not found
Job SlurmJob<job_id=683237, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683240, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683237, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683240, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683236/683236_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683237/683237_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683238/683238_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683240/683240_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683288/683288_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683290/683290_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683295/683295_0_log.err not found
Job SlurmJob<job_id=683287, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683289, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683291, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683287/683287_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683289/683289_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683291/683291_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683292/683292_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683293/683293_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683294/683294_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683296/683296_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683299/683299_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683300/683300_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683301/683301_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683302/683302_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683303/683303_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683304/683304_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683305/683305_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683306/683306_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683307/683307_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683308/683308_0_log.err not found
Job SlurmJob<job_id=683336, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683342, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683335/683335_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683336/683336_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683338/683338_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683339/683339_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683340/683340_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683341/683341_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683342/683342_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683343/683343_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683344/683344_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683345/683345_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683346/683346_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683347/683347_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683348/683348_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683349/683349_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683350/683350_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683351/683351_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683352/683352_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683353/683353_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683354/683354_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683355/683355_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683371/683371_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683372/683372_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683373/683373_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683374/683374_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683375/683375_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683376/683376_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683377/683377_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683378/683378_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683379/683379_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683380/683380_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683381/683381_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683382/683382_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683383/683383_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683384/683384_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683385/683385_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683386/683386_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683387/683387_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683388/683388_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683389/683389_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683390/683390_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683402/683402_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683403/683403_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683404/683404_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683405/683405_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683406/683406_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683407/683407_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683408/683408_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683409/683409_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683410/683410_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683411/683411_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683412/683412_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683413/683413_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683414/683414_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683415/683415_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683416/683416_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683417/683417_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683418/683418_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683419/683419_0_log.err not found
Job SlurmJob<job_id=683401, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683400/683400_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683401/683401_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683439/683439_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683440/683440_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683441/683441_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683442/683442_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683443/683443_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683444/683444_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683445/683445_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683446/683446_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683447/683447_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683448/683448_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683449/683449_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683450/683450_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683451/683451_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683453/683453_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683454/683454_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683455/683455_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683456/683456_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683457/683457_0_log.err not found
Job SlurmJob<job_id=683437, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683438, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683437/683437_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683438/683438_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683482/683482_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683483/683483_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683484/683484_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683485/683485_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683486/683486_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683487/683487_0_log.err not found
⚠ Job 683488 (task: 0) with path /home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683488/683488_0_result.pkl
has not produced any output (state: OUT_OF_MEMORY)
No error stream produced. Look at stdout: /home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683488/683488_0_log.out
----------------------------------------
submitit INFO (2025-04-10 17:13:08,837) - Starting with JobEnvironment(job_id=683488, hostname=i8035, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2025-04-10 17:13:08,837) - Loading pickle: /home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683488/683488_submitted.pkl
/var/spool/slurmd/job683488/slurm_script: line 17: 2994348 Killed /home/s3811141/.omniax_x86_64_Python_3.10.4_alpha/bin/python3 -u -m submitit.core._submit /home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/%j
slurmstepd: error: Detected 2 oom_kill events in StepId=683488.batch. Some of the step tasks have been OOM Killed.
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683488/683488_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683489/683489_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683490/683490_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683491/683491_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683492/683492_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683493/683493_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683494/683494_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683495/683495_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683496/683496_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683497/683497_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683498/683498_0_log.err not found
Job SlurmJob<job_id=683479, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
Job SlurmJob<job_id=683481, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683479/683479_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683480/683480_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683481/683481_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683519/683519_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683520/683520_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683521/683521_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683522/683522_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683523/683523_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683524/683524_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683525/683525_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683526/683526_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683527/683527_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683528/683528_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683529/683529_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683530/683530_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683531/683531_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683532/683532_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683533/683533_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683534/683534_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683535/683535_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683536/683536_0_log.err not found
Job SlurmJob<job_id=683518, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683517/683517_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683518/683518_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683561/683561_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683562/683562_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683563/683563_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683564/683564_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683565/683565_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683566/683566_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683567/683567_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683568/683568_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683569/683569_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683570/683570_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683571/683571_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683572/683572_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683573/683573_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683574/683574_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683575/683575_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683576/683576_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683577/683577_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683578/683578_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683579/683579_0_log.err not found
Job SlurmJob<job_id=683560, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683560/683560_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683622/683622_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683623/683623_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683624/683624_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683625/683625_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683626/683626_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683627/683627_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683628/683628_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683629/683629_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683630/683630_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683631/683631_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683632/683632_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683633/683633_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683634/683634_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683635/683635_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683636/683636_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683637/683637_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683638/683638_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683639/683639_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683640/683640_0_log.err not found
Job SlurmJob<job_id=683621, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683621/683621_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683668/683668_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683669/683669_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683670/683670_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683671/683671_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683672/683672_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683673/683673_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683674/683674_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683675/683675_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683676/683676_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683677/683677_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683678/683678_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683679/683679_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683680/683680_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683681/683681_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683682/683682_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683683/683683_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683684/683684_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683685/683685_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683686/683686_0_log.err not found
Job SlurmJob<job_id=683667, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683667/683667_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683716/683716_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683717/683717_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683718/683718_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683719/683719_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683720/683720_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683721/683721_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683722/683722_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683723/683723_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683724/683724_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683725/683725_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683726/683726_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683727/683727_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683728/683728_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683729/683729_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683730/683730_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683731/683731_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683732/683732_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683733/683733_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683734/683734_0_log.err not found
Job SlurmJob<job_id=683715, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683715/683715_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683873/683873_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683874/683874_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683875/683875_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683876/683876_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683877/683877_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683878/683878_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683879/683879_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683880/683880_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683881/683881_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683882/683882_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683883/683883_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683884/683884_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683885/683885_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683886/683886_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683887/683887_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683888/683888_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683889/683889_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683890/683890_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683891/683891_0_log.err not found
Job SlurmJob<job_id=683872, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683872/683872_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683998/683998_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683999/683999_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684000/684000_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684001/684001_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684002/684002_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684003/684003_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684004/684004_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684005/684005_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684006/684006_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684007/684007_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684008/684008_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684009/684009_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684010/684010_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684011/684011_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684012/684012_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684013/684013_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684014/684014_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684015/684015_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684016/684016_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684017/684017_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684145/684145_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684146/684146_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684147/684147_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684148/684148_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684149/684149_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684150/684150_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684151/684151_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684152/684152_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684153/684153_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684154/684154_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684155/684155_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684156/684156_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684157/684157_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684158/684158_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684159/684159_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684160/684160_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684161/684161_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684162/684162_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684163/684163_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684144/684144_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684166/684166_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684167/684167_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684168/684168_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684169/684169_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684170/684170_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684171/684171_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684172/684172_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684173/684173_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684174/684174_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684175/684175_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684176/684176_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684177/684177_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684178/684178_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684179/684179_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684180/684180_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684181/684181_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684182/684182_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684183/684183_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684184/684184_0_log.err not found
Job SlurmJob<job_id=684165, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684165/684165_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684187/684187_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684188/684188_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684189/684189_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684190/684190_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684191/684191_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684192/684192_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684193/684193_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684194/684194_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684195/684195_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684196/684196_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684197/684197_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684198/684198_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684199/684199_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684200/684200_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684201/684201_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684202/684202_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684203/684203_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684204/684204_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684205/684205_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684186/684186_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684254/684254_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684256/684256_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684262/684262_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684265/684265_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684268/684268_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684270/684270_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684272/684272_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684275/684275_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684280/684280_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684283/684283_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684286/684286_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684289/684289_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684291/684291_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684294/684294_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684299/684299_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684302/684302_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684305/684305_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684308/684308_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684310/684310_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684251/684251_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684418/684418_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684419/684419_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684420/684420_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684421/684421_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684422/684422_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684423/684423_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684424/684424_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684425/684425_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684426/684426_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684427/684427_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684428/684428_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684429/684429_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684430/684430_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684431/684431_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684432/684432_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684433/684433_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684434/684434_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684435/684435_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684436/684436_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684417/684417_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684529/684529_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684530/684530_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684532/684532_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684535/684535_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684536/684536_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684538/684538_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684540/684540_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684542/684542_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684544/684544_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684546/684546_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684548/684548_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684550/684550_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684552/684552_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684554/684554_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684556/684556_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684558/684558_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684560/684560_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684562/684562_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684564/684564_0_log.err not found
Job SlurmJob<job_id=684528, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684528/684528_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684655/684655_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684656/684656_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684657/684657_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684658/684658_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684659/684659_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684660/684660_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684661/684661_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684662/684662_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684663/684663_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684664/684664_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684665/684665_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684666/684666_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684667/684667_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684668/684668_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684669/684669_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684670/684670_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684671/684671_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684672/684672_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684673/684673_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684654/684654_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684778/684778_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684779/684779_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684780/684780_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684781/684781_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684782/684782_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684783/684783_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684784/684784_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684785/684785_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684786/684786_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684787/684787_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684788/684788_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684789/684789_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684790/684790_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684791/684791_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684792/684792_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684793/684793_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684794/684794_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684795/684795_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684796/684796_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684777/684777_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684911/684911_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684912/684912_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684913/684913_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684914/684914_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684915/684915_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684916/684916_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684917/684917_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684918/684918_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684919/684919_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684920/684920_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684921/684921_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684922/684922_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684923/684923_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684924/684924_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684925/684925_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684927/684927_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684928/684928_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684929/684929_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684930/684930_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684910/684910_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685019/685019_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685020/685020_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685021/685021_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685022/685022_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685023/685023_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685024/685024_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685025/685025_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685026/685026_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685027/685027_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685028/685028_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685029/685029_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685030/685030_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685033/685033_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685034/685034_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685035/685035_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685036/685036_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685037/685037_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685038/685038_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685039/685039_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685017/685017_0_log.err not found
Error computing Pareto frontier for LOSS and ACCURACY: `StandardizeY` transform requires non-empty data.
Error computing Pareto frontier for LOSS and TIME: `StandardizeY` transform requires non-empty data.
Error computing Pareto frontier for ACCURACY and TIME: `StandardizeY` transform requires non-empty data.
Best LOSS could not be determined
Current git-hash: b1529297cd78d2719bb86fb6f7b7ecfc4ffdd77c (version: 5634, 8b08442850db3d474b90a66583a4082d71988c57, 1 commit ago)
The current version was not thoroughly tested. It may contain bugs. Checkout to 8b08442850db3d474b90a66583a4082d71988c57 to get the latest tested version.Use --checkout_to_latest_tested_version to automatically use the latest tested version.
- Waiting for tail -f /home/h8/s3811141/test/randomtest_13424/omniopt/slurm-683157.out . To cancel, press CTRLc, then run 'scancel 683157'
Run-UUID: be5c2779-9113-4a8e-b5dd-5ed8643d861b
________________________________________
| OmniOpt2 - Dialing in the magic numbers. |
========================================
\
\
\
\
/ \\ //\\
|\\___/| / \\// \\\\
/0 0 \\__ / // | \\ \\
/ / \\/_/ // | \\ \\
\@_^_\@'/ \\/_ // | \\ \\
//_^_/ \\/_ // | \\ \\
( //) | \\/// | \\ \\
( / /) _|_ / ) // | \\ _\\
( // /) '/,_ _ _/ ( ; -. | _ _\\.-~ .-~~~^-.
(( / / )) ,-{ _ `-.|.-~-. .~ `.
(( // / )) '/\\ / ~-. _ .-~ .-~^-. \\
(( /// )) `. { } / \\ \\
(( / )) .----~-.\\ \\-' .~ \\ `. \\^-.
///.----..> \\ _ -~ `. ^-` ^-_
///-._ _ _ _ _ _ _}^ - - - - ~ ~-- ,.-~
/.-~
omniopt --partition=alpha --experiment_name=LLM_Example --mem_gb=128 --time=4320 --worker_timeout=240 --max_eval=500 --num_parallel_jobs=20 --gpus=1 --num_random_steps=20 --follow --live_share --send_anonymized_usage_stats --result_names LOSS=min ACCURACY=max TIME=min --run_program=YmFzaCAvaG9tZS9zMzgxMTE0MS90ZXN0L3JhbmRvbXRlc3RfMTM0MjQvb21uaW9wdC8udGVzdHMvZXhhbXBsZV9sbG0vcnVuIC0tZXBvY2hzICUoZXBvY2hzKSAtLWxlYXJuaW5nX3JhdGUgJShscikgLS1oaWRkZW5fc2l6ZSAlKGhpZGRlbl9zaXplKSAtLW5fbGF5ZXJzICUobmxheWVycykgLS1uX2hlYWRzICUobmhlYWRzKSAtLWRyb3BvdXQgJShkcm9wb3V0KSAtLW1heF9sZW4gJShtYXhsZW4pIC0tZGF0YXNldCB5ZWxwX3Jldmlld19mdWxs --cpus_per_task=1 --nodes_per_job=1 --generate_all_jobs_at_once --revert_to_random_when_seemingly_exhausted --model=BOTORCH_MODULAR --run_mode=local --decimalrounding=4 --occ_type=euclid --main_process_gb=8 --max_nr_of_zero_results=50 --pareto_front_confidence=1 --slurm_signal_delay_s=0 --n_estimators_randomforest=100 --parameter epochs range 10 100 int false --parameter lr range 1e-7 0.1 float false --parameter hidden_size range 1 10000 int false --parameter nlayers range 1 100 int false --parameter dropout range 0 0.8 float false --parameter maxlen range 1 10000 int false --parameter nheads range 0 10000 int false --ui_url aHR0cHM6Ly9pbWFnZXNlZy5zY2Fkcy5kZS9vbW5pYXgvZ3VpP3BhcnRpdGlvbj1hbHBoYSZleHBlcmltZW50bWU9TExNX0V4YW1wbGUmcmVzZXJ2YXRpb249JmFjY291bnQ9Jm1lbV9nYj0yNCZ0aW1lPTQzMjAmd29ya2VyX3RpbWVvdXQ9MjQwJm1heF9ldmFsPTUwMCZudW1fcGFyYWxsZWxfam9icz0yMCZncHVzPTEmbnVtX3JhbmRvbV9zdGVwcz0yMCZmb2xsb3c9MSZsaXZlX3NoYXJlPTEmc2VuZF9hbm9ueW1pemVkX3VzYWdlX3N0YXRzPTEmY2hlY2tvdXRfdG9fbGF0ZXN0X3Rlc3RlZF92ZXJzaW9uPTAmY29uc3RyYWludHM9JnJlc3VsdF9uYW1lcz1MT1NTJTNEbWluJTIwQUNDVVJBQ1klM0RtYXglMjBUSU1FJTNEbWluJnJ1bl9wcm9ncmFtPWJhc2glMjAlMkZob21lJTJGczM4MTExNDElMkZ0ZXN0JTJGcmFuZG9tdGVzdF8xMzQyNCUyRm9tbmlvcHQlMkYudGVzdHMlMkZleGFtcGxlX2xsbSUyRnJ1biUyMC0tZXBvY2hzJTIwJTI1KGVwb2NocyklMjAtLWxlYXJuaW5nX3JhdGUlMjAlMjUobHIpJTIwLS1oaWRkZW5fc2l6ZSUyMCUyNShoaWRkZW5fc2l6ZSklMjAtLW5fbGF5ZXJzJTIwJTI1KG5sYXllcnMpJTIwLS1uX2hlYWRzJTIwJTI1KG5oZWFkcyklMjAtLWRyb3BvdXQlMjAlMjUoZHJvcG91dCklMjAtLW1heF9sZW4lMjAlMjUobWF4bGVuKSUyMC0tZGF0YXNldCUyMHllbHBfcmV2aWV3X2Z1bGwmY3B1c19wZXJfdGFzaz0xJm5vZGVzX3Blcl9qb2I9MSZzZWVkPSZ2ZXJib3NlPTAmZ2VuZXJhdGVfYWxsX2pvYnNfYXRfb25jZT0xJmRlYnVnPTAmcmV2ZXJ0X3RvX3JhbmRvbV93aGVuX3NlZW1pbmdseV9leGhhdXN0ZWQ9MSZncmlkc2VhcmNoPTAmbW9kZWw9Qk9UT1JDSF9NT0RVTEFSJmluc3RhbGxhdGlvbl9tZXRob2Q9Y2xvbmUmcnVuX21vZGU9bG9jYWwmZGVjaW1hbHJvdW5kaW5nPTQmZGlzYWJsZV90cWRtPTAmdmVyYm9zZV90cWRtPTAmZm9yY2VfbG9jYWxfZXhlY3V0aW9uPTAmYXV0b19leGNsdWRlX2RlZmVjdGl2ZV9ob3N0cz0wJnNob3dfc2l4ZWxfZ2VuZXJhbD0wJnNob3dfc2l4ZWxfdHJpYWxfaW5kZXhfcmVzdWx0PTAmc2hvd19zaXhlbF9zY2F0dGVyPTAmc2hvd193b3JrZXJfcGVyY2VudGFnZV90YWJsZV9hdF9lbmQ9MCZlbmZvcmNlX3NlcXVlbnRpYWxfb3B0aW1pemF0aW9uPTAmb2NjPTAmb2NjX3R5cGU9ZXVjbGlkJm5vX3NsZWVwPTAmc2x1cm1fdXNlX3NydW49MCZ2ZXJib3NlX2JyZWFrX3J1bl9zZWFyY2hfdGFibGU9MCZhYmJyZXZpYXRlX2pvYl9uYW1lcz0wJm1haW5fcHJvY2Vzc19nYj04Jm1heF9ucl9vZl96ZXJvX3Jlc3VsdHM9NTAmcGFyZXRvX2Zyb250X2NvbmZpZGVuY2U9MSZzbHVybV9zaWduYWxfZGVsYXlfcz0wJmV4Y2x1ZGU9JmdlbmVyYXRpb25fc3RyYXRlZ3k9JnJvb3RfdmVudl9kaXI9JndvcmtkaXI9Jm5fZXN0aW1hdG9yc19yYW5kb21mb3Jlc3Q9MTAwJnBhcmFtZXRlcl8wbWU9ZXBvY2hzJnBhcmFtZXRlcl8wX3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzBfbWluPTEwJnBhcmFtZXRlcl8wX21heD0xMDAmcGFyYW1ldGVyXzBfbnVtYmVyX3R5cGU9aW50JnBhcmFtZXRlcl8wX2xvZ19zY2FsZT1mYWxzZSZwYXJhbWV0ZXJfMV9uYW1lPWxyJnBhcmFtZXRlcl8xX3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzFfbWluPTAuMDAwMDAwMSZwYXJhbWV0ZXJfMV9tYXg9MC4xJnBhcmFtZXRlcl8xX251bWJlcl90eXBlPWZsb2F0JnBhcmFtZXRlcl8xX2xvZ19zY2FsZT1mYWxzZSZwYXJhbWV0ZXJfMl9uYW1lPWhpZGRlbl9zaXplJnBhcmFtZXRlcl8yX3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzJfbWluPTEmcGFyYW1ldGVyXzJfbWF4PTEwMDAwJnBhcmFtZXRlcl8yX251bWJlcl90eXBlPWludCZwYXJhbWV0ZXJfMl9sb2dfc2NhbGU9ZmFsc2UmcGFyYW1ldGVyXzNfbmFtZT1ubGF5ZXJzJnBhcmFtZXRlcl8zX3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzNfbWluPTEmcGFyYW1ldGVyXzNfbWF4PTEwMCZwYXJhbWV0ZXJfM19udW1iZXJfdHlwZT1pbnQmcGFyYW1ldGVyXzNfbG9nX3NjYWxlPWZhbHNlJnBhcmFtZXRlcl80bWU9ZHJvcG91dCZwYXJhbWV0ZXJfNF90eXBlPXJhbmdlJnBhcmFtZXRlcl80X21pbj0wJnBhcmFtZXRlcl80X21heD0wLjgmcGFyYW1ldGVyXzRfbnVtYmVyX3R5cGU9ZmxvYXQmcGFyYW1ldGVyXzR
Run-folder: runs/LLM_Example/3
You have 2 CPUs available for the main process.Using CUDA device NVIDIA A100-SXM4-40GB.Generation strategy: SOBOL for 20 steps and then BOTORCH_MODULAR for 480 steps.
Run-Program: bash /home/s3811141/test/randomtest_13424/omniopt/.tests/example_llm/run --epochs %(epochs) --learning_rate %(lr) --hidden_size %(hidden_size) --n_layers %(nlayers) --n_heads %(nheads) --dropout %(dropout) --max_len %(maxlen) --dataset yelp_review_full
Experiment parameters:
┏━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃Name ┃Type ┃Lower bound┃Upper bound┃Type ┃Log Scale?┃
┡━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│epochs │range│10 │100 │int │No │
│lr │range│0 │0.1 │float│No │
│hidden_size│range│1 │10000 │int │No │
│nlayers │range│1 │100 │int │No │
│dropout │range│0 │0.8 │float│No │
│maxlen │range│1 │10000 │int │No │
│nheads │range│0 │10000 │int │No │
└─────────────┴───────┴─────────────┴─────────────┴───────┴────────────┘
Result-Names:
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃Result-Name┃Min or max?┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│LOSS │ min│
│ACCURACY │ max│
│TIME │ min│
└─────────────┴─────────────┘
See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=LLM_Example&run_nr=3 for live-results.
█▀▀▀▀▀█ ▀▄▄▀▀ ▀ ▄▀█▄▄▀ █ ▀ ▀ █▀▀▀▀▀█
█ ███ █ ▀ █▀ ▀ ▄ ▄██▄▄▀█▀█▀ █ ███ █
█ ▀▀▀ █ ▀▄▀▄█▄ █▄▀▀▄▀█▄▀▄ ▀ █ ▀▀▀ █
▀▀▀▀▀▀▀ █ █▄▀▄▀▄▀ ▀ ▀ █▄█▄▀ ▀ ▀▀▀▀▀▀▀
██ ▀█ ▀ ▀▀▀▄▄██ ▀▄ ▀ ▀▄▄▄██▀ ▀▄▄ ▄▀
▄▄█▀ ▄▀██ ██▄▀ █ █ █▀ ▀▄ ▀▀█▀▀▀ ▀█▀▀
▄▀█▀▀▀ ▀█▀▀▄ █▀▄▀▀ ██ ▄▄▄█ ▀ ▄█▄█▀
█ ▄ ▀▄█ █▀ ▄▀ █ ▀▄ ▀ █▀▄██▀ ▄▀█
▀▀█▄ ▀▀▀▄▀ ▄▄███ ▀▄██ ▄▀▄▄ ▄▀ █▀▀▀
▄▀▄ ▀▀▀ █▀▄█ ▀█▀▄█ ▀█▀▀▄█▀▄ █ ▄▄ █
▀███▀▀█▀ █ ▀ ▄▀▄▄█ █▄▀ █▄▀█▀██▀ █▀
▀█▀▄▄▄▀ ▀█▄ ▀▀█▀▄█▄ ▄ ██▄ ▄█▀▀▀█▀▄▄▄█
██▀▄ ▄▀█▄ ▀ ▄▀██▄█▄ ▀ ▄▀ ▀▄█▀▀▀▄█▄▄▀
█▀█▄█▄▀▄█▄ ▄▄█▀ ▀█▀ ▄ ▀█ ▀ █▄█▀█
▀ ▀▀ ▀▀▄ ▀█▀ ▄ ▀ ██▄▀▄ ▀▄ █▀▀▀█▀██
█▀▀▀▀▀█ █▄▀█▀▀ ▀█▀▀███ █ ▀ █▄▀ █
█ ███ █ ██▄▀▄█ █▀ ▄█▀ ▄ ▀█████▀█▄█▄
█ ▀▀▀ █ ▄▀▀█▄█ ▀▄█ ▄▀▄█ ▀█ ▀▄█▀▄█▄█
▀▀▀▀▀▀▀ ▀ ▀▀▀ ▀▀ ▀▀ ▀▀▀ ▀▀ ▀
SobolGenerator, running/pending/unknown 17/1/2 (100%/20), started new job : 0%|░░░░░░░░░░| 0/500 00:38<?, ?it/s] Y2tvdXRfdG9fbGF0ZXN0X3Rlc3RlZF92ZXJzaW9uPTAmY29uc3RyYWludHM9JnJlc3VsdF9uYW1lcz1MT1NTJTNEbWluJTIwQUNDVVJBQ1klM0RtYXglMjBUSU1FJTNEbWluJnJ1bl9wcm9ncmFtPWJhc2glMjAlMkZob21lJTJGczM4MTExNDElMkZ0ZXN0JTJGcmFuZG9tdGVzdF8xMzQyNCUyRm9tbmlvcHQlMkYudGVzdHMlMkZleGFtcGxlX2xsbSUyRnJ1biUyMC0tZXBvY2hzJTIwJTI1KGVwb2NocyklMjAtLWxlYXJuaW5nX3JhdGUlMjAlMjUobHIpJTIwLS1oaWRkZW5fc2l6ZSUyMCUyNShoaWRkZW5fc2l6ZSklMjAtLW5fbGF5ZXJzJTIwJTI1KG5sYXllcnMpJTIwLS1uX2hlYWRzJTIwJTI1KG5oZWFkcyklMjAtLWRyb3BvdXQlMjAlMjUoZHJvcG91dCklMjAtLW1heF9sZW4lMjAlMjUobWF4bGVuKSUyMC0tZGF0YXNldCUyMHllbHBfcmV2aWV3X2Z1bGwmY3B1c19wZXJfdGFzaz0xJm5vZGVzX3Blcl9qb2I9MSZzZWVkPSZ2ZXJib3NlPTAmZ2VuZXJhdGVfYWxsX2pvYnNfYXRfb25jZT0xJmRlYnVnPTAmcmV2ZXJ0X3RvX3JhbmRvbV93aGVuX3NlZW1pbmdseV9leGhhdXN0ZWQ9MSZncmlkc2VhcmNoPTAmbW9kZWw9Qk9UT1JDSF9NT0RVTEFSJmluc3RhbGxhdGlvbl9tZXRob2Q9Y2xvbmUmcnVuX21vZGU9bG9jYWwmZGVjaW1hbHJvdW5kaW5nPTQmZGlzYWJsZV90cWRtPTAmdmVyYm9zZV90cWRtPTAmZm9yY2VfbG9jYWxfZXhlY3V0aW9uPTAmYXV0b19leGNsdWRlX2RlZmVjdGl2ZV9ob3N0cz0wJnNob3dfc2l4ZWxfZ2VuZXJhbD0wJnNob3dfc2l4ZWxfdHJpYWxfaW5kZXhfcmVzdWx0PTAmc2hvd19zaXhlbF9zY2F0dGVyPTAmc2hvd193b3JrZXJfcGVyY2VudGFnZV90YWJsZV9hdF9lbmQ9MCZlbmZvcmNlX3NlcXVlbnRpYWxfb3B0aW1pemF0aW9uPTAmb2NjPTAmb2NjX3R5cGU9ZXVjbGlkJm5vX3NsZWVwPTAmc2x1cm1fdXNlX3NydW49MCZ2ZXJib3NlX2JyZWFrX3J1bl9zZWFyY2hfdGFibGU9MCZhYmJyZXZpYXRlX2pvYl9uYW1lcz0wJm1haW5fcHJvY2Vzc19nYj04Jm1heF9ucl9vZl96ZXJvX3Jlc3VsdHM9NTAmcGFyZXRvX2Zyb250X2NvbmZpZGVuY2U9MSZzbHVybV9zaWduYWxfZGVsYXlfcz0wJmV4Y2x1ZGU9JmdlbmVyYXRpb25fc3RyYXRlZ3k9JnJvb3RfdmVudl9kaXI9JndvcmtkaXI9Jm5fZXN0aW1hdG9yc19yYW5kb21mb3Jlc3Q9MTAwJnBhcmFtZXRlcl8wbWU9ZXBvY2hzJnBhcmFtZXRlcl8wX3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzBfbWluPTEwJnBhcmFtZXRlcl8wX21heD0xMDAmcGFyYW1ldGVyXzBfbnVtYmVyX3R5cGU9aW50JnBhcmFtZXRlcl8wX2xvZ19zY2FsZT1mYWxzZSZwYXJhbWV0ZXJfMV9uYW1lPWxyJnBhcmFtZXRlcl8xX3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzFfbWluPTAuMDAwMDAwMSZwYXJhbWV0ZXJfMV9tYXg9MC4xJnBhcmFtZXRlcl8xX251bWJlcl90eXBlPWZsb2F0JnBhcmFtZXRlcl8xX2xvZ19zY2FsZT1mYWxzZSZwYXJhbWV0ZXJfMl9uYW1lPWhpZGRlbl9zaXplJnBhcmFtZXRlcl8yX3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzJfbWluPTEmcGFyYW1ldGVyXzJfbWF4PTEwMDAwJnBhcmFtZXRlcl8yX251bWJlcl90eXBlPWludCZwYXJhbWV0ZXJfMl9sb2dfc2NhbGU9ZmFsc2UmcGFyYW1ldGVyXzNfbmFtZT1ubGF5ZXJzJnBhcmFtZXRlcl8zX3R5cGU9cmFuZ2UmcGFyYW1ldGVyXzNfbWluPTEmcGFyYW1ldGVyXzNfbWF4PTEwMCZwYXJhbWV0ZXJfM19udW1iZXJfdHlwZT1pbnQmcGFyYW1ldGVyXzNfbG9nX3NjYWxlPWZhbHNlJnBhcmFtZXRlcl80bWU9ZHJvcG91dCZwYXJhbWV0ZXJfNF90eXBlPXJhbmdlJnBhcmFtZXRlcl80X21pbj0wJnBhcmFtZXRlcl80X21heD0wLjgmcGFyYW1ldGVyXzRfbnVtYmVyX3R5cGU9ZmxvYXQmcGFyYW1ldGVyXzR
Run-folder: runs/LLM_Example/3
You have 2 CPUs available for the main process.Using CUDA device NVIDIA A100-SXM4-40GB.Generation strategy: SOBOL for 20 steps and then BOTORCH_MODULAR for 480 steps.
Run-Program: bash /home/s3811141/test/randomtest_13424/omniopt/.tests/example_llm/run --epochs %(epochs) --learning_rate %(lr) --hidden_size %(hidden_size) --n_layers %(nlayers) --n_heads %(nheads) --dropout %(dropout) --max_len %(maxlen) --dataset yelp_review_full
Experiment parameters:
┏━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃Name ┃Type ┃Lower bound┃Upper bound┃Type ┃Log Scale?┃
┡━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│epochs │range│10 │100 │int │No │
│lr │range│0 │0.1 │float│No │
│hidden_size│range│1 │10000 │int │No │
│nlayers │range│1 │100 │int │No │
│dropout │range│0 │0.8 │float│No │
│maxlen │range│1 │10000 │int │No │
│nheads │range│0 │10000 │int │No │
└─────────────┴───────┴─────────────┴─────────────┴───────┴────────────┘
Result-Names:
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃Result-Name┃Min or max?┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│LOSS │ min│
│ACCURACY │ max│
│TIME │ min│
└─────────────┴─────────────┘
See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=LLM_Example&run_nr=3 for live-results.
█▀▀▀▀▀█ ▀▄▄▀▀ ▀ ▄▀█▄▄▀ █ ▀ ▀ █▀▀▀▀▀█
█ ███ █ ▀ █▀ ▀ ▄ ▄██▄▄▀█▀█▀ █ ███ █
█ ▀▀▀ █ ▀▄▀▄█▄ █▄▀▀▄▀█▄▀▄ ▀ █ ▀▀▀ █
▀▀▀▀▀▀▀ █ █▄▀▄▀▄▀ ▀ ▀ █▄█▄▀ ▀ ▀▀▀▀▀▀▀
██ ▀█ ▀ ▀▀▀▄▄██ ▀▄ ▀ ▀▄▄▄██▀ ▀▄▄ ▄▀
▄▄█▀ ▄▀██ ██▄▀ █ █ █▀ ▀▄ ▀▀█▀▀▀ ▀█▀▀
▄▀█▀▀▀ ▀█▀▀▄ █▀▄▀▀ ██ ▄▄▄█ ▀ ▄█▄█▀
█ ▄ ▀▄█ █▀ ▄▀ █ ▀▄ ▀ █▀▄██▀ ▄▀█
▀▀█▄ ▀▀▀▄▀ ▄▄███ ▀▄██ ▄▀▄▄ ▄▀ █▀▀▀
▄▀▄ ▀▀▀ █▀▄█ ▀█▀▄█ ▀█▀▀▄█▀▄ █ ▄▄ █
▀███▀▀█▀ █ ▀ ▄▀▄▄█ █▄▀ █▄▀█▀██▀ █▀
▀█▀▄▄▄▀ ▀█▄ ▀▀█▀▄█▄ ▄ ██▄ ▄█▀▀▀█▀▄▄▄█
██▀▄ ▄▀█▄ ▀ ▄▀██▄█▄ ▀ ▄▀ ▀▄█▀▀▀▄█▄▄▀
█▀█▄█▄▀▄█▄ ▄▄█▀ ▀█▀ ▄ ▀█ ▀ █▄█▀█
▀ ▀▀ ▀▀▄ ▀█▀ ▄ ▀ ██▄▀▄ ▀▄ █▀▀▀█▀██
█▀▀▀▀▀█ █▄▀█▀▀ ▀█▀▀███ █ ▀ █▄▀ █
█ ███ █ ██▄▀▄█ █▀ ▄█▀ ▄ ▀█████▀█▄█▄
█ ▀▀▀ █ ▄▀▀█▄█ ▀▄█ ▄▀▄█ ▀█ ▀▄█▀▄█▄█
▀▀▀▀▀▀▀ ▀ ▀▀▀ ▀▀ ▀▀ ▀▀▀ ▀▀ ▀
SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), waiting for jobs (6 left), finished 6 jobs : 0%|░░░░░░░░░░| 0/500 03:48<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683168/683168_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683169/683169_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683170/683170_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683171/683171_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683172/683172_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683173/683173_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683174/683174_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683175/683175_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683177/683177_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683179/683179_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683181/683181_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683183/683183_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683185/683185_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683188/683188_0_log.err not foundJob SlurmJob<job_id=683162, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683163, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683165, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683162, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683163, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683165, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683162/683162_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683163/683163_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683164/683164_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683165/683165_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683166/683166_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683167/683167_0_log.err not found
SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), waiting for jobs (4 left), finished 4 jobs : 0%|░░░░░░░░░░| 0/500 07:58<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683239/683239_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683241/683241_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683242/683242_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683243/683243_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683245/683245_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683246/683246_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683247/683247_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683248/683248_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683249/683249_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683250/683250_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683251/683251_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683252/683252_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683253/683253_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683254/683254_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683255/683255_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683256/683256_0_log.err not foundJob SlurmJob<job_id=683237, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683240, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683237, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683240, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683236/683236_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683237/683237_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683238/683238_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683240/683240_0_log.err not found
SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (17), finished 17 jobs : 0%|░░░░░░░░░░| 0/500 12:26<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683288/683288_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683290/683290_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683295/683295_0_log.err not foundJob SlurmJob<job_id=683287, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683289, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683291, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683287/683287_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683289/683289_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683291/683291_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683292/683292_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683293/683293_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683294/683294_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683296/683296_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683299/683299_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683300/683300_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683301/683301_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683302/683302_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683303/683303_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683304/683304_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683305/683305_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683306/683306_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683307/683307_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683308/683308_0_log.err not found
SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (20), finished 20 jobs : 0%|░░░░░░░░░░| 0/500 18:04<?, ?it/s]Job SlurmJob<job_id=683336, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683342, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683335/683335_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683336/683336_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683338/683338_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683339/683339_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683340/683340_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683341/683341_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683342/683342_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683343/683343_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683344/683344_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683345/683345_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683346/683346_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683347/683347_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683348/683348_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683349/683349_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683350/683350_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683351/683351_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683352/683352_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683353/683353_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683354/683354_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683355/683355_0_log.err not found
SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (20), finished 20 jobs : 0%|░░░░░░░░░░| 0/500 24:21<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683371/683371_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683372/683372_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683373/683373_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683374/683374_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683375/683375_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683376/683376_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683377/683377_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683378/683378_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683379/683379_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683380/683380_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683381/683381_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683382/683382_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683383/683383_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683384/683384_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683385/683385_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683386/683386_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683387/683387_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683388/683388_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683389/683389_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683390/683390_0_log.err not found
SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (2), finished 2 jobs : 0%|░░░░░░░░░░| 0/500 31:01<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683402/683402_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683403/683403_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683404/683404_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683405/683405_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683406/683406_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683407/683407_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683408/683408_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683409/683409_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683410/683410_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683411/683411_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683412/683412_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683413/683413_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683414/683414_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683415/683415_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683416/683416_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683417/683417_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683418/683418_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683419/683419_0_log.err not foundJob SlurmJob<job_id=683401, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683400/683400_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683401/683401_0_log.err not found
SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (2), finished 2 jobs : 0%|░░░░░░░░░░| 0/500 38:37<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683439/683439_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683440/683440_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683441/683441_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683442/683442_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683443/683443_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683444/683444_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683445/683445_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683446/683446_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683447/683447_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683448/683448_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683449/683449_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683450/683450_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683451/683451_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683453/683453_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683454/683454_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683455/683455_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683456/683456_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683457/683457_0_log.err not foundJob SlurmJob<job_id=683437, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683438, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683437/683437_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683438/683438_0_log.err not found
SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (3), finished 3 jobs : 0%|░░░░░░░░░░| 0/500 47:18<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683482/683482_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683483/683483_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683484/683484_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683485/683485_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683486/683486_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683487/683487_0_log.err not found
⚠ Job 683488 (task: 0) with path /home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683488/683488_0_result.pkl
has not produced any output (state: OUT_OF_MEMORY)
No error stream produced. Look at stdout: /home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683488/683488_0_log.out
----------------------------------------
submitit INFO (2025-04-10 17:13:08,837) - Starting with JobEnvironment(job_id=683488, hostname=i8035, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2025-04-10 17:13:08,837) - Loading pickle: /home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683488/683488_submitted.pkl
/var/spool/slurmd/job683488/slurm_script: line 17: 2994348 Killed /home/s3811141/.omniax_x86_64_Python_3.10.4_alpha/bin/python3 -u -m submitit.core._submit /home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/%j
slurmstepd: error: Detected 2 oom_kill events in StepId=683488.batch. Some of the step tasks have been OOM Killed.
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683488/683488_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683489/683489_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683490/683490_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683491/683491_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683492/683492_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683493/683493_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683494/683494_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683495/683495_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683496/683496_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683497/683497_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683498/683498_0_log.err not foundJob SlurmJob<job_id=683479, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memoryJob SlurmJob<job_id=683481, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683479/683479_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683480/683480_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683481/683481_0_log.err not found
SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (2), finished 2 jobs : 0%|░░░░░░░░░░| 0/500 56:45<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683519/683519_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683520/683520_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683521/683521_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683522/683522_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683523/683523_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683524/683524_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683525/683525_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683526/683526_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683527/683527_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683528/683528_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683529/683529_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683530/683530_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683531/683531_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683532/683532_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683533/683533_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683534/683534_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683535/683535_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683536/683536_0_log.err not foundJob SlurmJob<job_id=683518, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683517/683517_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683518/683518_0_log.err not found
SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 1:07:09<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683561/683561_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683562/683562_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683563/683563_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683564/683564_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683565/683565_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683566/683566_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683567/683567_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683568/683568_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683569/683569_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683570/683570_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683571/683571_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683572/683572_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683573/683573_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683574/683574_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683575/683575_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683576/683576_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683577/683577_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683578/683578_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683579/683579_0_log.err not foundJob SlurmJob<job_id=683560, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683560/683560_0_log.err not found
SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 1:18:14<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683622/683622_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683623/683623_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683624/683624_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683625/683625_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683626/683626_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683627/683627_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683628/683628_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683629/683629_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683630/683630_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683631/683631_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683632/683632_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683633/683633_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683634/683634_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683635/683635_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683636/683636_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683637/683637_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683638/683638_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683639/683639_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683640/683640_0_log.err not foundJob SlurmJob<job_id=683621, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683621/683621_0_log.err not found
SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 1:30:15<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683668/683668_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683669/683669_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683670/683670_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683671/683671_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683672/683672_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683673/683673_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683674/683674_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683675/683675_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683676/683676_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683677/683677_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683678/683678_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683679/683679_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683680/683680_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683681/683681_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683682/683682_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683683/683683_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683684/683684_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683685/683685_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683686/683686_0_log.err not foundJob SlurmJob<job_id=683667, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683667/683667_0_log.err not found
SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 1:43:42<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683716/683716_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683717/683717_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683718/683718_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683719/683719_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683720/683720_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683721/683721_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683722/683722_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683723/683723_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683724/683724_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683725/683725_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683726/683726_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683727/683727_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683728/683728_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683729/683729_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683730/683730_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683731/683731_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683732/683732_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683733/683733_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683734/683734_0_log.err not foundJob SlurmJob<job_id=683715, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683715/683715_0_log.err not found
SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 2:33:21<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683873/683873_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683874/683874_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683875/683875_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683876/683876_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683877/683877_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683878/683878_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683879/683879_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683880/683880_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683881/683881_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683882/683882_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683883/683883_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683884/683884_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683885/683885_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683886/683886_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683887/683887_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683888/683888_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683889/683889_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683890/683890_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683891/683891_0_log.err not foundJob SlurmJob<job_id=683872, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683872/683872_0_log.err not found
SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), got 20, requested 20 : 0%|░░░░░░░░░░| 0/500 3:23:33<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683998/683998_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/683999/683999_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684000/684000_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684001/684001_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684002/684002_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684003/684003_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684004/684004_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684005/684005_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684006/684006_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684007/684007_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684008/684008_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684009/684009_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684010/684010_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684011/684011_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684012/684012_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684013/684013_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684014/684014_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684015/684015_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684016/684016_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684017/684017_0_log.err not found
SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 3:41:57<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684145/684145_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684146/684146_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684147/684147_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684148/684148_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684149/684149_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684150/684150_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684151/684151_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684152/684152_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684153/684153_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684154/684154_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684155/684155_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684156/684156_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684157/684157_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684158/684158_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684159/684159_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684160/684160_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684161/684161_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684162/684162_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684163/684163_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684144/684144_0_log.err not found
SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 3:59:08<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684166/684166_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684167/684167_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684168/684168_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684169/684169_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684170/684170_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684171/684171_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684172/684172_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684173/684173_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684174/684174_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684175/684175_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684176/684176_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684177/684177_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684178/684178_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684179/684179_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684180/684180_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684181/684181_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684182/684182_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684183/684183_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684184/684184_0_log.err not foundJob SlurmJob<job_id=684165, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684165/684165_0_log.err not found
SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 4:16:27<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684187/684187_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684188/684188_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684189/684189_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684190/684190_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684191/684191_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684192/684192_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684193/684193_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684194/684194_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684195/684195_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684196/684196_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684197/684197_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684198/684198_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684199/684199_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684200/684200_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684201/684201_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684202/684202_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684203/684203_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684204/684204_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684205/684205_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684186/684186_0_log.err not found
SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 4:34:48<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684254/684254_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684256/684256_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684262/684262_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684265/684265_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684268/684268_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684270/684270_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684272/684272_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684275/684275_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684280/684280_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684283/684283_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684286/684286_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684289/684289_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684291/684291_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684294/684294_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684299/684299_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684302/684302_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684305/684305_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684308/684308_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684310/684310_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684251/684251_0_log.err not found
SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 4:54:30<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684418/684418_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684419/684419_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684420/684420_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684421/684421_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684422/684422_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684423/684423_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684424/684424_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684425/684425_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684426/684426_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684427/684427_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684428/684428_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684429/684429_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684430/684430_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684431/684431_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684432/684432_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684433/684433_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684434/684434_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684435/684435_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684436/684436_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684417/684417_0_log.err not found
SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 5:15:35<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684529/684529_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684530/684530_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684532/684532_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684535/684535_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684536/684536_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684538/684538_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684540/684540_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684542/684542_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684544/684544_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684546/684546_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684548/684548_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684550/684550_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684552/684552_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684554/684554_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684556/684556_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684558/684558_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684560/684560_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684562/684562_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684564/684564_0_log.err not foundJob SlurmJob<job_id=684528, task_id=0, state="OUT_OF_MEMORY">, state not in 'early_stopped', 'abandoned', 'cancelled', 'timeout', 'interrupted'], which would be removed from the job list, or 'completed', 'unknown', 'pending', 'running', 'completing'], which would be ignored: out_of_memory
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684528/684528_0_log.err not found
SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 5:37:01<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684655/684655_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684656/684656_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684657/684657_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684658/684658_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684659/684659_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684660/684660_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684661/684661_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684662/684662_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684663/684663_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684664/684664_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684665/684665_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684666/684666_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684667/684667_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684668/684668_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684669/684669_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684670/684670_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684671/684671_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684672/684672_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684673/684673_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684654/684654_0_log.err not found
SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 6:00:06<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684778/684778_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684779/684779_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684780/684780_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684781/684781_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684782/684782_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684783/684783_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684784/684784_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684785/684785_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684786/684786_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684787/684787_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684788/684788_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684789/684789_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684790/684790_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684791/684791_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684792/684792_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684793/684793_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684794/684794_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684795/684795_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684796/684796_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684777/684777_0_log.err not found
SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 6:24:38<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684911/684911_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684912/684912_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684913/684913_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684914/684914_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684915/684915_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684916/684916_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684917/684917_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684918/684918_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684919/684919_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684920/684920_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684921/684921_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684922/684922_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684923/684923_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684924/684924_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684925/684925_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684927/684927_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684928/684928_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684929/684929_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684930/684930_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/684910/684910_0_log.err not found
SobolGenerator, Failed jobs: 500 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job : 0%|░░░░░░░░░░| 0/500 6:51:59<?, ?it/s]
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685019/685019_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685020/685020_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685021/685021_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685022/685022_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685023/685023_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685024/685024_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685025/685025_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685026/685026_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685027/685027_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685028/685028_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685029/685029_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685030/685030_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685033/685033_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685034/685034_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685035/685035_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685036/685036_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685037/685037_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685038/685038_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685039/685039_0_log.err not found
/home/h8/s3811141/test/randomtest_13424/omniopt/runs/LLM_Example/3/single_runs/685017/685017_0_log.err not foundError computing Pareto frontier for LOSS and ACCURACY: `StandardizeY` transform requires non-empty data.Error computing Pareto frontier for LOSS and TIME: `StandardizeY` transform requires non-empty data.Error computing Pareto frontier for ACCURACY and TIME: `StandardizeY` transform requires non-empty data.Best LOSS could not be determined Runtime Infos:
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃Number of values┃Min time ┃ Max time ┃Average time┃Median time┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ 155 │ 24.00 sec │ 101.00 sec │ 47.55 sec │ 44.00 sec │
└──────────────────┴───────────┴────────────┴──────────────┴─────────────┘
2025-04-10 16:34:00: SOBOL, Started OmniOpt2 run...
2025-04-10 16:34:03: SOBOL, trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 16:34:03: SOBOL, getting new hyperparameter set #1/20
2025-04-10 16:34:04: SobolGenerator, getting new hyperparameter set #2/20 | ETA: 5s
2025-04-10 16:34:04: SobolGenerator, getting new hyperparameter set #3/20 | ETA: 3s
2025-04-10 16:34:04: SobolGenerator, getting new hyperparameter set #4/20 | ETA: 3s
2025-04-10 16:34:04: SobolGenerator, getting new hyperparameter set #5/20 | ETA: 2s
2025-04-10 16:34:04: SobolGenerator, getting new hyperparameter set #6/20 | ETA: 2s
2025-04-10 16:34:04: SobolGenerator, getting new hyperparameter set #7/20 | ETA: 2s
2025-04-10 16:34:05: SobolGenerator, getting new hyperparameter set #8/20 | ETA: 2s
2025-04-10 16:34:05: SobolGenerator, getting new hyperparameter set #9/20 | ETA: 1s
2025-04-10 16:34:05: SobolGenerator, getting new hyperparameter set #10/20 | ETA: 1s
2025-04-10 16:34:05: SobolGenerator, getting new hyperparameter set #11/20 | ETA: 1s
2025-04-10 16:34:05: SobolGenerator, getting new hyperparameter set #12/20 | ETA: 1s
2025-04-10 16:34:05: SobolGenerator, getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 16:34:05: SobolGenerator, getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 16:34:06: SobolGenerator, getting new hyperparameter set #15/20 | ETA: 0s
2025-04-10 16:34:06: SobolGenerator, getting new hyperparameter set #16/20 | ETA: 0s
2025-04-10 16:34:06: SobolGenerator, getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 16:34:06: SobolGenerator, getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 16:34:06: SobolGenerator, getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 16:34:06: SobolGenerator, getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 16:34:07: SobolGenerator, starting parameter set
2025-04-10 16:34:07: SobolGenerator, starting new job
2025-04-10 16:34:08: SobolGenerator, unknown 1 (5%/20), started new job
2025-04-10 16:34:08: SobolGenerator, unknown 1 (5%/20), starting parameter set
2025-04-10 16:34:09: SobolGenerator, unknown 1 (5%/20), starting new job
2025-04-10 16:34:10: SobolGenerator, unknown 2 (10%/20), started new job
2025-04-10 16:34:10: SobolGenerator, unknown 2 (10%/20), starting parameter set
2025-04-10 16:34:10: SobolGenerator, unknown 2 (10%/20), starting new job
2025-04-10 16:34:11: SobolGenerator, pending/unknown 2/1 (15%/20), started new job
2025-04-10 16:34:11: SobolGenerator, pending/unknown 2/1 (15%/20), starting parameter set
2025-04-10 16:34:12: SobolGenerator, pending/unknown 2/1 (15%/20), starting new job
2025-04-10 16:34:13: SobolGenerator, pending/unknown 2/2 (20%/20), started new job
2025-04-10 16:34:13: SobolGenerator, pending/unknown 2/2 (20%/20), starting parameter set
2025-04-10 16:34:13: SobolGenerator, pending/unknown 2/2 (20%/20), starting new job
2025-04-10 16:34:15: SobolGenerator, pending/unknown 2/3 (25%/20), started new job
2025-04-10 16:34:15: SobolGenerator, pending/unknown 2/3 (25%/20), starting parameter set
2025-04-10 16:34:15: SobolGenerator, pending/unknown 2/3 (25%/20), starting new job
2025-04-10 16:34:16: SobolGenerator, pending/unknown 2/4 (30%/20), started new job
2025-04-10 16:34:16: SobolGenerator, pending/unknown 2/4 (30%/20), starting parameter set
2025-04-10 16:34:17: SobolGenerator, pending/unknown 2/4 (30%/20), starting new job
2025-04-10 16:34:18: SobolGenerator, pending/unknown 6/1 (35%/20), started new job
2025-04-10 16:34:18: SobolGenerator, pending/unknown 6/1 (35%/20), starting parameter set
2025-04-10 16:34:18: SobolGenerator, pending/unknown 6/1 (35%/20), starting new job
2025-04-10 16:34:19: SobolGenerator, pending/unknown 6/2 (40%/20), started new job
2025-04-10 16:34:19: SobolGenerator, pending/unknown 6/2 (40%/20), starting parameter set
2025-04-10 16:34:20: SobolGenerator, pending/unknown 6/2 (40%/20), starting new job
2025-04-10 16:34:22: SobolGenerator, pending 9 (45%/20), started new job
2025-04-10 16:34:22: SobolGenerator, pending 9 (45%/20), starting parameter set
2025-04-10 16:34:22: SobolGenerator, pending 9 (45%/20), starting new job
2025-04-10 16:34:23: SobolGenerator, pending/unknown 9/1 (50%/20), started new job
2025-04-10 16:34:23: SobolGenerator, pending/unknown 9/1 (50%/20), starting parameter set
2025-04-10 16:34:24: SobolGenerator, pending/unknown 9/1 (50%/20), starting new job
2025-04-10 16:34:25: SobolGenerator, running/pending/unknown 9/1/1 (55%/20), started new job
2025-04-10 16:34:25: SobolGenerator, running/pending/unknown 9/1/1 (55%/20), starting parameter set
2025-04-10 16:34:25: SobolGenerator, running/pending/unknown 9/1/1 (55%/20), starting new job
2025-04-10 16:34:26: SobolGenerator, running/pending/unknown 9/1/2 (60%/20), started new job
2025-04-10 16:34:26: SobolGenerator, running/pending/unknown 9/1/2 (60%/20), starting parameter set
2025-04-10 16:34:27: SobolGenerator, running/pending/unknown 9/1/2 (60%/20), starting new job
2025-04-10 16:34:28: SobolGenerator, running/pending/unknown 10/2/1 (65%/20), started new job
2025-04-10 16:34:28: SobolGenerator, running/pending/unknown 10/2/1 (65%/20), starting parameter set
2025-04-10 16:34:28: SobolGenerator, running/pending/unknown 10/2/1 (65%/20), starting new job
2025-04-10 16:34:29: SobolGenerator, running/pending/unknown 10/2/2 (70%/20), started new job
2025-04-10 16:34:29: SobolGenerator, running/pending/unknown 10/2/2 (70%/20), starting parameter set
2025-04-10 16:34:30: SobolGenerator, running/pending/unknown 10/2/2 (70%/20), starting new job
2025-04-10 16:34:31: SobolGenerator, running/pending/unknown 10/2/3 (75%/20), started new job
2025-04-10 16:34:31: SobolGenerator, running/pending/unknown 10/2/3 (75%/20), starting parameter set
2025-04-10 16:34:31: SobolGenerator, running/pending/unknown 10/2/3 (75%/20), starting new job
2025-04-10 16:34:32: SobolGenerator, running/pending/unknown 10/5/1 (80%/20), started new job
2025-04-10 16:34:32: SobolGenerator, running/pending/unknown 10/5/1 (80%/20), starting parameter set
2025-04-10 16:34:33: SobolGenerator, running/pending/unknown 10/5/1 (80%/20), starting new job
2025-04-10 16:34:34: SobolGenerator, running/pending/unknown 10/5/2 (85%/20), started new job
2025-04-10 16:34:34: SobolGenerator, running/pending/unknown 10/5/2 (85%/20), starting parameter set
2025-04-10 16:34:34: SobolGenerator, running/pending/unknown 10/5/2 (85%/20), starting new job
2025-04-10 16:34:35: SobolGenerator, running/pending/unknown 10/5/3 (90%/20), started new job
2025-04-10 16:34:35: SobolGenerator, running/pending/unknown 10/5/3 (90%/20), starting parameter set
2025-04-10 16:34:36: SobolGenerator, running/pending/unknown 10/5/3 (90%/20), starting new job
2025-04-10 16:34:37: SobolGenerator, running/pending/unknown 10/8/1 (95%/20), started new job
2025-04-10 16:34:37: SobolGenerator, running/pending/unknown 10/8/1 (95%/20), starting parameter set
2025-04-10 16:34:37: SobolGenerator, running/pending/unknown 10/8/1 (95%/20), starting new job
2025-04-10 16:34:38: SobolGenerator, running/pending/unknown 17/1/2 (100%/20), started new job
2025-04-10 16:34:47: SobolGenerator, running 20 (100%/20), got 20, requested 20
2025-04-10 16:34:51: SobolGenerator, running 20 (100%/20), waiting for old jobs to finish (20 left)
2025-04-10 16:34:59: SobolGenerator, running 20 (100%/20), waiting for old jobs to finish (20 left)
2025-04-10 16:35:06: SobolGenerator, running/completed 18/2 (100%/20), job_failed
2025-04-10 16:35:15: SobolGenerator, Failed jobs: 1 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed 18/1 (95%/20), job_failed
2025-04-10 16:35:24: SobolGenerator, Failed jobs: 2 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed 17/1 (90%/20), job_failed
2025-04-10 16:35:34: SobolGenerator, Failed jobs: 3 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 17 (85%/20), job_failed
2025-04-10 16:35:44: SobolGenerator, Failed jobs: 4 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 7/8/1 (80%/20), job_failed
2025-04-10 16:35:51: SobolGenerator, Failed jobs: 5 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 7/7/1 (75%/20), job_failed
2025-04-10 16:35:59: SobolGenerator, Failed jobs: 6 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/7 (70%/20), job_failed
2025-04-10 16:36:08: SobolGenerator, Failed jobs: 7 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/7 (65%/20), job_failed
2025-04-10 16:36:14: SobolGenerator, Failed jobs: 8 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/6 (60%/20), job_failed
2025-04-10 16:36:21: SobolGenerator, Failed jobs: 9 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/6 (55%/20), job_failed
2025-04-10 16:36:29: SobolGenerator, Failed jobs: 10 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/5 (50%/20), job_failed
2025-04-10 16:36:36: SobolGenerator, Failed jobs: 11 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/5 (45%/20), job_failed
2025-04-10 16:36:44: SobolGenerator, Failed jobs: 12 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/4 (40%/20), job_failed
2025-04-10 16:36:52: SobolGenerator, Failed jobs: 13 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/4 (35%/20), job_failed
2025-04-10 16:37:01: SobolGenerator, Failed jobs: 14 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/3 (30%/20), waiting for jobs (20 left), finished 14 jobs
2025-04-10 16:37:01: SobolGenerator, Failed jobs: 14 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/3 (30%/20), waiting for old jobs to finish (6 left)
2025-04-10 16:37:06: SobolGenerator, Failed jobs: 14 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/3 (30%/20), job_failed
2025-04-10 16:37:14: SobolGenerator, Failed jobs: 15 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/3 (25%/20), job_failed
2025-04-10 16:37:20: SobolGenerator, Failed jobs: 16 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/1 (20%/20), job_failed
2025-04-10 16:37:28: SobolGenerator, Failed jobs: 17 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 1/2 (15%/20), job_failed
2025-04-10 16:37:35: SobolGenerator, Failed jobs: 18 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 16:37:41: SobolGenerator, Failed jobs: 19 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 16:37:48: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), waiting for jobs (6 left), finished 6 jobs
2025-04-10 16:37:48: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 16:37:48: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 16:37:49: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 3s
2025-04-10 16:37:49: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 2s
2025-04-10 16:37:49: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 2s
2025-04-10 16:37:49: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 2s
2025-04-10 16:37:49: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 2s
2025-04-10 16:37:49: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 2s
2025-04-10 16:37:49: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 1s
2025-04-10 16:37:50: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 1s
2025-04-10 16:37:50: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 1s
2025-04-10 16:37:50: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 1s
2025-04-10 16:37:50: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 1s
2025-04-10 16:37:50: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 16:37:51: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 16:37:51: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 0s
2025-04-10 16:37:51: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 0s
2025-04-10 16:37:51: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 16:37:51: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 16:37:51: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 16:37:52: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 16:37:52: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 16:37:52: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 16:37:53: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 16:37:53: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 16:37:54: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 16:37:55: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 16:37:55: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 16:37:55: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 16:37:56: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), started new job
2025-04-10 16:37:57: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), starting parameter set
2025-04-10 16:37:57: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), starting new job
2025-04-10 16:37:58: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 4 (20%/20), started new job
2025-04-10 16:37:58: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 4 (20%/20), starting parameter set
2025-04-10 16:37:58: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 4 (20%/20), starting new job
2025-04-10 16:37:59: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/1 (25%/20), started new job
2025-04-10 16:37:59: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/1 (25%/20), starting parameter set
2025-04-10 16:38:00: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/1 (25%/20), starting new job
2025-04-10 16:38:01: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/2 (30%/20), started new job
2025-04-10 16:38:01: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/2 (30%/20), starting parameter set
2025-04-10 16:38:01: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/2 (30%/20), starting new job
2025-04-10 16:38:02: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/3 (35%/20), started new job
2025-04-10 16:38:02: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/3 (35%/20), starting parameter set
2025-04-10 16:38:03: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/3 (35%/20), starting new job
2025-04-10 16:38:04: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 7/1 (40%/20), started new job
2025-04-10 16:38:04: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 7/1 (40%/20), starting parameter set
2025-04-10 16:38:04: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 7/1 (40%/20), starting new job
2025-04-10 16:38:05: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/1 (45%/20), started new job
2025-04-10 16:38:05: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/1 (45%/20), starting parameter set
2025-04-10 16:38:06: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/1 (45%/20), starting new job
2025-04-10 16:38:07: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/2 (50%/20), started new job
2025-04-10 16:38:07: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/2 (50%/20), starting parameter set
2025-04-10 16:38:07: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/2 (50%/20), starting new job
2025-04-10 16:38:08: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/3/1 (55%/20), started new job
2025-04-10 16:38:08: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/3/1 (55%/20), starting parameter set
2025-04-10 16:38:09: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/3/1 (55%/20), starting new job
2025-04-10 16:38:11: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), started new job
2025-04-10 16:38:11: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), starting parameter set
2025-04-10 16:38:11: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), starting new job
2025-04-10 16:38:12: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/2 (65%/20), started new job
2025-04-10 16:38:12: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/2 (65%/20), starting parameter set
2025-04-10 16:38:13: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/2 (65%/20), starting new job
2025-04-10 16:38:14: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/1 (70%/20), started new job
2025-04-10 16:38:14: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/1 (70%/20), starting parameter set
2025-04-10 16:38:14: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/1 (70%/20), starting new job
2025-04-10 16:38:15: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/2 (75%/20), started new job
2025-04-10 16:38:16: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/2 (75%/20), starting parameter set
2025-04-10 16:38:16: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/2 (75%/20), starting new job
2025-04-10 16:38:17: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/3 (80%/20), started new job
2025-04-10 16:38:17: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/3 (80%/20), starting parameter set
2025-04-10 16:38:17: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/3 (80%/20), starting new job
2025-04-10 16:38:18: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/4 (85%/20), started new job
2025-04-10 16:38:18: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/4 (85%/20), starting parameter set
2025-04-10 16:38:19: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/3/4 (85%/20), starting new job
2025-04-10 16:38:20: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/1 (90%/20), started new job
2025-04-10 16:38:20: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/1 (90%/20), starting parameter set
2025-04-10 16:38:20: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/1 (90%/20), starting new job
2025-04-10 16:38:21: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/2 (95%/20), started new job
2025-04-10 16:38:21: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/2 (95%/20), starting parameter set
2025-04-10 16:38:22: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/2 (95%/20), starting new job
2025-04-10 16:38:23: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/3 (100%/20), started new job
2025-04-10 16:38:38: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 20 (100%/20), got 20, requested 20
2025-04-10 16:38:45: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 20 (100%/20), waiting for old jobs to finish (20 left)
2025-04-10 16:38:51: SobolGenerator, Failed jobs: 20 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 20 (100%/20), job_failed
2025-04-10 16:39:03: SobolGenerator, Failed jobs: 21 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 19 (95%/20), job_failed
2025-04-10 16:39:11: SobolGenerator, Failed jobs: 22 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 7/5/6 (90%/20), job_failed
2025-04-10 16:39:20: SobolGenerator, Failed jobs: 23 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 7/4/6 (85%/20), job_failed
2025-04-10 16:39:29: SobolGenerator, Failed jobs: 24 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 7/3/6 (80%/20), job_failed
2025-04-10 16:39:37: SobolGenerator, Failed jobs: 25 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 6/3/6 (75%/20), job_failed
2025-04-10 16:39:46: SobolGenerator, Failed jobs: 26 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 6/2/6 (70%/20), job_failed
2025-04-10 16:39:54: SobolGenerator, Failed jobs: 27 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 6/2/5 (65%/20), job_failed
2025-04-10 16:40:03: SobolGenerator, Failed jobs: 28 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/6 (60%/20), job_failed
2025-04-10 16:40:11: SobolGenerator, Failed jobs: 29 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/5 (55%/20), job_failed
2025-04-10 16:40:21: SobolGenerator, Failed jobs: 30 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/5 (50%/20), job_failed
2025-04-10 16:40:29: SobolGenerator, Failed jobs: 31 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/5 (45%/20), job_failed
2025-04-10 16:40:39: SobolGenerator, Failed jobs: 32 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/5 (40%/20), job_failed
2025-04-10 16:40:47: SobolGenerator, Failed jobs: 33 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/4 (35%/20), job_failed
2025-04-10 16:40:56: SobolGenerator, Failed jobs: 34 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/3 (30%/20), job_failed
2025-04-10 16:41:05: SobolGenerator, Failed jobs: 35 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/2 (25%/20), job_failed
2025-04-10 16:41:15: SobolGenerator, Failed jobs: 36 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/2 (20%/20), waiting for jobs (20 left), finished 16 jobs
2025-04-10 16:41:15: SobolGenerator, Failed jobs: 36 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/2 (20%/20), waiting for old jobs to finish (4 left)
2025-04-10 16:41:21: SobolGenerator, Failed jobs: 36 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/2 (20%/20), job_failed
2025-04-10 16:41:32: SobolGenerator, Failed jobs: 37 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), job_failed
2025-04-10 16:41:41: SobolGenerator, Failed jobs: 38 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/1 (10%/20), job_failed
2025-04-10 16:41:50: SobolGenerator, Failed jobs: 39 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 16:41:59: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), waiting for jobs (4 left), finished 4 jobs
2025-04-10 16:41:59: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 16:41:59: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 16:41:59: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 3s
2025-04-10 16:42:00: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 2s
2025-04-10 16:42:00: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 2s
2025-04-10 16:42:00: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 2s
2025-04-10 16:42:00: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 2s
2025-04-10 16:42:00: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 2s
2025-04-10 16:42:00: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 2s
2025-04-10 16:42:01: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 1s
2025-04-10 16:42:01: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 1s
2025-04-10 16:42:01: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 1s
2025-04-10 16:42:01: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 1s
2025-04-10 16:42:01: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 16:42:02: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 16:42:02: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 0s
2025-04-10 16:42:02: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 0s
2025-04-10 16:42:02: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 16:42:02: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 16:42:03: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 16:42:03: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 16:42:03: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 16:42:03: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 16:42:05: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 16:42:05: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 16:42:05: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 16:42:06: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), started new job
2025-04-10 16:42:06: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting parameter set
2025-04-10 16:42:06: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting new job
2025-04-10 16:42:08: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/1 (15%/20), started new job
2025-04-10 16:42:08: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/1 (15%/20), starting parameter set
2025-04-10 16:42:08: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/1 (15%/20), starting new job
2025-04-10 16:42:09: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/2 (20%/20), started new job
2025-04-10 16:42:09: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/2 (20%/20), starting parameter set
2025-04-10 16:42:09: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/2 (20%/20), starting new job
2025-04-10 16:42:11: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/1 (25%/20), started new job
2025-04-10 16:42:11: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/1 (25%/20), starting parameter set
2025-04-10 16:42:11: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/1 (25%/20), starting new job
2025-04-10 16:42:12: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/2 (30%/20), started new job
2025-04-10 16:42:12: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/2 (30%/20), starting parameter set
2025-04-10 16:42:13: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/2 (30%/20), starting new job
2025-04-10 16:42:14: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/3 (35%/20), started new job
2025-04-10 16:42:14: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/3 (35%/20), starting parameter set
2025-04-10 16:42:14: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/3 (35%/20), starting new job
2025-04-10 16:42:15: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/5/1 (40%/20), started new job
2025-04-10 16:42:15: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/5/1 (40%/20), starting parameter set
2025-04-10 16:42:16: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/5/1 (40%/20), starting new job
2025-04-10 16:42:17: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/5/2 (45%/20), started new job
2025-04-10 16:42:17: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/5/2 (45%/20), starting parameter set
2025-04-10 16:42:17: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/5/2 (45%/20), starting new job
2025-04-10 16:42:18: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/5/3 (50%/20), started new job
2025-04-10 16:42:18: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/5/3 (50%/20), starting parameter set
2025-04-10 16:42:19: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/5/3 (50%/20), starting new job
2025-04-10 16:42:20: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/1 (55%/20), started new job
2025-04-10 16:42:20: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/1 (55%/20), starting parameter set
2025-04-10 16:42:20: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/1 (55%/20), starting new job
2025-04-10 16:42:21: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/2 (60%/20), started new job
2025-04-10 16:42:21: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/2 (60%/20), starting parameter set
2025-04-10 16:42:22: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/2 (60%/20), starting new job
2025-04-10 16:42:23: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/3 (65%/20), started new job
2025-04-10 16:42:23: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/3 (65%/20), starting parameter set
2025-04-10 16:42:23: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/3 (65%/20), starting new job
2025-04-10 16:42:24: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/4 (70%/20), started new job
2025-04-10 16:42:24: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/4 (70%/20), starting parameter set
2025-04-10 16:42:25: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 10/4 (70%/20), starting new job
2025-04-10 16:42:26: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/1 (75%/20), started new job
2025-04-10 16:42:26: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/1 (75%/20), starting parameter set
2025-04-10 16:42:26: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/1 (75%/20), starting new job
2025-04-10 16:42:27: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), started new job
2025-04-10 16:42:28: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), starting parameter set
2025-04-10 16:42:28: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), starting new job
2025-04-10 16:42:29: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/3 (85%/20), started new job
2025-04-10 16:42:29: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/3 (85%/20), starting parameter set
2025-04-10 16:42:29: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/3 (85%/20), starting new job
2025-04-10 16:42:30: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/1 (90%/20), started new job
2025-04-10 16:42:31: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/1 (90%/20), starting parameter set
2025-04-10 16:42:31: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/1 (90%/20), starting new job
2025-04-10 16:42:32: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/2 (95%/20), started new job
2025-04-10 16:42:32: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/2 (95%/20), starting parameter set
2025-04-10 16:42:33: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/2 (95%/20), starting new job
2025-04-10 16:42:34: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/3 (100%/20), started new job
2025-04-10 16:42:45: SobolGenerator, Failed jobs: 40 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed 19/1 (100%/20), job_failed
2025-04-10 16:42:57: SobolGenerator, Failed jobs: 41 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed 18/1 (95%/20), job_failed
2025-04-10 16:43:09: SobolGenerator, Failed jobs: 42 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 18 (90%/20), job_failed
2025-04-10 16:43:25: SobolGenerator, Failed jobs: 43 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/running/completed 3/12/2 (85%/20), finishing jobs after starting them, finished 3 jobs
2025-04-10 16:43:25: SobolGenerator, Failed jobs: 43 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/running/completed 3/12/2 (85%/20), got 20, requested 20
2025-04-10 16:43:25: SobolGenerator, Failed jobs: 43 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/running/completed 3/12/2 (85%/20), job_failed
2025-04-10 16:43:36: SobolGenerator, Failed jobs: 44 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/running/completed 2/12/2 (80%/20), job_failed
2025-04-10 16:43:47: SobolGenerator, Failed jobs: 45 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/running/completed 1/12/2 (75%/20), job_failed
2025-04-10 16:43:58: SobolGenerator, Failed jobs: 46 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed 12/2 (70%/20), job_failed
2025-04-10 16:44:08: SobolGenerator, Failed jobs: 47 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/6 (65%/20), job_failed
2025-04-10 16:44:20: SobolGenerator, Failed jobs: 48 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/6 (60%/20), job_failed
2025-04-10 16:44:30: SobolGenerator, Failed jobs: 49 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/5 (55%/20), job_failed
2025-04-10 16:44:41: SobolGenerator, Failed jobs: 50 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/5 (50%/20), job_failed
2025-04-10 16:44:51: SobolGenerator, Failed jobs: 51 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/4 (45%/20), job_failed
2025-04-10 16:45:02: SobolGenerator, Failed jobs: 52 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/4 (40%/20), job_failed
2025-04-10 16:45:12: SobolGenerator, Failed jobs: 53 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/3 (35%/20), job_failed
2025-04-10 16:45:23: SobolGenerator, Failed jobs: 54 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/3 (30%/20), job_failed
2025-04-10 16:45:33: SobolGenerator, Failed jobs: 55 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/2 (25%/20), job_failed
2025-04-10 16:45:44: SobolGenerator, Failed jobs: 56 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/2 (20%/20), job_failed
2025-04-10 16:45:55: SobolGenerator, Failed jobs: 57 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/2 (15%/20), job_failed
2025-04-10 16:46:06: SobolGenerator, Failed jobs: 58 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 2 (10%/20), job_failed
2025-04-10 16:46:17: SobolGenerator, Failed jobs: 59 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 16:46:27: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (17), finished 17 jobs
2025-04-10 16:46:28: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 16:46:29: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 16:46:29: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 3s
2025-04-10 16:46:29: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 3s
2025-04-10 16:46:29: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 2s
2025-04-10 16:46:30: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 2s
2025-04-10 16:46:30: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 2s
2025-04-10 16:46:30: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 2s
2025-04-10 16:46:30: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 2s
2025-04-10 16:46:30: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 2s
2025-04-10 16:46:31: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 1s
2025-04-10 16:46:31: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 1s
2025-04-10 16:46:31: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 1s
2025-04-10 16:46:31: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 16:46:31: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 16:46:32: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 16:46:32: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 0s
2025-04-10 16:46:32: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 16:46:32: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 16:46:33: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 16:46:33: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 16:46:33: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 16:46:33: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 16:46:34: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 16:46:35: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 16:46:35: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 16:46:36: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 16:46:36: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 16:46:36: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 16:46:37: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 2/1 (15%/20), started new job
2025-04-10 16:46:37: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 2/1 (15%/20), starting parameter set
2025-04-10 16:46:38: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 2/1 (15%/20), starting new job
2025-04-10 16:46:39: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 2/2 (20%/20), started new job
2025-04-10 16:46:39: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 2/2 (20%/20), starting parameter set
2025-04-10 16:46:39: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 2/2 (20%/20), starting new job
2025-04-10 16:46:40: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 2/3 (25%/20), started new job
2025-04-10 16:46:40: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 2/3 (25%/20), starting parameter set
2025-04-10 16:46:41: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 2/3 (25%/20), starting new job
2025-04-10 16:46:42: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/3/1 (30%/20), started new job
2025-04-10 16:46:42: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/3/1 (30%/20), starting parameter set
2025-04-10 16:46:42: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/3/1 (30%/20), starting new job
2025-04-10 16:46:43: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/3/2 (35%/20), started new job
2025-04-10 16:46:43: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/3/2 (35%/20), starting parameter set
2025-04-10 16:46:44: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/3/2 (35%/20), starting new job
2025-04-10 16:46:45: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/1 (40%/20), started new job
2025-04-10 16:46:45: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/1 (40%/20), starting parameter set
2025-04-10 16:46:45: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/1 (40%/20), starting new job
2025-04-10 16:46:47: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 7/2 (45%/20), started new job
2025-04-10 16:46:47: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 7/2 (45%/20), starting parameter set
2025-04-10 16:46:48: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 7/2 (45%/20), starting new job
2025-04-10 16:46:49: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/1 (50%/20), started new job
2025-04-10 16:46:49: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/1 (50%/20), starting parameter set
2025-04-10 16:46:49: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/1 (50%/20), starting new job
2025-04-10 16:46:50: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/2 (55%/20), started new job
2025-04-10 16:46:50: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/2 (55%/20), starting parameter set
2025-04-10 16:46:51: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/2 (55%/20), starting new job
2025-04-10 16:46:52: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/1 (60%/20), started new job
2025-04-10 16:46:52: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/1 (60%/20), starting parameter set
2025-04-10 16:46:52: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/1 (60%/20), starting new job
2025-04-10 16:46:53: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/2 (65%/20), started new job
2025-04-10 16:46:54: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/2 (65%/20), starting parameter set
2025-04-10 16:46:54: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/2 (65%/20), starting new job
2025-04-10 16:46:55: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/3 (70%/20), started new job
2025-04-10 16:46:55: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/3 (70%/20), starting parameter set
2025-04-10 16:46:55: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/3 (70%/20), starting new job
2025-04-10 16:46:56: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/4 (75%/20), started new job
2025-04-10 16:46:56: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/4 (75%/20), starting parameter set
2025-04-10 16:46:57: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/4 (75%/20), starting new job
2025-04-10 16:46:58: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/1 (80%/20), started new job
2025-04-10 16:46:58: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/1 (80%/20), starting parameter set
2025-04-10 16:46:58: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/1 (80%/20), starting new job
2025-04-10 16:46:59: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/2 (85%/20), started new job
2025-04-10 16:46:59: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/2 (85%/20), starting parameter set
2025-04-10 16:47:00: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/2 (85%/20), starting new job
2025-04-10 16:47:01: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/3 (90%/20), started new job
2025-04-10 16:47:01: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/3 (90%/20), starting parameter set
2025-04-10 16:47:01: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/3 (90%/20), starting new job
2025-04-10 16:47:02: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/11/1 (95%/20), started new job
2025-04-10 16:47:02: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/11/1 (95%/20), starting parameter set
2025-04-10 16:47:03: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/11/1 (95%/20), starting new job
2025-04-10 16:47:04: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/11/2 (100%/20), started new job
2025-04-10 16:47:30: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 4/2/14 (100%/20), got 20, requested 20
2025-04-10 16:47:30: SobolGenerator, Failed jobs: 60 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 4/2/14 (100%/20), job_failed
2025-04-10 16:47:43: SobolGenerator, Failed jobs: 61 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 2/3/14 (95%/20), job_failed
2025-04-10 16:47:55: SobolGenerator, Failed jobs: 62 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/running/out_of_memory 3/14/1 (90%/20), job_failed
2025-04-10 16:48:07: SobolGenerator, Failed jobs: 63 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/10 (85%/20), job_failed
2025-04-10 16:48:19: SobolGenerator, Failed jobs: 64 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/6 (80%/20), job_failed
2025-04-10 16:48:31: SobolGenerator, Failed jobs: 65 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/6 (75%/20), job_failed
2025-04-10 16:48:43: SobolGenerator, Failed jobs: 66 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/8 (70%/20), job_failed
2025-04-10 16:48:56: SobolGenerator, Failed jobs: 67 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/5 (65%/20), job_failed
2025-04-10 16:49:09: SobolGenerator, Failed jobs: 68 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/5 (60%/20), job_failed
2025-04-10 16:49:24: SobolGenerator, Failed jobs: 69 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/5 (55%/20), job_failed
2025-04-10 16:49:40: SobolGenerator, Failed jobs: 70 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/5 (50%/20), job_failed
2025-04-10 16:49:56: SobolGenerator, Failed jobs: 71 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/5 (45%/20), job_failed
2025-04-10 16:50:09: SobolGenerator, Failed jobs: 72 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/3 (40%/20), job_failed
2025-04-10 16:50:22: SobolGenerator, Failed jobs: 73 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/4 (35%/20), job_failed
2025-04-10 16:50:35: SobolGenerator, Failed jobs: 74 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/2 (30%/20), job_failed
2025-04-10 16:50:48: SobolGenerator, Failed jobs: 75 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/3 (25%/20), job_failed
2025-04-10 16:51:01: SobolGenerator, Failed jobs: 76 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/3 (20%/20), job_failed
2025-04-10 16:51:14: SobolGenerator, Failed jobs: 77 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 3 (15%/20), job_failed
2025-04-10 16:51:33: SobolGenerator, Failed jobs: 78 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 2 (10%/20), job_failed
2025-04-10 16:51:49: SobolGenerator, Failed jobs: 79 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 16:52:04: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (20), finished 20 jobs
2025-04-10 16:52:06: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 16:52:07: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 16:52:07: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 3s
2025-04-10 16:52:07: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 3s
2025-04-10 16:52:07: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 3s
2025-04-10 16:52:08: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 3s
2025-04-10 16:52:08: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 2s
2025-04-10 16:52:08: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 2s
2025-04-10 16:52:08: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 2s
2025-04-10 16:52:08: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 2s
2025-04-10 16:52:09: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 2s
2025-04-10 16:52:09: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 1s
2025-04-10 16:52:09: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 1s
2025-04-10 16:52:09: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 16:52:10: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 16:52:10: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 16:52:10: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 16:52:10: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 16:52:11: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 16:52:11: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 16:52:11: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 16:52:11: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 16:52:11: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 16:52:13: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 1 (5%/20), started new job
2025-04-10 16:52:13: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 1 (5%/20), starting parameter set
2025-04-10 16:52:13: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 1 (5%/20), starting new job
2025-04-10 16:52:14: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), started new job
2025-04-10 16:52:15: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting parameter set
2025-04-10 16:52:15: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting new job
2025-04-10 16:52:16: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), started new job
2025-04-10 16:52:16: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), starting parameter set
2025-04-10 16:52:16: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), starting new job
2025-04-10 16:52:19: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), started new job
2025-04-10 16:52:19: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), starting parameter set
2025-04-10 16:52:19: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), starting new job
2025-04-10 16:52:20: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), started new job
2025-04-10 16:52:20: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), starting parameter set
2025-04-10 16:52:21: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), starting new job
2025-04-10 16:52:22: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), started new job
2025-04-10 16:52:22: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), starting parameter set
2025-04-10 16:52:22: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), starting new job
2025-04-10 16:52:23: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/3 (35%/20), started new job
2025-04-10 16:52:23: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/3 (35%/20), starting parameter set
2025-04-10 16:52:24: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/3 (35%/20), starting new job
2025-04-10 16:52:25: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/6/1 (40%/20), started new job
2025-04-10 16:52:25: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/6/1 (40%/20), starting parameter set
2025-04-10 16:52:25: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/6/1 (40%/20), starting new job
2025-04-10 16:52:27: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/6/2 (45%/20), started new job
2025-04-10 16:52:28: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/6/2 (45%/20), starting parameter set
2025-04-10 16:52:28: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/6/2 (45%/20), starting new job
2025-04-10 16:52:29: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/6/3 (50%/20), started new job
2025-04-10 16:52:29: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/6/3 (50%/20), starting parameter set
2025-04-10 16:52:29: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/6/3 (50%/20), starting new job
2025-04-10 16:52:30: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/1 (55%/20), started new job
2025-04-10 16:52:30: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/1 (55%/20), starting parameter set
2025-04-10 16:52:31: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/1 (55%/20), starting new job
2025-04-10 16:52:32: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/2 (60%/20), started new job
2025-04-10 16:52:33: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/2 (60%/20), starting parameter set
2025-04-10 16:52:33: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/2 (60%/20), starting new job
2025-04-10 16:52:34: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/2/1 (65%/20), started new job
2025-04-10 16:52:34: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/2/1 (65%/20), starting parameter set
2025-04-10 16:52:34: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/2/1 (65%/20), starting new job
2025-04-10 16:52:36: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/1 (70%/20), started new job
2025-04-10 16:52:36: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/1 (70%/20), starting parameter set
2025-04-10 16:52:36: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/1 (70%/20), starting new job
2025-04-10 16:52:37: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/2 (75%/20), started new job
2025-04-10 16:52:37: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/2 (75%/20), starting parameter set
2025-04-10 16:52:37: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/2 (75%/20), starting new job
2025-04-10 16:52:39: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/2/1 (80%/20), started new job
2025-04-10 16:52:39: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/2/1 (80%/20), starting parameter set
2025-04-10 16:52:39: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/2/1 (80%/20), starting new job
2025-04-10 16:52:40: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/1 (85%/20), started new job
2025-04-10 16:52:40: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/1 (85%/20), starting parameter set
2025-04-10 16:52:41: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/1 (85%/20), starting new job
2025-04-10 16:52:42: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/2 (90%/20), started new job
2025-04-10 16:52:42: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/2 (90%/20), starting parameter set
2025-04-10 16:52:42: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/2 (90%/20), starting new job
2025-04-10 16:52:43: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/3 (95%/20), started new job
2025-04-10 16:52:43: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/3 (95%/20), starting parameter set
2025-04-10 16:52:44: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/3 (95%/20), starting new job
2025-04-10 16:52:45: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/4/1 (100%/20), started new job
2025-04-10 16:53:20: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/running 3/17 (100%/20), got 20, requested 20
2025-04-10 16:53:20: SobolGenerator, Failed jobs: 80 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/running 3/17 (100%/20), job_failed
2025-04-10 16:53:39: SobolGenerator, Failed jobs: 81 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed 17/2 (95%/20), job_failed
2025-04-10 16:53:55: SobolGenerator, Failed jobs: 82 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/8 (90%/20), job_failed
2025-04-10 16:54:11: SobolGenerator, Failed jobs: 83 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/9 (85%/20), job_failed
2025-04-10 16:54:27: SobolGenerator, Failed jobs: 84 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/9 (80%/20), job_failed
2025-04-10 16:54:41: SobolGenerator, Failed jobs: 85 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/9 (75%/20), job_failed
2025-04-10 16:54:57: SobolGenerator, Failed jobs: 86 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/9 (70%/20), job_failed
2025-04-10 16:55:11: SobolGenerator, Failed jobs: 87 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/4 (65%/20), job_failed
2025-04-10 16:55:26: SobolGenerator, Failed jobs: 88 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/8 (60%/20), job_failed
2025-04-10 16:55:40: SobolGenerator, Failed jobs: 89 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/8 (55%/20), job_failed
2025-04-10 16:55:54: SobolGenerator, Failed jobs: 90 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/2 (50%/20), job_failed
2025-04-10 16:56:09: SobolGenerator, Failed jobs: 91 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/2 (45%/20), job_failed
2025-04-10 16:56:24: SobolGenerator, Failed jobs: 92 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/2 (40%/20), job_failed
2025-04-10 16:56:38: SobolGenerator, Failed jobs: 93 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/2 (35%/20), job_failed
2025-04-10 16:56:52: SobolGenerator, Failed jobs: 94 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/2 (30%/20), job_failed
2025-04-10 16:57:06: SobolGenerator, Failed jobs: 95 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/2 (25%/20), job_failed
2025-04-10 16:57:20: SobolGenerator, Failed jobs: 96 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/2 (20%/20), job_failed
2025-04-10 16:57:36: SobolGenerator, Failed jobs: 97 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), job_failed
2025-04-10 16:57:51: SobolGenerator, Failed jobs: 98 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/1 (10%/20), job_failed
2025-04-10 16:58:05: SobolGenerator, Failed jobs: 99 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 16:58:21: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (20), finished 20 jobs
2025-04-10 16:58:23: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 16:58:24: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 16:58:24: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 4s
2025-04-10 16:58:24: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 3s
2025-04-10 16:58:25: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 3s
2025-04-10 16:58:25: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 3s
2025-04-10 16:58:25: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 2s
2025-04-10 16:58:25: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 2s
2025-04-10 16:58:25: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 2s
2025-04-10 16:58:26: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 2s
2025-04-10 16:58:26: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 2s
2025-04-10 16:58:26: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 1s
2025-04-10 16:58:26: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 1s
2025-04-10 16:58:27: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 16:58:27: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 16:58:27: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 16:58:27: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 16:58:27: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 16:58:28: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 16:58:28: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 16:58:28: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 16:58:28: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 16:58:29: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 16:58:30: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 16:58:30: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 16:58:30: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 16:58:31: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), started new job
2025-04-10 16:58:31: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting parameter set
2025-04-10 16:58:32: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting new job
2025-04-10 16:58:33: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), started new job
2025-04-10 16:58:33: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting parameter set
2025-04-10 16:58:33: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting new job
2025-04-10 16:58:34: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/3 (20%/20), started new job
2025-04-10 16:58:34: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/3 (20%/20), starting parameter set
2025-04-10 16:58:35: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/3 (20%/20), starting new job
2025-04-10 16:58:37: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 4/1 (25%/20), started new job
2025-04-10 16:58:37: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 4/1 (25%/20), starting parameter set
2025-04-10 16:58:37: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 4/1 (25%/20), starting new job
2025-04-10 16:58:38: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), started new job
2025-04-10 16:58:38: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), starting parameter set
2025-04-10 16:58:39: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), starting new job
2025-04-10 16:58:40: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/2 (35%/20), started new job
2025-04-10 16:58:40: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/2 (35%/20), starting parameter set
2025-04-10 16:58:40: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/2 (35%/20), starting new job
2025-04-10 16:58:42: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/1 (40%/20), started new job
2025-04-10 16:58:42: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/1 (40%/20), starting parameter set
2025-04-10 16:58:42: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/1 (40%/20), starting new job
2025-04-10 16:58:43: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/2 (45%/20), started new job
2025-04-10 16:58:43: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/2 (45%/20), starting parameter set
2025-04-10 16:58:44: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/2 (45%/20), starting new job
2025-04-10 16:58:46: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/4/1 (50%/20), started new job
2025-04-10 16:58:46: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/4/1 (50%/20), starting parameter set
2025-04-10 16:58:46: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/4/1 (50%/20), starting new job
2025-04-10 16:58:47: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/4/2 (55%/20), started new job
2025-04-10 16:58:47: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/4/2 (55%/20), starting parameter set
2025-04-10 16:58:48: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/4/2 (55%/20), starting new job
2025-04-10 16:58:49: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/4/3 (60%/20), started new job
2025-04-10 16:58:49: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/4/3 (60%/20), starting parameter set
2025-04-10 16:58:49: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/4/3 (60%/20), starting new job
2025-04-10 16:58:50: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/1 (65%/20), started new job
2025-04-10 16:58:50: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/1 (65%/20), starting parameter set
2025-04-10 16:58:51: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/1 (65%/20), starting new job
2025-04-10 16:58:52: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/2 (70%/20), started new job
2025-04-10 16:58:52: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/2 (70%/20), starting parameter set
2025-04-10 16:58:52: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/2 (70%/20), starting new job
2025-04-10 16:58:53: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/3 (75%/20), started new job
2025-04-10 16:58:53: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/3 (75%/20), starting parameter set
2025-04-10 16:58:54: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/3 (75%/20), starting new job
2025-04-10 16:58:55: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/4 (80%/20), started new job
2025-04-10 16:58:55: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/4 (80%/20), starting parameter set
2025-04-10 16:58:55: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/4 (80%/20), starting new job
2025-04-10 16:58:56: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/11/1 (85%/20), started new job
2025-04-10 16:58:56: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/11/1 (85%/20), starting parameter set
2025-04-10 16:58:57: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/11/1 (85%/20), starting new job
2025-04-10 16:58:58: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 16/2 (90%/20), started new job
2025-04-10 16:58:58: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 16/2 (90%/20), starting parameter set
2025-04-10 16:58:59: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 16/2 (90%/20), starting new job
2025-04-10 16:59:00: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 16/3 (95%/20), started new job
2025-04-10 16:59:00: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 16/3 (95%/20), starting parameter set
2025-04-10 16:59:00: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 16/3 (95%/20), starting new job
2025-04-10 16:59:01: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 16/3/1 (100%/20), started new job
2025-04-10 16:59:21: SobolGenerator, Failed jobs: 100 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/running 4/16 (100%/20), job_failed
2025-04-10 16:59:41: SobolGenerator, Failed jobs: 101 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 4/1/14 (95%/20), job_failed
2025-04-10 17:00:02: SobolGenerator, Failed jobs: 102 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 3/1/14 (90%/20), job_failed
2025-04-10 17:00:18: SobolGenerator, Failed jobs: 103 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 2/1/14 (85%/20), job_failed
2025-04-10 17:00:35: SobolGenerator, Failed jobs: 104 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/8 (80%/20), job_failed
2025-04-10 17:00:51: SobolGenerator, Failed jobs: 105 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/8 (75%/20), job_failed
2025-04-10 17:01:08: SobolGenerator, Failed jobs: 106 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/7 (70%/20), job_failed
2025-04-10 17:01:25: SobolGenerator, Failed jobs: 107 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/6 (65%/20), job_failed
2025-04-10 17:01:42: SobolGenerator, Failed jobs: 108 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/6 (60%/20), job_failed
2025-04-10 17:01:59: SobolGenerator, Failed jobs: 109 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/5 (55%/20), job_failed
2025-04-10 17:02:15: SobolGenerator, Failed jobs: 110 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/5 (50%/20), job_failed
2025-04-10 17:02:32: SobolGenerator, Failed jobs: 111 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/4 (45%/20), job_failed
2025-04-10 17:02:48: SobolGenerator, Failed jobs: 112 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/4 (40%/20), job_failed
2025-04-10 17:03:04: SobolGenerator, Failed jobs: 113 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/3 (35%/20), job_failed
2025-04-10 17:03:21: SobolGenerator, Failed jobs: 114 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/3 (30%/20), job_failed
2025-04-10 17:03:37: SobolGenerator, Failed jobs: 115 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/3 (25%/20), job_failed
2025-04-10 17:03:54: SobolGenerator, Failed jobs: 116 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/2 (20%/20), job_failed
2025-04-10 17:04:10: SobolGenerator, Failed jobs: 117 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/2 (15%/20), job_failed
2025-04-10 17:04:28: SobolGenerator, Failed jobs: 118 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/1 (10%/20), finishing jobs after starting them, finished 18 jobs
2025-04-10 17:04:28: SobolGenerator, Failed jobs: 118 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/1 (10%/20), got 20, requested 20
2025-04-10 17:04:28: SobolGenerator, Failed jobs: 118 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/1 (10%/20), job_failed
2025-04-10 17:04:44: SobolGenerator, Failed jobs: 119 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 17:05:02: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (2), finished 2 jobs
2025-04-10 17:05:04: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 17:05:06: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 17:05:06: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 3s
2025-04-10 17:05:06: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 3s
2025-04-10 17:05:06: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 3s
2025-04-10 17:05:06: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 3s
2025-04-10 17:05:07: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 3s
2025-04-10 17:05:07: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 2s
2025-04-10 17:05:07: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 2s
2025-04-10 17:05:07: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 2s
2025-04-10 17:05:08: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 2s
2025-04-10 17:05:08: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 2s
2025-04-10 17:05:08: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 1s
2025-04-10 17:05:08: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 17:05:09: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 17:05:09: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 17:05:09: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 17:05:09: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 17:05:09: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 17:05:10: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 17:05:10: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 17:05:10: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 17:05:10: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 17:05:12: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 17:05:12: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 17:05:12: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 17:05:13: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 17:05:14: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 17:05:14: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 17:05:15: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), started new job
2025-04-10 17:05:15: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), starting parameter set
2025-04-10 17:05:15: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), starting new job
2025-04-10 17:05:17: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 4 (20%/20), started new job
2025-04-10 17:05:18: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 4 (20%/20), starting parameter set
2025-04-10 17:05:18: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 4 (20%/20), starting new job
2025-04-10 17:05:19: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/1 (25%/20), started new job
2025-04-10 17:05:19: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/1 (25%/20), starting parameter set
2025-04-10 17:05:19: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 4/1 (25%/20), starting new job
2025-04-10 17:05:20: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), started new job
2025-04-10 17:05:20: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), starting parameter set
2025-04-10 17:05:21: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), starting new job
2025-04-10 17:05:22: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/1 (35%/20), started new job
2025-04-10 17:05:22: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/1 (35%/20), starting parameter set
2025-04-10 17:05:22: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/1 (35%/20), starting new job
2025-04-10 17:05:23: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/2 (40%/20), started new job
2025-04-10 17:05:23: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/2 (40%/20), starting parameter set
2025-04-10 17:05:24: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/2 (40%/20), starting new job
2025-04-10 17:05:25: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/3 (45%/20), started new job
2025-04-10 17:05:25: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/3 (45%/20), starting parameter set
2025-04-10 17:05:25: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/3 (45%/20), starting new job
2025-04-10 17:05:28: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 5/5 (50%/20), started new job
2025-04-10 17:05:28: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 5/5 (50%/20), starting parameter set
2025-04-10 17:05:28: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 5/5 (50%/20), starting new job
2025-04-10 17:05:29: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/1 (55%/20), started new job
2025-04-10 17:05:29: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/1 (55%/20), starting parameter set
2025-04-10 17:05:30: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/1 (55%/20), starting new job
2025-04-10 17:05:31: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/2 (60%/20), started new job
2025-04-10 17:05:31: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/2 (60%/20), starting parameter set
2025-04-10 17:05:31: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/2 (60%/20), starting new job
2025-04-10 17:05:32: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/1 (65%/20), started new job
2025-04-10 17:05:33: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/1 (65%/20), starting parameter set
2025-04-10 17:05:33: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/1 (65%/20), starting new job
2025-04-10 17:05:34: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/2 (70%/20), started new job
2025-04-10 17:05:34: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/2 (70%/20), starting parameter set
2025-04-10 17:05:34: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/2 (70%/20), starting new job
2025-04-10 17:05:36: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/3 (75%/20), started new job
2025-04-10 17:05:36: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/3 (75%/20), starting parameter set
2025-04-10 17:05:36: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/7/3 (75%/20), starting new job
2025-04-10 17:05:37: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 15/1 (80%/20), started new job
2025-04-10 17:05:37: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 15/1 (80%/20), starting parameter set
2025-04-10 17:05:38: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 15/1 (80%/20), starting new job
2025-04-10 17:05:39: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 15/2 (85%/20), started new job
2025-04-10 17:05:39: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 15/2 (85%/20), starting parameter set
2025-04-10 17:05:39: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 15/2 (85%/20), starting new job
2025-04-10 17:05:41: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/2/1 (90%/20), started new job
2025-04-10 17:05:42: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/2/1 (90%/20), starting parameter set
2025-04-10 17:05:42: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/2/1 (90%/20), starting new job
2025-04-10 17:05:43: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 17/1/1 (95%/20), started new job
2025-04-10 17:05:43: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 17/1/1 (95%/20), starting parameter set
2025-04-10 17:05:43: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 17/1/1 (95%/20), starting new job
2025-04-10 17:05:44: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 17/1/2 (100%/20), started new job
2025-04-10 17:06:09: SobolGenerator, Failed jobs: 120 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed 19/1 (100%/20), job_failed
2025-04-10 17:06:27: SobolGenerator, Failed jobs: 121 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed 18/1 (95%/20), job_failed
2025-04-10 17:06:46: SobolGenerator, Failed jobs: 122 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 7/9/2 (90%/20), job_failed
2025-04-10 17:07:06: SobolGenerator, Failed jobs: 123 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 6/9/2 (85%/20), job_failed
2025-04-10 17:07:25: SobolGenerator, Failed jobs: 124 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 6/8/2 (80%/20), job_failed
2025-04-10 17:07:45: SobolGenerator, Failed jobs: 125 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 5/8/2 (75%/20), job_failed
2025-04-10 17:08:04: SobolGenerator, Failed jobs: 126 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/7 (70%/20), job_failed
2025-04-10 17:08:24: SobolGenerator, Failed jobs: 127 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/7 (65%/20), job_failed
2025-04-10 17:08:43: SobolGenerator, Failed jobs: 128 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/6 (60%/20), job_failed
2025-04-10 17:09:02: SobolGenerator, Failed jobs: 129 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/5 (55%/20), job_failed
2025-04-10 17:09:21: SobolGenerator, Failed jobs: 130 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/4 (50%/20), job_failed
2025-04-10 17:09:40: SobolGenerator, Failed jobs: 131 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/3 (45%/20), job_failed
2025-04-10 17:10:01: SobolGenerator, Failed jobs: 132 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/2 (40%/20), job_failed
2025-04-10 17:10:20: SobolGenerator, Failed jobs: 133 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/2 (35%/20), job_failed
2025-04-10 17:10:40: SobolGenerator, Failed jobs: 134 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/1 (30%/20), job_failed
2025-04-10 17:10:59: SobolGenerator, Failed jobs: 135 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 5 (25%/20), job_failed
2025-04-10 17:11:18: SobolGenerator, Failed jobs: 136 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 4 (20%/20), job_failed
2025-04-10 17:11:37: SobolGenerator, Failed jobs: 137 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 3 (15%/20), job_failed
2025-04-10 17:11:57: SobolGenerator, Failed jobs: 138 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 2 (10%/20), finishing jobs after starting them, finished 18 jobs
2025-04-10 17:11:57: SobolGenerator, Failed jobs: 138 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 2 (10%/20), got 20, requested 20
2025-04-10 17:11:58: SobolGenerator, Failed jobs: 138 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 2 (10%/20), job_failed
2025-04-10 17:12:17: SobolGenerator, Failed jobs: 139 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 17:12:37: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (2), finished 2 jobs
2025-04-10 17:12:40: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 17:12:42: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 17:12:42: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 4s
2025-04-10 17:12:42: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 3s
2025-04-10 17:12:42: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 3s
2025-04-10 17:12:42: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 3s
2025-04-10 17:12:43: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 3s
2025-04-10 17:12:43: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 3s
2025-04-10 17:12:43: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 2s
2025-04-10 17:12:43: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 2s
2025-04-10 17:12:44: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 2s
2025-04-10 17:12:44: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 2s
2025-04-10 17:12:44: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 1s
2025-04-10 17:12:44: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 17:12:45: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 17:12:45: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 17:12:45: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 17:12:45: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 17:12:46: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 17:12:46: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 17:12:46: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 17:12:47: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 17:12:47: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 17:12:48: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 17:12:48: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 17:12:48: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 17:12:49: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), started new job
2025-04-10 17:12:49: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting parameter set
2025-04-10 17:12:50: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting new job
2025-04-10 17:12:51: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), started new job
2025-04-10 17:12:51: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), starting parameter set
2025-04-10 17:12:51: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), starting new job
2025-04-10 17:12:52: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), started new job
2025-04-10 17:12:53: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), starting parameter set
2025-04-10 17:12:53: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), starting new job
2025-04-10 17:12:54: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), started new job
2025-04-10 17:12:54: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), starting parameter set
2025-04-10 17:12:54: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), starting new job
2025-04-10 17:12:55: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), started new job
2025-04-10 17:12:55: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), starting parameter set
2025-04-10 17:12:56: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), starting new job
2025-04-10 17:12:57: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/3 (35%/20), started new job
2025-04-10 17:12:57: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/3 (35%/20), starting parameter set
2025-04-10 17:12:57: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/3 (35%/20), starting new job
2025-04-10 17:12:58: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/4 (40%/20), started new job
2025-04-10 17:12:58: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/4 (40%/20), starting parameter set
2025-04-10 17:12:59: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/4 (40%/20), starting new job
2025-04-10 17:13:00: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/1 (45%/20), started new job
2025-04-10 17:13:00: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/1 (45%/20), starting parameter set
2025-04-10 17:13:00: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/1 (45%/20), starting new job
2025-04-10 17:13:01: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/2 (50%/20), started new job
2025-04-10 17:13:01: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/2 (50%/20), starting parameter set
2025-04-10 17:13:02: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/2 (50%/20), starting new job
2025-04-10 17:13:03: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/3 (55%/20), started new job
2025-04-10 17:13:03: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/3 (55%/20), starting parameter set
2025-04-10 17:13:03: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/3 (55%/20), starting new job
2025-04-10 17:13:05: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/10/1 (60%/20), started new job
2025-04-10 17:13:05: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/10/1 (60%/20), starting parameter set
2025-04-10 17:13:05: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/10/1 (60%/20), starting new job
2025-04-10 17:13:06: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/10/2 (65%/20), started new job
2025-04-10 17:13:06: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/10/2 (65%/20), starting parameter set
2025-04-10 17:13:06: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/10/2 (65%/20), starting new job
2025-04-10 17:13:08: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/3 (70%/20), started new job
2025-04-10 17:13:08: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/3 (70%/20), starting parameter set
2025-04-10 17:13:08: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/3 (70%/20), starting new job
2025-04-10 17:13:09: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/1 (75%/20), started new job
2025-04-10 17:13:09: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/1 (75%/20), starting parameter set
2025-04-10 17:13:10: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/1 (75%/20), starting new job
2025-04-10 17:13:12: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/2 (80%/20), started new job
2025-04-10 17:13:12: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/2 (80%/20), starting parameter set
2025-04-10 17:13:12: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/2 (80%/20), starting new job
2025-04-10 17:13:14: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/3 (85%/20), started new job
2025-04-10 17:13:14: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/3 (85%/20), starting parameter set
2025-04-10 17:13:14: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/3 (85%/20), starting new job
2025-04-10 17:13:15: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/1 (90%/20), started new job
2025-04-10 17:13:15: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/1 (90%/20), starting parameter set
2025-04-10 17:13:15: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/1 (90%/20), starting new job
2025-04-10 17:13:17: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/2 (95%/20), started new job
2025-04-10 17:13:17: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/2 (95%/20), starting parameter set
2025-04-10 17:13:17: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/2 (95%/20), starting new job
2025-04-10 17:13:18: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/3 (100%/20), started new job
2025-04-10 17:13:46: SobolGenerator, Failed jobs: 140 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/running 1/19 (100%/20), job_failed
2025-04-10 17:14:09: SobolGenerator, Failed jobs: 141 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 8/5/6 (95%/20), job_failed
2025-04-10 17:14:31: SobolGenerator, Failed jobs: 142 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 7/5/6 (90%/20), job_failed
2025-04-10 17:14:53: SobolGenerator, Failed jobs: 143 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 6/5/6 (85%/20), job_failed
2025-04-10 17:15:13: SobolGenerator, Failed jobs: 144 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/9 (80%/20), job_failed
2025-04-10 17:15:34: SobolGenerator, Failed jobs: 145 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/8 (75%/20), job_failed
2025-04-10 17:16:11: SobolGenerator, Failed jobs: 146 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/7 (70%/20), job_failed
2025-04-10 17:16:33: SobolGenerator, Failed jobs: 147 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/7 (65%/20), job_failed
2025-04-10 17:16:56: SobolGenerator, Failed jobs: 148 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/6 (60%/20), job_failed
2025-04-10 17:17:18: SobolGenerator, Failed jobs: 149 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/6 (55%/20), job_failed
2025-04-10 17:17:39: SobolGenerator, Failed jobs: 150 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/5 (50%/20), job_failed
2025-04-10 17:18:01: SobolGenerator, Failed jobs: 151 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/5 (45%/20), job_failed
2025-04-10 17:18:22: SobolGenerator, Failed jobs: 152 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/4 (40%/20), job_failed
2025-04-10 17:18:44: SobolGenerator, Failed jobs: 153 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/3 (35%/20), job_failed
2025-04-10 17:19:05: SobolGenerator, Failed jobs: 154 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/3 (30%/20), job_failed
2025-04-10 17:19:27: SobolGenerator, Failed jobs: 155 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/3 (25%/20), job_failed
2025-04-10 17:19:48: SobolGenerator, Failed jobs: 156 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/2 (20%/20), job_failed
2025-04-10 17:20:12: SobolGenerator, Failed jobs: 157 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), finishing jobs after starting them, finished 17 jobs
2025-04-10 17:20:12: SobolGenerator, Failed jobs: 157 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), got 20, requested 20
2025-04-10 17:20:12: SobolGenerator, Failed jobs: 157 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), job_failed
2025-04-10 17:20:34: SobolGenerator, Failed jobs: 158 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/1 (10%/20), job_failed
2025-04-10 17:20:55: SobolGenerator, Failed jobs: 159 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 17:21:18: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (3), finished 3 jobs
2025-04-10 17:21:22: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 17:21:24: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 17:21:24: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 4s
2025-04-10 17:21:24: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 3s
2025-04-10 17:21:24: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 3s
2025-04-10 17:21:25: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 3s
2025-04-10 17:21:25: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 3s
2025-04-10 17:21:25: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 3s
2025-04-10 17:21:25: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 2s
2025-04-10 17:21:26: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 2s
2025-04-10 17:21:26: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 2s
2025-04-10 17:21:26: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 2s
2025-04-10 17:21:26: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 2s
2025-04-10 17:21:27: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 17:21:27: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 17:21:27: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 17:21:27: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 17:21:28: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 17:21:28: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 17:21:28: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 17:21:28: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 17:21:29: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 17:21:29: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 17:21:30: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 17:21:30: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 17:21:31: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 17:21:32: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 17:21:32: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 17:21:32: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 17:21:34: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), started new job
2025-04-10 17:21:34: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), starting parameter set
2025-04-10 17:21:35: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), starting new job
2025-04-10 17:21:36: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/1 (20%/20), started new job
2025-04-10 17:21:36: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/1 (20%/20), starting parameter set
2025-04-10 17:21:36: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/1 (20%/20), starting new job
2025-04-10 17:21:37: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/1/1 (25%/20), started new job
2025-04-10 17:21:37: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/1/1 (25%/20), starting parameter set
2025-04-10 17:21:38: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/1/1 (25%/20), starting new job
2025-04-10 17:21:39: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/1/2 (30%/20), started new job
2025-04-10 17:21:39: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/1/2 (30%/20), starting parameter set
2025-04-10 17:21:39: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/1/2 (30%/20), starting new job
2025-04-10 17:21:40: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/2/1 (35%/20), started new job
2025-04-10 17:21:40: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/2/1 (35%/20), starting parameter set
2025-04-10 17:21:41: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/2/1 (35%/20), starting new job
2025-04-10 17:21:43: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/2/2 (40%/20), started new job
2025-04-10 17:21:43: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/2/2 (40%/20), starting parameter set
2025-04-10 17:21:43: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/2/2 (40%/20), starting new job
2025-04-10 17:21:44: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/2/1 (45%/20), started new job
2025-04-10 17:21:44: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/2/1 (45%/20), starting parameter set
2025-04-10 17:21:45: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/2/1 (45%/20), starting new job
2025-04-10 17:21:46: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/1 (50%/20), started new job
2025-04-10 17:21:46: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/1 (50%/20), starting parameter set
2025-04-10 17:21:46: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/1 (50%/20), starting new job
2025-04-10 17:21:47: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/2 (55%/20), started new job
2025-04-10 17:21:47: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/2 (55%/20), starting parameter set
2025-04-10 17:21:48: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/2 (55%/20), starting new job
2025-04-10 17:21:49: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/3 (60%/20), started new job
2025-04-10 17:21:49: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/3 (60%/20), starting parameter set
2025-04-10 17:21:49: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/3 (60%/20), starting new job
2025-04-10 17:21:50: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/1 (65%/20), started new job
2025-04-10 17:21:50: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/1 (65%/20), starting parameter set
2025-04-10 17:21:51: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/1 (65%/20), starting new job
2025-04-10 17:21:52: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/2 (70%/20), started new job
2025-04-10 17:21:52: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/2 (70%/20), starting parameter set
2025-04-10 17:21:52: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/2 (70%/20), starting new job
2025-04-10 17:21:53: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/3 (75%/20), started new job
2025-04-10 17:21:53: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/3 (75%/20), starting parameter set
2025-04-10 17:21:54: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/3 (75%/20), starting new job
2025-04-10 17:21:55: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/4 (80%/20), started new job
2025-04-10 17:21:55: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/4 (80%/20), starting parameter set
2025-04-10 17:21:55: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/4 (80%/20), starting new job
2025-04-10 17:21:56: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/8/1 (85%/20), started new job
2025-04-10 17:21:56: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/8/1 (85%/20), starting parameter set
2025-04-10 17:21:57: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/8/1 (85%/20), starting new job
2025-04-10 17:21:58: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/8/2 (90%/20), started new job
2025-04-10 17:21:58: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/8/2 (90%/20), starting parameter set
2025-04-10 17:21:58: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/8/2 (90%/20), starting new job
2025-04-10 17:21:59: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/8/3 (95%/20), started new job
2025-04-10 17:21:59: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/8/3 (95%/20), starting parameter set
2025-04-10 17:22:00: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/8/3 (95%/20), starting new job
2025-04-10 17:22:01: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/11/1 (100%/20), started new job
2025-04-10 17:22:30: SobolGenerator, Failed jobs: 160 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/running 3/17 (100%/20), job_failed
2025-04-10 17:22:58: SobolGenerator, Failed jobs: 161 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 10/8/1 (95%/20), job_failed
2025-04-10 17:23:22: SobolGenerator, Failed jobs: 162 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 9/8/1 (90%/20), job_failed
2025-04-10 17:23:46: SobolGenerator, Failed jobs: 163 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 8/8/1 (85%/20), job_failed
2025-04-10 17:24:10: SobolGenerator, Failed jobs: 164 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/8 (80%/20), job_failed
2025-04-10 17:24:34: SobolGenerator, Failed jobs: 165 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/7 (75%/20), job_failed
2025-04-10 17:24:58: SobolGenerator, Failed jobs: 166 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/7 (70%/20), job_failed
2025-04-10 17:25:22: SobolGenerator, Failed jobs: 167 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/6 (65%/20), job_failed
2025-04-10 17:25:46: SobolGenerator, Failed jobs: 168 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/6 (60%/20), job_failed
2025-04-10 17:26:09: SobolGenerator, Failed jobs: 169 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/5 (55%/20), job_failed
2025-04-10 17:26:35: SobolGenerator, Failed jobs: 170 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/5 (50%/20), job_failed
2025-04-10 17:27:00: SobolGenerator, Failed jobs: 171 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/5 (45%/20), job_failed
2025-04-10 17:27:24: SobolGenerator, Failed jobs: 172 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/4 (40%/20), job_failed
2025-04-10 17:27:48: SobolGenerator, Failed jobs: 173 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/4 (35%/20), job_failed
2025-04-10 17:28:14: SobolGenerator, Failed jobs: 174 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/3 (30%/20), job_failed
2025-04-10 17:28:38: SobolGenerator, Failed jobs: 175 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/3 (25%/20), job_failed
2025-04-10 17:29:02: SobolGenerator, Failed jobs: 176 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/2 (20%/20), job_failed
2025-04-10 17:29:27: SobolGenerator, Failed jobs: 177 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/2 (15%/20), job_failed
2025-04-10 17:29:54: SobolGenerator, Failed jobs: 178 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/1 (10%/20), finishing jobs after starting them, finished 18 jobs
2025-04-10 17:29:54: SobolGenerator, Failed jobs: 178 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/1 (10%/20), got 20, requested 20
2025-04-10 17:29:54: SobolGenerator, Failed jobs: 178 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 1/1 (10%/20), job_failed
2025-04-10 17:30:18: SobolGenerator, Failed jobs: 179 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 17:30:45: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (2), finished 2 jobs
2025-04-10 17:30:49: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 17:30:51: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 17:30:52: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 5s
2025-04-10 17:30:52: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 4s
2025-04-10 17:30:52: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 4s
2025-04-10 17:30:52: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 3s
2025-04-10 17:30:53: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 3s
2025-04-10 17:30:53: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 3s
2025-04-10 17:30:53: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 3s
2025-04-10 17:30:53: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 2s
2025-04-10 17:30:54: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 2s
2025-04-10 17:30:54: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 2s
2025-04-10 17:30:54: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 2s
2025-04-10 17:30:54: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 1s
2025-04-10 17:30:55: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 17:30:55: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 17:30:55: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 17:30:56: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 0s
2025-04-10 17:30:56: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 17:30:56: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 17:30:56: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 17:30:57: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 17:30:57: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 17:30:58: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 17:30:58: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 17:30:59: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 17:31:00: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 17:31:00: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 17:31:00: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 17:31:01: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), started new job
2025-04-10 17:31:01: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), starting parameter set
2025-04-10 17:31:02: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 3 (15%/20), starting new job
2025-04-10 17:31:03: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/1 (20%/20), started new job
2025-04-10 17:31:03: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/1 (20%/20), starting parameter set
2025-04-10 17:31:04: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/1 (20%/20), starting new job
2025-04-10 17:31:05: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/2 (25%/20), started new job
2025-04-10 17:31:05: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/2 (25%/20), starting parameter set
2025-04-10 17:31:05: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/2 (25%/20), starting new job
2025-04-10 17:31:06: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/3 (30%/20), started new job
2025-04-10 17:31:06: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/3 (30%/20), starting parameter set
2025-04-10 17:31:07: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/3 (30%/20), starting new job
2025-04-10 17:31:08: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/1 (35%/20), started new job
2025-04-10 17:31:08: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/1 (35%/20), starting parameter set
2025-04-10 17:31:08: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/1 (35%/20), starting new job
2025-04-10 17:31:09: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/2 (40%/20), started new job
2025-04-10 17:31:09: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/2 (40%/20), starting parameter set
2025-04-10 17:31:10: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/2 (40%/20), starting new job
2025-04-10 17:31:11: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/2/1 (45%/20), started new job
2025-04-10 17:31:11: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/2/1 (45%/20), starting parameter set
2025-04-10 17:31:11: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/2/1 (45%/20), starting new job
2025-04-10 17:31:12: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/1 (50%/20), started new job
2025-04-10 17:31:12: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/1 (50%/20), starting parameter set
2025-04-10 17:31:13: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/1 (50%/20), starting new job
2025-04-10 17:31:14: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/2 (55%/20), started new job
2025-04-10 17:31:14: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/2 (55%/20), starting parameter set
2025-04-10 17:31:14: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/2 (55%/20), starting new job
2025-04-10 17:31:15: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/3 (60%/20), started new job
2025-04-10 17:31:16: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/3 (60%/20), starting parameter set
2025-04-10 17:31:16: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/1/3 (60%/20), starting new job
2025-04-10 17:31:17: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/1 (65%/20), started new job
2025-04-10 17:31:17: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/1 (65%/20), starting parameter set
2025-04-10 17:31:18: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/1 (65%/20), starting new job
2025-04-10 17:31:19: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/2 (70%/20), started new job
2025-04-10 17:31:19: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/2 (70%/20), starting parameter set
2025-04-10 17:31:19: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/2 (70%/20), starting new job
2025-04-10 17:31:20: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/3 (75%/20), started new job
2025-04-10 17:31:20: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/3 (75%/20), starting parameter set
2025-04-10 17:31:21: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/4/3 (75%/20), starting new job
2025-04-10 17:31:22: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/7/1 (80%/20), started new job
2025-04-10 17:31:22: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/7/1 (80%/20), starting parameter set
2025-04-10 17:31:22: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/7/1 (80%/20), starting new job
2025-04-10 17:31:23: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/7/2 (85%/20), started new job
2025-04-10 17:31:23: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/7/2 (85%/20), starting parameter set
2025-04-10 17:31:24: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/7/2 (85%/20), starting new job
2025-04-10 17:31:26: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/7/3 (90%/20), started new job
2025-04-10 17:31:26: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/7/3 (90%/20), starting parameter set
2025-04-10 17:31:26: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/7/3 (90%/20), starting new job
2025-04-10 17:31:27: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/10/1 (95%/20), started new job
2025-04-10 17:31:27: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/10/1 (95%/20), starting parameter set
2025-04-10 17:31:28: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/10/1 (95%/20), starting new job
2025-04-10 17:31:29: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/10/2 (100%/20), started new job
2025-04-10 17:32:00: SobolGenerator, Failed jobs: 180 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed/out_of_memory 14/5/1 (100%/20), job_failed
2025-04-10 17:32:30: SobolGenerator, Failed jobs: 181 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/out_of_memory/completed 14/1/4 (95%/20), job_failed
2025-04-10 17:32:58: SobolGenerator, Failed jobs: 182 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/9 (90%/20), job_failed
2025-04-10 17:33:24: SobolGenerator, Failed jobs: 183 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/8 (85%/20), job_failed
2025-04-10 17:33:52: SobolGenerator, Failed jobs: 184 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/7 (80%/20), job_failed
2025-04-10 17:34:19: SobolGenerator, Failed jobs: 185 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/6 (75%/20), job_failed
2025-04-10 17:34:45: SobolGenerator, Failed jobs: 186 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/6 (70%/20), job_failed
2025-04-10 17:35:12: SobolGenerator, Failed jobs: 187 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/5 (65%/20), job_failed
2025-04-10 17:35:40: SobolGenerator, Failed jobs: 188 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/5 (60%/20), job_failed
2025-04-10 17:36:07: SobolGenerator, Failed jobs: 189 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/5 (55%/20), job_failed
2025-04-10 17:36:33: SobolGenerator, Failed jobs: 190 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/5 (50%/20), job_failed
2025-04-10 17:37:00: SobolGenerator, Failed jobs: 191 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/4 (45%/20), job_failed
2025-04-10 17:37:26: SobolGenerator, Failed jobs: 192 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/3 (40%/20), job_failed
2025-04-10 17:37:57: SobolGenerator, Failed jobs: 193 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/2 (35%/20), job_failed
2025-04-10 17:38:23: SobolGenerator, Failed jobs: 194 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/1 (30%/20), job_failed
2025-04-10 17:38:51: SobolGenerator, Failed jobs: 195 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/1 (25%/20), job_failed
2025-04-10 17:39:17: SobolGenerator, Failed jobs: 196 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/1 (20%/20), job_failed
2025-04-10 17:39:44: SobolGenerator, Failed jobs: 197 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), job_failed
2025-04-10 17:40:10: SobolGenerator, Failed jobs: 198 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 2 (10%/20), job_failed
2025-04-10 17:40:39: SobolGenerator, Failed jobs: 199 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 17:40:40: SobolGenerator, Failed jobs: 199 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), got 20, requested 20
2025-04-10 17:40:40: SobolGenerator, Failed jobs: 199 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 17:41:09: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 17:41:13: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 17:41:16: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 17:41:17: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 4s
2025-04-10 17:41:17: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 4s
2025-04-10 17:41:17: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 4s
2025-04-10 17:41:18: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 3s
2025-04-10 17:41:18: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 3s
2025-04-10 17:41:18: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 3s
2025-04-10 17:41:18: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 3s
2025-04-10 17:41:19: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 2s
2025-04-10 17:41:19: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 2s
2025-04-10 17:41:19: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 2s
2025-04-10 17:41:19: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 2s
2025-04-10 17:41:20: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 2s
2025-04-10 17:41:20: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 17:41:20: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 17:41:21: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 17:41:21: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 17:41:21: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 17:41:21: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 17:41:22: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 17:41:22: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 17:41:22: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 17:41:23: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 17:41:24: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 17:41:24: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 17:41:25: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), started new job
2025-04-10 17:41:25: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting parameter set
2025-04-10 17:41:25: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting new job
2025-04-10 17:41:26: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), started new job
2025-04-10 17:41:27: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting parameter set
2025-04-10 17:41:27: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting new job
2025-04-10 17:41:28: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/3 (20%/20), started new job
2025-04-10 17:41:28: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/3 (20%/20), starting parameter set
2025-04-10 17:41:28: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/3 (20%/20), starting new job
2025-04-10 17:41:30: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/1 (25%/20), started new job
2025-04-10 17:41:30: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/1 (25%/20), starting parameter set
2025-04-10 17:41:30: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/1 (25%/20), starting new job
2025-04-10 17:41:31: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/2 (30%/20), started new job
2025-04-10 17:41:31: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/2 (30%/20), starting parameter set
2025-04-10 17:41:31: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/2 (30%/20), starting new job
2025-04-10 17:41:32: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/3 (35%/20), started new job
2025-04-10 17:41:32: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/3 (35%/20), starting parameter set
2025-04-10 17:41:33: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/3 (35%/20), starting new job
2025-04-10 17:41:34: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/4 (40%/20), started new job
2025-04-10 17:41:34: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/4 (40%/20), starting parameter set
2025-04-10 17:41:34: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/4 (40%/20), starting new job
2025-04-10 17:41:35: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/1 (45%/20), started new job
2025-04-10 17:41:35: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/1 (45%/20), starting parameter set
2025-04-10 17:41:36: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/1 (45%/20), starting new job
2025-04-10 17:41:37: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/2 (50%/20), started new job
2025-04-10 17:41:37: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/2 (50%/20), starting parameter set
2025-04-10 17:41:37: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/2 (50%/20), starting new job
2025-04-10 17:41:38: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/2/1 (55%/20), started new job
2025-04-10 17:41:38: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/2/1 (55%/20), starting parameter set
2025-04-10 17:41:39: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 8/2/1 (55%/20), starting new job
2025-04-10 17:41:40: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), started new job
2025-04-10 17:41:40: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), starting parameter set
2025-04-10 17:41:40: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), starting new job
2025-04-10 17:41:41: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/2 (65%/20), started new job
2025-04-10 17:41:41: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/2 (65%/20), starting parameter set
2025-04-10 17:41:42: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/2 (65%/20), starting new job
2025-04-10 17:41:43: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/3 (70%/20), started new job
2025-04-10 17:41:43: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/3 (70%/20), starting parameter set
2025-04-10 17:41:43: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/3 (70%/20), starting new job
2025-04-10 17:41:44: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/1 (75%/20), started new job
2025-04-10 17:41:45: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/1 (75%/20), starting parameter set
2025-04-10 17:41:45: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/1 (75%/20), starting new job
2025-04-10 17:41:46: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), started new job
2025-04-10 17:41:46: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), starting parameter set
2025-04-10 17:41:46: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), starting new job
2025-04-10 17:41:48: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/3 (85%/20), started new job
2025-04-10 17:41:48: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/3 (85%/20), starting parameter set
2025-04-10 17:41:48: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/3 (85%/20), starting new job
2025-04-10 17:41:49: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/1 (90%/20), started new job
2025-04-10 17:41:49: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/1 (90%/20), starting parameter set
2025-04-10 17:41:50: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/1 (90%/20), starting new job
2025-04-10 17:41:51: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/2 (95%/20), started new job
2025-04-10 17:41:51: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/2 (95%/20), starting parameter set
2025-04-10 17:41:51: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/2 (95%/20), starting new job
2025-04-10 17:41:52: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/3 (100%/20), started new job
2025-04-10 17:42:25: SobolGenerator, Failed jobs: 200 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 2/6/12 (100%/20), job_failed
2025-04-10 17:42:54: SobolGenerator, Failed jobs: 201 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 2/5/12 (95%/20), job_failed
2025-04-10 17:43:25: SobolGenerator, Failed jobs: 202 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 10/8 (90%/20), job_failed
2025-04-10 17:43:54: SobolGenerator, Failed jobs: 203 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 10/7 (85%/20), job_failed
2025-04-10 17:44:23: SobolGenerator, Failed jobs: 204 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 10/6 (80%/20), job_failed
2025-04-10 17:44:53: SobolGenerator, Failed jobs: 205 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/6 (75%/20), job_failed
2025-04-10 17:45:22: SobolGenerator, Failed jobs: 206 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/5 (70%/20), job_failed
2025-04-10 17:45:51: SobolGenerator, Failed jobs: 207 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/5 (65%/20), job_failed
2025-04-10 17:46:20: SobolGenerator, Failed jobs: 208 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/4 (60%/20), job_failed
2025-04-10 17:46:49: SobolGenerator, Failed jobs: 209 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/4 (55%/20), job_failed
2025-04-10 17:47:18: SobolGenerator, Failed jobs: 210 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/3 (50%/20), job_failed
2025-04-10 17:47:47: SobolGenerator, Failed jobs: 211 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/3 (45%/20), job_failed
2025-04-10 17:48:16: SobolGenerator, Failed jobs: 212 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/3 (40%/20), job_failed
2025-04-10 17:48:45: SobolGenerator, Failed jobs: 213 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/2 (35%/20), job_failed
2025-04-10 17:49:14: SobolGenerator, Failed jobs: 214 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/2 (30%/20), job_failed
2025-04-10 17:49:43: SobolGenerator, Failed jobs: 215 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/1 (25%/20), job_failed
2025-04-10 17:50:12: SobolGenerator, Failed jobs: 216 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/1 (20%/20), job_failed
2025-04-10 17:50:40: SobolGenerator, Failed jobs: 217 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), job_failed
2025-04-10 17:51:09: SobolGenerator, Failed jobs: 218 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 1/1 (10%/20), job_failed
2025-04-10 17:51:42: SobolGenerator, Failed jobs: 219 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 17:51:42: SobolGenerator, Failed jobs: 219 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), got 20, requested 20
2025-04-10 17:51:42: SobolGenerator, Failed jobs: 219 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 17:52:14: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 17:52:19: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 17:52:22: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 17:52:22: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 5s
2025-04-10 17:52:23: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 4s
2025-04-10 17:52:23: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 4s
2025-04-10 17:52:23: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 4s
2025-04-10 17:52:23: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 3s
2025-04-10 17:52:24: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 3s
2025-04-10 17:52:24: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 3s
2025-04-10 17:52:24: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 2s
2025-04-10 17:52:24: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 2s
2025-04-10 17:52:25: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 2s
2025-04-10 17:52:25: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 2s
2025-04-10 17:52:25: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 2s
2025-04-10 17:52:26: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 17:52:26: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 17:52:26: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 17:52:27: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 17:52:27: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 17:52:27: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 17:52:27: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 17:52:28: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 17:52:28: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 17:52:29: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 17:52:30: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 17:52:30: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 17:52:31: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 17:52:31: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 17:52:31: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 17:52:33: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), started new job
2025-04-10 17:52:33: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), starting parameter set
2025-04-10 17:52:33: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), starting new job
2025-04-10 17:52:34: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/2 (20%/20), started new job
2025-04-10 17:52:34: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/2 (20%/20), starting parameter set
2025-04-10 17:52:34: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/2 (20%/20), starting new job
2025-04-10 17:52:36: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/3 (25%/20), started new job
2025-04-10 17:52:36: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/3 (25%/20), starting parameter set
2025-04-10 17:52:36: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/3 (25%/20), starting new job
2025-04-10 17:52:37: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 5/1 (30%/20), started new job
2025-04-10 17:52:37: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 5/1 (30%/20), starting parameter set
2025-04-10 17:52:37: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 5/1 (30%/20), starting new job
2025-04-10 17:52:38: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/1 (35%/20), started new job
2025-04-10 17:52:38: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/1 (35%/20), starting parameter set
2025-04-10 17:52:39: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/1 (35%/20), starting new job
2025-04-10 17:52:40: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/1 (40%/20), started new job
2025-04-10 17:52:40: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/1 (40%/20), starting parameter set
2025-04-10 17:52:40: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/1 (40%/20), starting new job
2025-04-10 17:52:41: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/2 (45%/20), started new job
2025-04-10 17:52:41: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/2 (45%/20), starting parameter set
2025-04-10 17:52:42: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/2/2 (45%/20), starting new job
2025-04-10 17:52:43: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/1 (50%/20), started new job
2025-04-10 17:52:43: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/1 (50%/20), starting parameter set
2025-04-10 17:52:43: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/1 (50%/20), starting new job
2025-04-10 17:52:44: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/2 (55%/20), started new job
2025-04-10 17:52:44: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/2 (55%/20), starting parameter set
2025-04-10 17:52:45: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/2 (55%/20), starting new job
2025-04-10 17:52:46: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/3 (60%/20), started new job
2025-04-10 17:52:46: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/3 (60%/20), starting parameter set
2025-04-10 17:52:46: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/2/3 (60%/20), starting new job
2025-04-10 17:52:47: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/5/1 (65%/20), started new job
2025-04-10 17:52:47: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/5/1 (65%/20), starting parameter set
2025-04-10 17:52:48: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/5/1 (65%/20), starting new job
2025-04-10 17:52:49: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/5/2 (70%/20), started new job
2025-04-10 17:52:49: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/5/2 (70%/20), starting parameter set
2025-04-10 17:52:49: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/5/2 (70%/20), starting new job
2025-04-10 17:52:50: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/5/3 (75%/20), started new job
2025-04-10 17:52:50: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/5/3 (75%/20), starting parameter set
2025-04-10 17:52:51: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/5/3 (75%/20), starting new job
2025-04-10 17:52:52: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/1 (80%/20), started new job
2025-04-10 17:52:52: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/1 (80%/20), starting parameter set
2025-04-10 17:52:52: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/1 (80%/20), starting new job
2025-04-10 17:52:53: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/2 (85%/20), started new job
2025-04-10 17:52:53: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/2 (85%/20), starting parameter set
2025-04-10 17:52:54: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/2 (85%/20), starting new job
2025-04-10 17:52:55: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/3 (90%/20), started new job
2025-04-10 17:52:55: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/3 (90%/20), starting parameter set
2025-04-10 17:52:55: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/3 (90%/20), starting new job
2025-04-10 17:52:56: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/4 (95%/20), started new job
2025-04-10 17:52:56: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/4 (95%/20), starting parameter set
2025-04-10 17:52:57: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/8/4 (95%/20), starting new job
2025-04-10 17:52:58: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/12/1 (100%/20), started new job
2025-04-10 17:53:31: SobolGenerator, Failed jobs: 220 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed/out_of_memory 16/3/1 (100%/20), job_failed
2025-04-10 17:54:08: SobolGenerator, Failed jobs: 221 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 6/5/8 (95%/20), job_failed
2025-04-10 17:54:40: SobolGenerator, Failed jobs: 222 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 5/5/8 (90%/20), job_failed
2025-04-10 17:55:12: SobolGenerator, Failed jobs: 223 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 5/4/8 (85%/20), job_failed
2025-04-10 17:55:44: SobolGenerator, Failed jobs: 224 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/7 (80%/20), job_failed
2025-04-10 17:56:14: SobolGenerator, Failed jobs: 225 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/6 (75%/20), job_failed
2025-04-10 17:56:47: SobolGenerator, Failed jobs: 226 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/6 (70%/20), job_failed
2025-04-10 17:57:19: SobolGenerator, Failed jobs: 227 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/5 (65%/20), job_failed
2025-04-10 17:57:50: SobolGenerator, Failed jobs: 228 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/5 (60%/20), job_failed
2025-04-10 17:58:21: SobolGenerator, Failed jobs: 229 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/4 (55%/20), job_failed
2025-04-10 17:58:52: SobolGenerator, Failed jobs: 230 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/4 (50%/20), job_failed
2025-04-10 17:59:23: SobolGenerator, Failed jobs: 231 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/3 (45%/20), job_failed
2025-04-10 17:59:55: SobolGenerator, Failed jobs: 232 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/3 (40%/20), job_failed
2025-04-10 18:00:25: SobolGenerator, Failed jobs: 233 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/3 (35%/20), job_failed
2025-04-10 18:00:57: SobolGenerator, Failed jobs: 234 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/2 (30%/20), job_failed
2025-04-10 18:01:29: SobolGenerator, Failed jobs: 235 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/2 (25%/20), job_failed
2025-04-10 18:02:00: SobolGenerator, Failed jobs: 236 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/2 (20%/20), job_failed
2025-04-10 18:02:31: SobolGenerator, Failed jobs: 237 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), job_failed
2025-04-10 18:03:04: SobolGenerator, Failed jobs: 238 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 1/1 (10%/20), job_failed
2025-04-10 18:03:39: SobolGenerator, Failed jobs: 239 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 18:03:40: SobolGenerator, Failed jobs: 239 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), got 20, requested 20
2025-04-10 18:03:40: SobolGenerator, Failed jobs: 239 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 18:04:16: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 18:04:21: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 18:04:25: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 18:04:25: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 5s
2025-04-10 18:04:25: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 4s
2025-04-10 18:04:26: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 4s
2025-04-10 18:04:26: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 4s
2025-04-10 18:04:26: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 4s
2025-04-10 18:04:27: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 3s
2025-04-10 18:04:27: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 3s
2025-04-10 18:04:27: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 3s
2025-04-10 18:04:27: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 2s
2025-04-10 18:04:28: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 2s
2025-04-10 18:04:28: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 2s
2025-04-10 18:04:28: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 2s
2025-04-10 18:04:29: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 1s
2025-04-10 18:04:29: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 18:04:29: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 18:04:30: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 18:04:30: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 18:04:30: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 18:04:30: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 18:04:31: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 18:04:31: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 18:04:32: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 18:04:32: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 18:04:33: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 18:04:34: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), started new job
2025-04-10 18:04:34: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting parameter set
2025-04-10 18:04:34: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting new job
2025-04-10 18:04:35: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), started new job
2025-04-10 18:04:36: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), starting parameter set
2025-04-10 18:04:36: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), starting new job
2025-04-10 18:04:37: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), started new job
2025-04-10 18:04:37: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), starting parameter set
2025-04-10 18:04:37: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), starting new job
2025-04-10 18:04:38: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/1 (25%/20), started new job
2025-04-10 18:04:38: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/1 (25%/20), starting parameter set
2025-04-10 18:04:39: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/1 (25%/20), starting new job
2025-04-10 18:04:40: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/2 (30%/20), started new job
2025-04-10 18:04:40: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/2 (30%/20), starting parameter set
2025-04-10 18:04:40: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/2 (30%/20), starting new job
2025-04-10 18:04:41: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/3 (35%/20), started new job
2025-04-10 18:04:41: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/3 (35%/20), starting parameter set
2025-04-10 18:04:42: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/3 (35%/20), starting new job
2025-04-10 18:04:43: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/4 (40%/20), started new job
2025-04-10 18:04:43: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/4 (40%/20), starting parameter set
2025-04-10 18:04:43: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 4/4 (40%/20), starting new job
2025-04-10 18:04:44: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/1 (45%/20), started new job
2025-04-10 18:04:44: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/1 (45%/20), starting parameter set
2025-04-10 18:04:45: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/1 (45%/20), starting new job
2025-04-10 18:04:46: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/2 (50%/20), started new job
2025-04-10 18:04:46: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/2 (50%/20), starting parameter set
2025-04-10 18:04:46: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/2 (50%/20), starting new job
2025-04-10 18:04:47: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/3 (55%/20), started new job
2025-04-10 18:04:47: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/3 (55%/20), starting parameter set
2025-04-10 18:04:48: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/4/3 (55%/20), starting new job
2025-04-10 18:04:49: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/7/1 (60%/20), started new job
2025-04-10 18:04:49: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/7/1 (60%/20), starting parameter set
2025-04-10 18:04:49: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/7/1 (60%/20), starting new job
2025-04-10 18:04:50: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/7/2 (65%/20), started new job
2025-04-10 18:04:50: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/7/2 (65%/20), starting parameter set
2025-04-10 18:04:51: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/7/2 (65%/20), starting new job
2025-04-10 18:04:52: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/7/3 (70%/20), started new job
2025-04-10 18:04:52: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/7/3 (70%/20), starting parameter set
2025-04-10 18:04:52: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/7/3 (70%/20), starting new job
2025-04-10 18:04:53: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 14/1 (75%/20), started new job
2025-04-10 18:04:53: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 14/1 (75%/20), starting parameter set
2025-04-10 18:04:54: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 14/1 (75%/20), starting new job
2025-04-10 18:04:55: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/1 (80%/20), started new job
2025-04-10 18:04:55: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/1 (80%/20), starting parameter set
2025-04-10 18:04:55: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/1 (80%/20), starting new job
2025-04-10 18:04:56: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/2 (85%/20), started new job
2025-04-10 18:04:57: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/2 (85%/20), starting parameter set
2025-04-10 18:04:57: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/2 (85%/20), starting new job
2025-04-10 18:04:58: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/3 (90%/20), started new job
2025-04-10 18:04:58: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/3 (90%/20), starting parameter set
2025-04-10 18:04:58: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/3 (90%/20), starting new job
2025-04-10 18:05:00: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/4/1 (95%/20), started new job
2025-04-10 18:05:00: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/4/1 (95%/20), starting parameter set
2025-04-10 18:05:00: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/4/1 (95%/20), starting new job
2025-04-10 18:05:01: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/4/2 (100%/20), started new job
2025-04-10 18:05:40: SobolGenerator, Failed jobs: 240 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 3/6/11 (100%/20), job_failed
2025-04-10 18:06:14: SobolGenerator, Failed jobs: 241 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 3/5/11 (95%/20), job_failed
2025-04-10 18:06:49: SobolGenerator, Failed jobs: 242 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/10 (90%/20), job_failed
2025-04-10 18:07:23: SobolGenerator, Failed jobs: 243 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/10 (85%/20), job_failed
2025-04-10 18:07:58: SobolGenerator, Failed jobs: 244 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/9 (80%/20), job_failed
2025-04-10 18:08:32: SobolGenerator, Failed jobs: 245 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/9 (75%/20), job_failed
2025-04-10 18:09:07: SobolGenerator, Failed jobs: 246 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/8 (70%/20), job_failed
2025-04-10 18:09:40: SobolGenerator, Failed jobs: 247 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/8 (65%/20), job_failed
2025-04-10 18:10:16: SobolGenerator, Failed jobs: 248 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/7 (60%/20), job_failed
2025-04-10 18:10:52: SobolGenerator, Failed jobs: 249 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/6 (55%/20), job_failed
2025-04-10 18:11:27: SobolGenerator, Failed jobs: 250 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/5 (50%/20), job_failed
2025-04-10 18:12:02: SobolGenerator, Failed jobs: 251 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/4 (45%/20), job_failed
2025-04-10 18:12:36: SobolGenerator, Failed jobs: 252 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/3 (40%/20), job_failed
2025-04-10 18:13:38: SobolGenerator, Failed jobs: 253 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/2 (35%/20), job_failed
2025-04-10 18:14:17: SobolGenerator, Failed jobs: 254 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/2 (30%/20), job_failed
2025-04-10 18:14:50: SobolGenerator, Failed jobs: 255 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/2 (25%/20), job_failed
2025-04-10 18:15:24: SobolGenerator, Failed jobs: 256 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/1 (20%/20), job_failed
2025-04-10 18:15:57: SobolGenerator, Failed jobs: 257 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), job_failed
2025-04-10 18:16:29: SobolGenerator, Failed jobs: 258 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 2 (10%/20), job_failed
2025-04-10 18:17:06: SobolGenerator, Failed jobs: 259 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 18:17:06: SobolGenerator, Failed jobs: 259 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), got 20, requested 20
2025-04-10 18:17:06: SobolGenerator, Failed jobs: 259 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 18:17:43: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 18:17:49: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 18:17:53: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 18:17:54: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 5s
2025-04-10 18:17:54: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 4s
2025-04-10 18:17:54: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 5s
2025-04-10 18:17:55: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 4s
2025-04-10 18:17:55: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 4s
2025-04-10 18:17:55: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 4s
2025-04-10 18:17:55: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 3s
2025-04-10 18:17:56: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 3s
2025-04-10 18:17:56: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 3s
2025-04-10 18:17:57: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 3s
2025-04-10 18:17:57: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 2s
2025-04-10 18:17:57: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 2s
2025-04-10 18:17:57: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 18:17:58: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 18:17:58: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 18:17:58: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 18:17:59: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 18:17:59: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 18:17:59: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 18:18:00: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 18:18:00: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 18:18:01: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 18:18:01: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 18:18:02: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 18:18:03: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), started new job
2025-04-10 18:18:03: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting parameter set
2025-04-10 18:18:03: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting new job
2025-04-10 18:18:04: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), started new job
2025-04-10 18:18:04: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting parameter set
2025-04-10 18:18:05: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting new job
2025-04-10 18:18:06: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/1 (20%/20), started new job
2025-04-10 18:18:06: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/1 (20%/20), starting parameter set
2025-04-10 18:18:06: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/1 (20%/20), starting new job
2025-04-10 18:18:07: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/2 (25%/20), started new job
2025-04-10 18:18:07: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/2 (25%/20), starting parameter set
2025-04-10 18:18:08: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/2 (25%/20), starting new job
2025-04-10 18:18:09: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/1 (30%/20), started new job
2025-04-10 18:18:09: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/1 (30%/20), starting parameter set
2025-04-10 18:18:09: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/1 (30%/20), starting new job
2025-04-10 18:18:10: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/1 (35%/20), started new job
2025-04-10 18:18:11: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/1 (35%/20), starting parameter set
2025-04-10 18:18:11: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/1 (35%/20), starting new job
2025-04-10 18:18:12: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/2 (40%/20), started new job
2025-04-10 18:18:12: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/2 (40%/20), starting parameter set
2025-04-10 18:18:12: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/2 (40%/20), starting new job
2025-04-10 18:18:13: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/3 (45%/20), started new job
2025-04-10 18:18:13: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/3 (45%/20), starting parameter set
2025-04-10 18:18:14: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/3 (45%/20), starting new job
2025-04-10 18:18:15: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/4 (50%/20), started new job
2025-04-10 18:18:15: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/4 (50%/20), starting parameter set
2025-04-10 18:18:15: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/1/4 (50%/20), starting new job
2025-04-10 18:18:16: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/1 (55%/20), started new job
2025-04-10 18:18:16: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/1 (55%/20), starting parameter set
2025-04-10 18:18:17: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/1 (55%/20), starting new job
2025-04-10 18:18:18: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/2 (60%/20), started new job
2025-04-10 18:18:18: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/2 (60%/20), starting parameter set
2025-04-10 18:18:18: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/2 (60%/20), starting new job
2025-04-10 18:18:19: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/3 (65%/20), started new job
2025-04-10 18:18:19: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/3 (65%/20), starting parameter set
2025-04-10 18:18:20: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/5/3 (65%/20), starting new job
2025-04-10 18:18:21: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/8/1 (70%/20), started new job
2025-04-10 18:18:21: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/8/1 (70%/20), starting parameter set
2025-04-10 18:18:21: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 5/8/1 (70%/20), starting new job
2025-04-10 18:18:22: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/1/1 (75%/20), started new job
2025-04-10 18:18:22: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/1/1 (75%/20), starting parameter set
2025-04-10 18:18:23: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/1/1 (75%/20), starting new job
2025-04-10 18:18:24: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/1/2 (80%/20), started new job
2025-04-10 18:18:24: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/1/2 (80%/20), starting parameter set
2025-04-10 18:18:24: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/1/2 (80%/20), starting new job
2025-04-10 18:18:25: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/1 (85%/20), started new job
2025-04-10 18:18:25: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/1 (85%/20), starting parameter set
2025-04-10 18:18:26: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/1 (85%/20), starting new job
2025-04-10 18:18:27: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/2 (90%/20), started new job
2025-04-10 18:18:27: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/2 (90%/20), starting parameter set
2025-04-10 18:18:27: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/2 (90%/20), starting new job
2025-04-10 18:18:30: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/3 (95%/20), started new job
2025-04-10 18:18:30: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/3 (95%/20), starting parameter set
2025-04-10 18:18:30: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/3 (95%/20), starting new job
2025-04-10 18:18:31: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/5/1 (100%/20), started new job
2025-04-10 18:19:09: SobolGenerator, Failed jobs: 260 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 2/3/15 (100%/20), job_failed
2025-04-10 18:19:45: SobolGenerator, Failed jobs: 261 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/running/completed 2/15/2 (95%/20), job_failed
2025-04-10 18:20:20: SobolGenerator, Failed jobs: 262 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/11 (90%/20), job_failed
2025-04-10 18:20:57: SobolGenerator, Failed jobs: 263 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/10 (85%/20), job_failed
2025-04-10 18:21:31: SobolGenerator, Failed jobs: 264 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/10 (80%/20), job_failed
2025-04-10 18:22:06: SobolGenerator, Failed jobs: 265 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/10 (75%/20), job_failed
2025-04-10 18:22:41: SobolGenerator, Failed jobs: 266 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/9 (70%/20), job_failed
2025-04-10 18:23:16: SobolGenerator, Failed jobs: 267 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/9 (65%/20), job_failed
2025-04-10 18:23:54: SobolGenerator, Failed jobs: 268 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/8 (60%/20), job_failed
2025-04-10 18:24:28: SobolGenerator, Failed jobs: 269 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/7 (55%/20), job_failed
2025-04-10 18:25:05: SobolGenerator, Failed jobs: 270 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/6 (50%/20), job_failed
2025-04-10 18:25:43: SobolGenerator, Failed jobs: 271 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/5 (45%/20), job_failed
2025-04-10 18:26:26: SobolGenerator, Failed jobs: 272 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/5 (40%/20), job_failed
2025-04-10 18:27:44: SobolGenerator, Failed jobs: 273 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/4 (35%/20), job_failed
2025-04-10 18:32:49: SobolGenerator, Failed jobs: 274 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/4 (30%/20), job_failed
2025-04-10 18:39:06: SobolGenerator, Failed jobs: 275 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/3 (25%/20), job_failed
2025-04-10 18:45:55: SobolGenerator, Failed jobs: 276 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 1/3 (20%/20), job_failed
2025-04-10 18:51:42: SobolGenerator, Failed jobs: 277 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 1/2 (15%/20), job_failed
2025-04-10 18:57:03: SobolGenerator, Failed jobs: 278 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 1/1 (10%/20), job_failed
2025-04-10 19:03:04: SobolGenerator, Failed jobs: 279 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 19:03:05: SobolGenerator, Failed jobs: 279 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), got 20, requested 20
2025-04-10 19:03:06: SobolGenerator, Failed jobs: 279 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 19:07:22: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 19:07:32: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 19:07:40: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 19:07:41: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 13s
2025-04-10 19:07:42: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 10s
2025-04-10 19:07:43: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 10s
2025-04-10 19:07:44: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 9s
2025-04-10 19:07:45: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 8s
2025-04-10 19:07:46: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 8s
2025-04-10 19:07:47: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 7s
2025-04-10 19:07:48: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 6s
2025-04-10 19:07:49: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 6s
2025-04-10 19:07:51: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 5s
2025-04-10 19:07:52: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 5s
2025-04-10 19:07:53: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 4s
2025-04-10 19:07:54: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 4s
2025-04-10 19:07:55: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 3s
2025-04-10 19:07:57: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 3s
2025-04-10 19:07:58: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 2s
2025-04-10 19:07:59: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 19:08:01: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 1s
2025-04-10 19:08:02: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 19:08:04: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 19:08:05: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 19:08:12: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 1 (5%/20), started new job
2025-04-10 19:08:13: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 1 (5%/20), starting parameter set
2025-04-10 19:08:15: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 1 (5%/20), starting new job
2025-04-10 19:08:20: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), started new job
2025-04-10 19:08:21: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting parameter set
2025-04-10 19:08:23: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 1/1 (10%/20), starting new job
2025-04-10 19:08:27: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 1/2 (15%/20), started new job
2025-04-10 19:08:28: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 1/2 (15%/20), starting parameter set
2025-04-10 19:08:29: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 1/2 (15%/20), starting new job
2025-04-10 19:08:34: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 3/1 (20%/20), started new job
2025-04-10 19:08:36: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 3/1 (20%/20), starting parameter set
2025-04-10 19:08:37: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 3/1 (20%/20), starting new job
2025-04-10 19:08:41: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 4/1 (25%/20), started new job
2025-04-10 19:08:43: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 4/1 (25%/20), starting parameter set
2025-04-10 19:08:45: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 4/1 (25%/20), starting new job
2025-04-10 19:08:48: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), started new job
2025-04-10 19:08:49: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 4/2 (30%/20), starting parameter set
2025-04-10 19:08:50: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 4/2 (30%/20), starting new job
2025-04-10 19:08:54: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 6/1 (35%/20), started new job
2025-04-10 19:08:55: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 6/1 (35%/20), starting parameter set
2025-04-10 19:08:55: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 6/1 (35%/20), starting new job
2025-04-10 19:08:58: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/1/1 (40%/20), started new job
2025-04-10 19:08:58: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 6/2 (40%/20), starting parameter set
2025-04-10 19:08:59: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 6/2 (40%/20), starting new job
2025-04-10 19:09:02: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/2/1 (45%/20), started new job
2025-04-10 19:09:03: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 6/3 (45%/20), starting parameter set
2025-04-10 19:09:04: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 6/3 (45%/20), starting new job
2025-04-10 19:09:06: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/1 (50%/20), started new job
2025-04-10 19:09:10: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 10 (50%/20), starting parameter set
2025-04-10 19:09:11: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 10 (50%/20), starting new job
2025-04-10 19:09:14: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/1 (55%/20), started new job
2025-04-10 19:09:15: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/1 (55%/20), starting parameter set
2025-04-10 19:09:16: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/1 (55%/20), starting new job
2025-04-10 19:09:21: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), started new job
2025-04-10 19:09:23: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), starting parameter set
2025-04-10 19:09:24: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), starting new job
2025-04-10 19:09:28: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/3 (65%/20), started new job
2025-04-10 19:09:30: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/3 (65%/20), starting parameter set
2025-04-10 19:09:31: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 13 (65%/20), starting new job
2025-04-10 19:09:36: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 13/1 (70%/20), started new job
2025-04-10 19:09:37: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 13/1 (70%/20), starting parameter set
2025-04-10 19:09:39: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running 14 (70%/20), starting new job
2025-04-10 19:09:42: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 14/1 (75%/20), started new job
2025-04-10 19:09:44: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 14/1 (75%/20), starting parameter set
2025-04-10 19:09:46: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 14/1 (75%/20), starting new job
2025-04-10 19:09:47: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/1/1 (80%/20), started new job
2025-04-10 19:09:49: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 14/2 (80%/20), starting parameter set
2025-04-10 19:09:49: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 14/2 (80%/20), starting new job
2025-04-10 19:09:53: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed/unknown 15/1/1 (85%/20), started new job
2025-04-10 19:09:54: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed/pending 15/1/1 (85%/20), starting parameter set
2025-04-10 19:09:55: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed/pending 15/1/1 (85%/20), starting new job
2025-04-10 19:09:59: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed/pending 15/1/2 (90%/20), started new job
2025-04-10 19:10:01: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed/pending 15/1/2 (90%/20), starting parameter set
2025-04-10 19:10:02: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/completed/pending 15/1/2 (90%/20), starting new job
2025-04-10 19:10:06: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running/pending/unknown 1/2/13/2/1 (95%/20), started new job
2025-04-10 19:10:08: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running/pending/unknown 1/2/13/2/1 (95%/20), starting parameter set
2025-04-10 19:10:13: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 2/2/15 (95%/20), starting new job
2025-04-10 19:10:18: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running/pending 2/2/15/1 (100%/20), started new job
2025-04-10 19:10:19: SobolGenerator, Failed jobs: 280 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running/pending 2/2/15/1 (100%/20), job_failed
2025-04-10 19:14:19: SobolGenerator, Failed jobs: 281 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 12/7 (95%/20), job_failed
2025-04-10 19:18:10: SobolGenerator, Failed jobs: 282 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/11 (90%/20), job_failed
2025-04-10 19:22:19: SobolGenerator, Failed jobs: 283 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/6 (85%/20), job_failed
2025-04-10 19:26:03: SobolGenerator, Failed jobs: 284 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/6 (80%/20), job_failed
2025-04-10 19:29:50: SobolGenerator, Failed jobs: 285 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/9 (75%/20), job_failed
2025-04-10 19:33:35: SobolGenerator, Failed jobs: 286 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/5 (70%/20), job_failed
2025-04-10 19:36:23: SobolGenerator, Failed jobs: 287 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/8 (65%/20), job_failed
2025-04-10 19:39:12: SobolGenerator, Failed jobs: 288 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/4 (60%/20), job_failed
2025-04-10 19:42:16: SobolGenerator, Failed jobs: 289 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/4 (55%/20), job_failed
2025-04-10 19:45:12: SobolGenerator, Failed jobs: 290 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/6 (50%/20), job_failed
2025-04-10 19:47:28: SobolGenerator, Failed jobs: 291 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/3 (45%/20), job_failed
2025-04-10 19:49:14: SobolGenerator, Failed jobs: 292 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/3 (40%/20), job_failed
2025-04-10 19:50:30: SobolGenerator, Failed jobs: 293 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/4 (35%/20), job_failed
2025-04-10 19:51:48: SobolGenerator, Failed jobs: 294 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/2 (30%/20), job_failed
2025-04-10 19:52:47: SobolGenerator, Failed jobs: 295 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/3 (25%/20), job_failed
2025-04-10 19:53:37: SobolGenerator, Failed jobs: 296 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 1/3 (20%/20), job_failed
2025-04-10 19:54:29: SobolGenerator, Failed jobs: 297 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 3 (15%/20), job_failed
2025-04-10 19:55:20: SobolGenerator, Failed jobs: 298 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 19:56:13: SobolGenerator, Failed jobs: 299 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 19:57:34: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing jobs after starting them, finished 20 jobs
2025-04-10 19:57:34: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), got 20, requested 20
2025-04-10 19:57:48: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 19:57:54: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 19:57:55: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 6s
2025-04-10 19:57:55: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 7s
2025-04-10 19:57:56: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 6s
2025-04-10 19:57:56: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 5s
2025-04-10 19:57:56: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 5s
2025-04-10 19:57:57: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 4s
2025-04-10 19:57:57: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 4s
2025-04-10 19:57:58: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 4s
2025-04-10 19:57:58: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 3s
2025-04-10 19:57:58: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 3s
2025-04-10 19:57:59: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 3s
2025-04-10 19:57:59: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 2s
2025-04-10 19:58:00: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 19:58:00: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 2s
2025-04-10 19:58:00: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 19:58:01: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 19:58:01: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 19:58:02: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 19:58:02: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 19:58:02: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 19:58:03: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 19:58:06: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 1 (5%/20), started new job
2025-04-10 19:58:06: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 1 (5%/20), starting parameter set
2025-04-10 19:58:06: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 1 (5%/20), starting new job
2025-04-10 19:58:08: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), started new job
2025-04-10 19:58:08: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting parameter set
2025-04-10 19:58:08: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting new job
2025-04-10 19:58:09: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/1 (15%/20), started new job
2025-04-10 19:58:10: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/1 (15%/20), starting parameter set
2025-04-10 19:58:10: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/1 (15%/20), starting new job
2025-04-10 19:58:11: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/2 (20%/20), started new job
2025-04-10 19:58:12: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/2 (20%/20), starting parameter set
2025-04-10 19:58:12: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/2 (20%/20), starting new job
2025-04-10 19:58:13: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/1 (25%/20), started new job
2025-04-10 19:58:14: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/1 (25%/20), starting parameter set
2025-04-10 19:58:14: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/1 (25%/20), starting new job
2025-04-10 19:58:15: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/2 (30%/20), started new job
2025-04-10 19:58:16: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/2 (30%/20), starting parameter set
2025-04-10 19:58:16: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/2/2 (30%/20), starting new job
2025-04-10 19:58:18: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/4/1 (35%/20), started new job
2025-04-10 19:58:18: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/4/1 (35%/20), starting parameter set
2025-04-10 19:58:18: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/4/1 (35%/20), starting new job
2025-04-10 19:58:19: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/4/2 (40%/20), started new job
2025-04-10 19:58:20: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/4/2 (40%/20), starting parameter set
2025-04-10 19:58:20: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/4/2 (40%/20), starting new job
2025-04-10 19:58:21: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/4/3 (45%/20), started new job
2025-04-10 19:58:22: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/4/3 (45%/20), starting parameter set
2025-04-10 19:58:22: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/4/3 (45%/20), starting new job
2025-04-10 19:58:23: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/7/1 (50%/20), started new job
2025-04-10 19:58:24: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/7/1 (50%/20), starting parameter set
2025-04-10 19:58:24: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/7/1 (50%/20), starting new job
2025-04-10 19:58:26: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 9/1/1 (55%/20), started new job
2025-04-10 19:58:26: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 9/1/1 (55%/20), starting parameter set
2025-04-10 19:58:26: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 9/1/1 (55%/20), starting new job
2025-04-10 19:58:27: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), started new job
2025-04-10 19:58:28: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), starting parameter set
2025-04-10 19:58:28: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/1 (60%/20), starting new job
2025-04-10 19:58:29: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/2 (65%/20), started new job
2025-04-10 19:58:30: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/2 (65%/20), starting parameter set
2025-04-10 19:58:30: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/2 (65%/20), starting new job
2025-04-10 19:58:31: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/3 (70%/20), started new job
2025-04-10 19:58:32: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/3 (70%/20), starting parameter set
2025-04-10 19:58:32: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/1/3 (70%/20), starting new job
2025-04-10 19:58:33: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/1 (75%/20), started new job
2025-04-10 19:58:34: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/1 (75%/20), starting parameter set
2025-04-10 19:58:34: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/1 (75%/20), starting new job
2025-04-10 19:58:35: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), started new job
2025-04-10 19:58:36: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), starting parameter set
2025-04-10 19:58:36: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), starting new job
2025-04-10 19:58:37: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/7 (85%/20), started new job
2025-04-10 19:58:37: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/7 (85%/20), starting parameter set
2025-04-10 19:58:38: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/7 (85%/20), starting new job
2025-04-10 19:58:39: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/1 (90%/20), started new job
2025-04-10 19:58:39: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/1 (90%/20), starting parameter set
2025-04-10 19:58:39: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/1 (90%/20), starting new job
2025-04-10 19:58:41: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/2 (95%/20), started new job
2025-04-10 19:58:41: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/2 (95%/20), starting parameter set
2025-04-10 19:58:41: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 17/2 (95%/20), starting new job
2025-04-10 19:58:44: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 17/2/1 (100%/20), started new job
2025-04-10 19:59:36: SobolGenerator, Failed jobs: 300 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 5/1/14 (100%/20), job_failed
2025-04-10 20:00:27: SobolGenerator, Failed jobs: 301 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 4/1/14 (95%/20), job_failed
2025-04-10 20:01:18: SobolGenerator, Failed jobs: 302 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/7 (90%/20), job_failed
2025-04-10 20:02:10: SobolGenerator, Failed jobs: 303 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/7 (85%/20), job_failed
2025-04-10 20:02:59: SobolGenerator, Failed jobs: 304 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/7 (80%/20), job_failed
2025-04-10 20:03:46: SobolGenerator, Failed jobs: 305 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/6 (75%/20), job_failed
2025-04-10 20:04:35: SobolGenerator, Failed jobs: 306 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/6 (70%/20), job_failed
2025-04-10 20:05:23: SobolGenerator, Failed jobs: 307 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/5 (65%/20), job_failed
2025-04-10 20:06:15: SobolGenerator, Failed jobs: 308 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/5 (60%/20), job_failed
2025-04-10 20:07:03: SobolGenerator, Failed jobs: 309 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/4 (55%/20), job_failed
2025-04-10 20:07:54: SobolGenerator, Failed jobs: 310 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/4 (50%/20), job_failed
2025-04-10 20:09:06: SobolGenerator, Failed jobs: 311 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/4 (45%/20), job_failed
2025-04-10 20:09:50: SobolGenerator, Failed jobs: 312 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/4 (40%/20), job_failed
2025-04-10 20:10:36: SobolGenerator, Failed jobs: 313 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/3 (35%/20), job_failed
2025-04-10 20:11:19: SobolGenerator, Failed jobs: 314 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/3 (30%/20), job_failed
2025-04-10 20:12:05: SobolGenerator, Failed jobs: 315 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/2 (25%/20), job_failed
2025-04-10 20:12:48: SobolGenerator, Failed jobs: 316 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/1 (20%/20), job_failed
2025-04-10 20:13:37: SobolGenerator, Failed jobs: 317 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/1 (15%/20), job_failed
2025-04-10 20:14:21: SobolGenerator, Failed jobs: 318 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 20:15:10: SobolGenerator, Failed jobs: 319 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 20:15:10: SobolGenerator, Failed jobs: 319 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), got 20, requested 20
2025-04-10 20:15:10: SobolGenerator, Failed jobs: 319 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 20:15:58: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 20:16:06: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 20:16:13: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 20:16:13: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 6s
2025-04-10 20:16:14: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 5s
2025-04-10 20:16:14: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 5s
2025-04-10 20:16:14: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 5s
2025-04-10 20:16:15: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 4s
2025-04-10 20:16:15: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 4s
2025-04-10 20:16:15: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 4s
2025-04-10 20:16:16: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 3s
2025-04-10 20:16:16: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 3s
2025-04-10 20:16:16: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 3s
2025-04-10 20:16:17: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 2s
2025-04-10 20:16:17: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 2s
2025-04-10 20:16:18: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 20:16:18: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 1s
2025-04-10 20:16:18: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 20:16:19: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 20:16:19: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 0s
2025-04-10 20:16:19: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 20:16:20: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 20:16:20: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 20:16:20: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 20:16:22: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 20:16:22: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 20:16:22: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 20:16:23: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), started new job
2025-04-10 20:16:23: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting parameter set
2025-04-10 20:16:24: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/1 (10%/20), starting new job
2025-04-10 20:16:25: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), started new job
2025-04-10 20:16:25: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), starting parameter set
2025-04-10 20:16:25: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/2 (15%/20), starting new job
2025-04-10 20:16:26: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), started new job
2025-04-10 20:16:27: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), starting parameter set
2025-04-10 20:16:27: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 1/3 (20%/20), starting new job
2025-04-10 20:16:28: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), started new job
2025-04-10 20:16:28: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), starting parameter set
2025-04-10 20:16:28: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), starting new job
2025-04-10 20:16:29: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), started new job
2025-04-10 20:16:29: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), starting parameter set
2025-04-10 20:16:30: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), starting new job
2025-04-10 20:16:31: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/3 (35%/20), started new job
2025-04-10 20:16:31: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/3 (35%/20), starting parameter set
2025-04-10 20:16:31: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/3 (35%/20), starting new job
2025-04-10 20:16:32: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/4 (40%/20), started new job
2025-04-10 20:16:32: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/4 (40%/20), starting parameter set
2025-04-10 20:16:33: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/4 (40%/20), starting new job
2025-04-10 20:16:34: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/1 (45%/20), started new job
2025-04-10 20:16:34: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/1 (45%/20), starting parameter set
2025-04-10 20:16:34: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/1 (45%/20), starting new job
2025-04-10 20:16:35: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/2 (50%/20), started new job
2025-04-10 20:16:35: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/2 (50%/20), starting parameter set
2025-04-10 20:16:36: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/2 (50%/20), starting new job
2025-04-10 20:16:37: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/3 (55%/20), started new job
2025-04-10 20:16:37: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/3 (55%/20), starting parameter set
2025-04-10 20:16:37: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/3 (55%/20), starting new job
2025-04-10 20:16:38: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/1 (60%/20), started new job
2025-04-10 20:16:38: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/1 (60%/20), starting parameter set
2025-04-10 20:16:39: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/1 (60%/20), starting new job
2025-04-10 20:16:40: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/2 (65%/20), started new job
2025-04-10 20:16:40: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/2 (65%/20), starting parameter set
2025-04-10 20:16:40: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/2 (65%/20), starting new job
2025-04-10 20:16:41: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/3 (70%/20), started new job
2025-04-10 20:16:41: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/3 (70%/20), starting parameter set
2025-04-10 20:16:42: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 11/3 (70%/20), starting new job
2025-04-10 20:16:43: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/1 (75%/20), started new job
2025-04-10 20:16:43: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/1 (75%/20), starting parameter set
2025-04-10 20:16:43: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/1 (75%/20), starting new job
2025-04-10 20:16:44: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/2 (80%/20), started new job
2025-04-10 20:16:45: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/2 (80%/20), starting parameter set
2025-04-10 20:16:45: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/2 (80%/20), starting new job
2025-04-10 20:16:47: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/3 (85%/20), started new job
2025-04-10 20:16:47: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/3 (85%/20), starting parameter set
2025-04-10 20:16:47: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 14/3 (85%/20), starting new job
2025-04-10 20:16:48: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/1 (90%/20), started new job
2025-04-10 20:16:48: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/1 (90%/20), starting parameter set
2025-04-10 20:16:49: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/1 (90%/20), starting new job
2025-04-10 20:16:50: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/2 (95%/20), started new job
2025-04-10 20:16:50: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/2 (95%/20), starting parameter set
2025-04-10 20:16:50: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/2 (95%/20), starting new job
2025-04-10 20:16:51: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/3 (100%/20), started new job
2025-04-10 20:17:34: SobolGenerator, Failed jobs: 320 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 2/2/16 (100%/20), job_failed
2025-04-10 20:18:20: SobolGenerator, Failed jobs: 321 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 10/9 (95%/20), job_failed
2025-04-10 20:19:04: SobolGenerator, Failed jobs: 322 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/9 (90%/20), job_failed
2025-04-10 20:19:50: SobolGenerator, Failed jobs: 323 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/8 (85%/20), job_failed
2025-04-10 20:20:36: SobolGenerator, Failed jobs: 324 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/7 (80%/20), job_failed
2025-04-10 20:21:20: SobolGenerator, Failed jobs: 325 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/7 (75%/20), job_failed
2025-04-10 20:22:19: SobolGenerator, Failed jobs: 326 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/6 (70%/20), job_failed
2025-04-10 20:23:02: SobolGenerator, Failed jobs: 327 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/6 (65%/20), job_failed
2025-04-10 20:23:47: SobolGenerator, Failed jobs: 328 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/6 (60%/20), job_failed
2025-04-10 20:24:33: SobolGenerator, Failed jobs: 329 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/5 (55%/20), job_failed
2025-04-10 20:25:18: SobolGenerator, Failed jobs: 330 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/5 (50%/20), job_failed
2025-04-10 20:26:02: SobolGenerator, Failed jobs: 331 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/5 (45%/20), job_failed
2025-04-10 20:26:47: SobolGenerator, Failed jobs: 332 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/5 (40%/20), job_failed
2025-04-10 20:27:31: SobolGenerator, Failed jobs: 333 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/4 (35%/20), job_failed
2025-04-10 20:28:25: SobolGenerator, Failed jobs: 334 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/3 (30%/20), job_failed
2025-04-10 20:29:09: SobolGenerator, Failed jobs: 335 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/2 (25%/20), job_failed
2025-04-10 20:29:54: SobolGenerator, Failed jobs: 336 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/1 (20%/20), job_failed
2025-04-10 20:30:37: SobolGenerator, Failed jobs: 337 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), job_failed
2025-04-10 20:31:22: SobolGenerator, Failed jobs: 338 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 1/1 (10%/20), job_failed
2025-04-10 20:32:16: SobolGenerator, Failed jobs: 339 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 20:32:16: SobolGenerator, Failed jobs: 339 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), got 20, requested 20
2025-04-10 20:32:16: SobolGenerator, Failed jobs: 339 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 20:33:08: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 20:33:17: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 20:33:25: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 20:33:25: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 6s
2025-04-10 20:33:26: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 5s
2025-04-10 20:33:26: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 5s
2025-04-10 20:33:26: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 5s
2025-04-10 20:33:27: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 5s
2025-04-10 20:33:27: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 4s
2025-04-10 20:33:27: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 4s
2025-04-10 20:33:28: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 4s
2025-04-10 20:33:28: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 3s
2025-04-10 20:33:29: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 3s
2025-04-10 20:33:29: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 3s
2025-04-10 20:33:29: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 2s
2025-04-10 20:33:30: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 20:33:30: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 2s
2025-04-10 20:33:30: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 20:33:31: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 20:33:31: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 20:33:32: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 20:33:32: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 20:33:32: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 20:33:33: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 20:33:34: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 20:33:34: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 20:33:34: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 20:33:35: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 20:33:36: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 20:33:36: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 20:33:37: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), started new job
2025-04-10 20:33:37: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), starting parameter set
2025-04-10 20:33:37: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), starting new job
2025-04-10 20:33:38: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/1 (20%/20), started new job
2025-04-10 20:33:38: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/1 (20%/20), starting parameter set
2025-04-10 20:33:39: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/1 (20%/20), starting new job
2025-04-10 20:33:40: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/2 (25%/20), started new job
2025-04-10 20:33:40: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/2 (25%/20), starting parameter set
2025-04-10 20:33:40: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/2 (25%/20), starting new job
2025-04-10 20:33:41: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/3 (30%/20), started new job
2025-04-10 20:33:41: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/3 (30%/20), starting parameter set
2025-04-10 20:33:42: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/3 (30%/20), starting new job
2025-04-10 20:33:43: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/1 (35%/20), started new job
2025-04-10 20:33:43: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/1 (35%/20), starting parameter set
2025-04-10 20:33:43: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/1 (35%/20), starting new job
2025-04-10 20:33:44: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/2 (40%/20), started new job
2025-04-10 20:33:44: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/2 (40%/20), starting parameter set
2025-04-10 20:33:45: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/2 (40%/20), starting new job
2025-04-10 20:33:46: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/3 (45%/20), started new job
2025-04-10 20:33:46: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/3 (45%/20), starting parameter set
2025-04-10 20:33:46: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/3 (45%/20), starting new job
2025-04-10 20:33:47: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/4 (50%/20), started new job
2025-04-10 20:33:47: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/4 (50%/20), starting parameter set
2025-04-10 20:33:48: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/4 (50%/20), starting new job
2025-04-10 20:33:49: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/7/1 (55%/20), started new job
2025-04-10 20:33:49: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/7/1 (55%/20), starting parameter set
2025-04-10 20:33:49: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/7/1 (55%/20), starting new job
2025-04-10 20:33:50: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/7/2 (60%/20), started new job
2025-04-10 20:33:50: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/7/2 (60%/20), starting parameter set
2025-04-10 20:33:51: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/7/2 (60%/20), starting new job
2025-04-10 20:33:52: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/7/3 (65%/20), started new job
2025-04-10 20:33:52: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/7/3 (65%/20), starting parameter set
2025-04-10 20:33:52: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/7/3 (65%/20), starting new job
2025-04-10 20:33:53: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/10/1 (70%/20), started new job
2025-04-10 20:33:53: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/10/1 (70%/20), starting parameter set
2025-04-10 20:33:54: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/10/1 (70%/20), starting new job
2025-04-10 20:33:55: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/10/2 (75%/20), started new job
2025-04-10 20:33:55: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/10/2 (75%/20), starting parameter set
2025-04-10 20:33:55: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/10/2 (75%/20), starting new job
2025-04-10 20:33:56: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/10/3 (80%/20), started new job
2025-04-10 20:33:56: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/10/3 (80%/20), starting parameter set
2025-04-10 20:33:57: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/10/3 (80%/20), starting new job
2025-04-10 20:33:58: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/1/3 (85%/20), started new job
2025-04-10 20:33:58: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/1/3 (85%/20), starting parameter set
2025-04-10 20:33:58: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 13/1/3 (85%/20), starting new job
2025-04-10 20:33:59: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/1 (90%/20), started new job
2025-04-10 20:33:59: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/1 (90%/20), starting parameter set
2025-04-10 20:34:00: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/1 (90%/20), starting new job
2025-04-10 20:34:01: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/2 (95%/20), started new job
2025-04-10 20:34:01: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/2 (95%/20), starting parameter set
2025-04-10 20:34:01: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/2 (95%/20), starting new job
2025-04-10 20:34:02: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/3/3 (100%/20), started new job
2025-04-10 20:34:47: SobolGenerator, Failed jobs: 340 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 7/1/12 (100%/20), job_failed
2025-04-10 20:35:33: SobolGenerator, Failed jobs: 341 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/9 (95%/20), job_failed
2025-04-10 20:36:20: SobolGenerator, Failed jobs: 342 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/9 (90%/20), job_failed
2025-04-10 20:37:07: SobolGenerator, Failed jobs: 343 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/8 (85%/20), job_failed
2025-04-10 20:37:56: SobolGenerator, Failed jobs: 344 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/8 (80%/20), job_failed
2025-04-10 20:38:41: SobolGenerator, Failed jobs: 345 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/7 (75%/20), job_failed
2025-04-10 20:39:25: SobolGenerator, Failed jobs: 346 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/7 (70%/20), job_failed
2025-04-10 20:40:15: SobolGenerator, Failed jobs: 347 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/6 (65%/20), job_failed
2025-04-10 20:41:03: SobolGenerator, Failed jobs: 348 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/5 (60%/20), job_failed
2025-04-10 20:41:49: SobolGenerator, Failed jobs: 349 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/5 (55%/20), job_failed
2025-04-10 20:42:34: SobolGenerator, Failed jobs: 350 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/4 (50%/20), job_failed
2025-04-10 20:43:19: SobolGenerator, Failed jobs: 351 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/4 (45%/20), job_failed
2025-04-10 20:44:05: SobolGenerator, Failed jobs: 352 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/4 (40%/20), job_failed
2025-04-10 20:44:51: SobolGenerator, Failed jobs: 353 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/3 (35%/20), job_failed
2025-04-10 20:45:37: SobolGenerator, Failed jobs: 354 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/3 (30%/20), job_failed
2025-04-10 20:46:23: SobolGenerator, Failed jobs: 355 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/2 (25%/20), job_failed
2025-04-10 20:47:10: SobolGenerator, Failed jobs: 356 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/1 (20%/20), job_failed
2025-04-10 20:47:54: SobolGenerator, Failed jobs: 357 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/1 (15%/20), job_failed
2025-04-10 20:48:40: SobolGenerator, Failed jobs: 358 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 20:49:34: SobolGenerator, Failed jobs: 359 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 20:49:34: SobolGenerator, Failed jobs: 359 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), got 20, requested 20
2025-04-10 20:49:34: SobolGenerator, Failed jobs: 359 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 20:50:28: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 20:50:37: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 20:50:46: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 20:50:46: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 6s
2025-04-10 20:50:46: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 5s
2025-04-10 20:50:47: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 5s
2025-04-10 20:50:47: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 5s
2025-04-10 20:50:47: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 4s
2025-04-10 20:50:48: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 4s
2025-04-10 20:50:48: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 4s
2025-04-10 20:50:49: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 4s
2025-04-10 20:50:49: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 3s
2025-04-10 20:50:49: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 3s
2025-04-10 20:50:50: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 3s
2025-04-10 20:50:50: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 2s
2025-04-10 20:50:51: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 20:50:51: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 2s
2025-04-10 20:50:51: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 20:50:52: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 20:50:52: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 20:50:53: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 20:50:53: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 20:50:53: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 20:50:53: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 20:50:55: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 20:50:55: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 20:50:55: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 20:50:56: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 20:50:56: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 20:50:57: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 20:50:58: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), started new job
2025-04-10 20:50:58: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), starting parameter set
2025-04-10 20:50:58: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), starting new job
2025-04-10 20:50:59: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/1 (20%/20), started new job
2025-04-10 20:50:59: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/1 (20%/20), starting parameter set
2025-04-10 20:51:00: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/1 (20%/20), starting new job
2025-04-10 20:51:01: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/2 (25%/20), started new job
2025-04-10 20:51:01: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/2 (25%/20), starting parameter set
2025-04-10 20:51:01: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/2 (25%/20), starting new job
2025-04-10 20:51:02: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/1 (30%/20), started new job
2025-04-10 20:51:02: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/1 (30%/20), starting parameter set
2025-04-10 20:51:03: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/1 (30%/20), starting new job
2025-04-10 20:51:04: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/2 (35%/20), started new job
2025-04-10 20:51:04: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/2 (35%/20), starting parameter set
2025-04-10 20:51:04: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/2 (35%/20), starting new job
2025-04-10 20:51:05: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/3 (40%/20), started new job
2025-04-10 20:51:05: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/3 (40%/20), starting parameter set
2025-04-10 20:51:06: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/2/3 (40%/20), starting new job
2025-04-10 20:51:08: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/5/1 (45%/20), started new job
2025-04-10 20:51:08: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/5/1 (45%/20), starting parameter set
2025-04-10 20:51:08: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/5/1 (45%/20), starting new job
2025-04-10 20:51:09: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/2 (50%/20), started new job
2025-04-10 20:51:09: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/2 (50%/20), starting parameter set
2025-04-10 20:51:10: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/2 (50%/20), starting new job
2025-04-10 20:51:11: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/3 (55%/20), started new job
2025-04-10 20:51:11: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/3 (55%/20), starting parameter set
2025-04-10 20:51:11: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/1/3 (55%/20), starting new job
2025-04-10 20:51:12: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/1 (60%/20), started new job
2025-04-10 20:51:12: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/1 (60%/20), starting parameter set
2025-04-10 20:51:13: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/1 (60%/20), starting new job
2025-04-10 20:51:14: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/2 (65%/20), started new job
2025-04-10 20:51:14: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/2 (65%/20), starting parameter set
2025-04-10 20:51:14: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/2 (65%/20), starting new job
2025-04-10 20:51:15: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/3 (70%/20), started new job
2025-04-10 20:51:16: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/3 (70%/20), starting parameter set
2025-04-10 20:51:16: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/4/3 (70%/20), starting new job
2025-04-10 20:51:17: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/7/1 (75%/20), started new job
2025-04-10 20:51:17: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/7/1 (75%/20), starting parameter set
2025-04-10 20:51:17: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 7/7/1 (75%/20), starting new job
2025-04-10 20:51:19: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), started new job
2025-04-10 20:51:20: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), starting parameter set
2025-04-10 20:51:20: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/4/2 (80%/20), starting new job
2025-04-10 20:51:22: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/7 (85%/20), started new job
2025-04-10 20:51:22: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/7 (85%/20), starting parameter set
2025-04-10 20:51:23: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 10/7 (85%/20), starting new job
2025-04-10 20:51:24: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/1 (90%/20), started new job
2025-04-10 20:51:24: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/1 (90%/20), starting parameter set
2025-04-10 20:51:24: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/1 (90%/20), starting new job
2025-04-10 20:51:25: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/2 (95%/20), started new job
2025-04-10 20:51:26: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/2 (95%/20), starting parameter set
2025-04-10 20:51:26: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/7/2 (95%/20), starting new job
2025-04-10 20:51:27: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/9/1 (100%/20), started new job
2025-04-10 20:52:19: SobolGenerator, Failed jobs: 360 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 2/4/14 (100%/20), job_failed
2025-04-10 20:53:09: SobolGenerator, Failed jobs: 361 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 2/3/14 (95%/20), job_failed
2025-04-10 20:53:57: SobolGenerator, Failed jobs: 362 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/7 (90%/20), job_failed
2025-04-10 20:54:45: SobolGenerator, Failed jobs: 363 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/6 (85%/20), job_failed
2025-04-10 20:55:34: SobolGenerator, Failed jobs: 364 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/5 (80%/20), job_failed
2025-04-10 20:56:22: SobolGenerator, Failed jobs: 365 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/4 (75%/20), job_failed
2025-04-10 20:57:12: SobolGenerator, Failed jobs: 366 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/4 (70%/20), job_failed
2025-04-10 20:58:02: SobolGenerator, Failed jobs: 367 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/4 (65%/20), job_failed
2025-04-10 20:58:50: SobolGenerator, Failed jobs: 368 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/4 (60%/20), job_failed
2025-04-10 20:59:38: SobolGenerator, Failed jobs: 369 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/4 (55%/20), job_failed
2025-04-10 21:00:28: SobolGenerator, Failed jobs: 370 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/3 (50%/20), job_failed
2025-04-10 21:01:15: SobolGenerator, Failed jobs: 371 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/3 (45%/20), job_failed
2025-04-10 21:02:03: SobolGenerator, Failed jobs: 372 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/3 (40%/20), job_failed
2025-04-10 21:02:51: SobolGenerator, Failed jobs: 373 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/3 (35%/20), job_failed
2025-04-10 21:03:41: SobolGenerator, Failed jobs: 374 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/2 (30%/20), job_failed
2025-04-10 21:04:29: SobolGenerator, Failed jobs: 375 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/2 (25%/20), job_failed
2025-04-10 21:05:17: SobolGenerator, Failed jobs: 376 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/1 (20%/20), job_failed
2025-04-10 21:06:05: SobolGenerator, Failed jobs: 377 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/1 (15%/20), job_failed
2025-04-10 21:06:53: SobolGenerator, Failed jobs: 378 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 21:07:50: SobolGenerator, Failed jobs: 379 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 21:07:51: SobolGenerator, Failed jobs: 379 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), got 20, requested 20
2025-04-10 21:07:51: SobolGenerator, Failed jobs: 379 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 21:08:48: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 21:08:59: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 21:09:08: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 21:09:09: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 6s
2025-04-10 21:09:09: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 6s
2025-04-10 21:09:09: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 5s
2025-04-10 21:09:10: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 5s
2025-04-10 21:09:10: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 5s
2025-04-10 21:09:11: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 4s
2025-04-10 21:09:11: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 4s
2025-04-10 21:09:11: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 4s
2025-04-10 21:09:12: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 3s
2025-04-10 21:09:12: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 3s
2025-04-10 21:09:13: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 3s
2025-04-10 21:09:13: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 2s
2025-04-10 21:09:13: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 21:09:14: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 2s
2025-04-10 21:09:14: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 21:09:15: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 21:09:15: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 21:09:15: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 21:09:16: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 21:09:16: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 21:09:16: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 21:09:19: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 21:09:19: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 21:09:19: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 21:09:22: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), started new job
2025-04-10 21:09:22: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting parameter set
2025-04-10 21:09:22: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting new job
2025-04-10 21:09:23: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/1 (15%/20), started new job
2025-04-10 21:09:23: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/1 (15%/20), starting parameter set
2025-04-10 21:09:24: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/1 (15%/20), starting new job
2025-04-10 21:09:25: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/2 (20%/20), started new job
2025-04-10 21:09:25: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/2 (20%/20), starting parameter set
2025-04-10 21:09:25: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/2 (20%/20), starting new job
2025-04-10 21:09:26: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/3 (25%/20), started new job
2025-04-10 21:09:26: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/3 (25%/20), starting parameter set
2025-04-10 21:09:27: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/1/3 (25%/20), starting new job
2025-04-10 21:09:28: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/4/1 (30%/20), started new job
2025-04-10 21:09:28: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/4/1 (30%/20), starting parameter set
2025-04-10 21:09:28: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/4/1 (30%/20), starting new job
2025-04-10 21:09:29: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/4/2 (35%/20), started new job
2025-04-10 21:09:29: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/4/2 (35%/20), starting parameter set
2025-04-10 21:09:30: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/4/2 (35%/20), starting new job
2025-04-10 21:09:31: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/4/3 (40%/20), started new job
2025-04-10 21:09:31: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/4/3 (40%/20), starting parameter set
2025-04-10 21:09:31: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/4/3 (40%/20), starting new job
2025-04-10 21:09:32: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/1 (45%/20), started new job
2025-04-10 21:09:32: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/1 (45%/20), starting parameter set
2025-04-10 21:09:33: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/1 (45%/20), starting new job
2025-04-10 21:09:34: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/2 (50%/20), started new job
2025-04-10 21:09:34: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/2 (50%/20), starting parameter set
2025-04-10 21:09:34: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/2 (50%/20), starting new job
2025-04-10 21:09:35: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/3 (55%/20), started new job
2025-04-10 21:09:35: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/3 (55%/20), starting parameter set
2025-04-10 21:09:36: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/7/3 (55%/20), starting new job
2025-04-10 21:09:37: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/10/1 (60%/20), started new job
2025-04-10 21:09:37: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/10/1 (60%/20), starting parameter set
2025-04-10 21:09:38: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/10/1 (60%/20), starting new job
2025-04-10 21:09:40: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 11/1/1 (65%/20), started new job
2025-04-10 21:09:40: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 11/1/1 (65%/20), starting parameter set
2025-04-10 21:09:41: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 11/1/1 (65%/20), starting new job
2025-04-10 21:09:42: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 11/1/2 (70%/20), started new job
2025-04-10 21:09:42: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 11/1/2 (70%/20), starting parameter set
2025-04-10 21:09:42: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 11/1/2 (70%/20), starting new job
2025-04-10 21:09:43: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/1 (75%/20), started new job
2025-04-10 21:09:43: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/1 (75%/20), starting parameter set
2025-04-10 21:09:44: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/1 (75%/20), starting new job
2025-04-10 21:09:45: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/2 (80%/20), started new job
2025-04-10 21:09:45: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/2 (80%/20), starting parameter set
2025-04-10 21:09:45: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/2 (80%/20), starting new job
2025-04-10 21:09:46: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/3 (85%/20), started new job
2025-04-10 21:09:46: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/3 (85%/20), starting parameter set
2025-04-10 21:09:47: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/3 (85%/20), starting new job
2025-04-10 21:09:48: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/1 (90%/20), started new job
2025-04-10 21:09:48: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/1 (90%/20), starting parameter set
2025-04-10 21:09:48: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/1 (90%/20), starting new job
2025-04-10 21:09:49: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/2 (95%/20), started new job
2025-04-10 21:09:49: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/2 (95%/20), starting parameter set
2025-04-10 21:09:50: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/2 (95%/20), starting new job
2025-04-10 21:09:51: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/3 (100%/20), started new job
2025-04-10 21:10:39: SobolGenerator, Failed jobs: 380 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 4/4/12 (100%/20), job_failed
2025-04-10 21:11:30: SobolGenerator, Failed jobs: 381 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 12/7 (95%/20), job_failed
2025-04-10 21:12:22: SobolGenerator, Failed jobs: 382 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/7 (90%/20), job_failed
2025-04-10 21:13:13: SobolGenerator, Failed jobs: 383 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/7 (85%/20), job_failed
2025-04-10 21:14:04: SobolGenerator, Failed jobs: 384 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/7 (80%/20), job_failed
2025-04-10 21:14:54: SobolGenerator, Failed jobs: 385 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/6 (75%/20), job_failed
2025-04-10 21:15:45: SobolGenerator, Failed jobs: 386 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/6 (70%/20), job_failed
2025-04-10 21:16:36: SobolGenerator, Failed jobs: 387 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/5 (65%/20), job_failed
2025-04-10 21:17:28: SobolGenerator, Failed jobs: 388 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/4 (60%/20), job_failed
2025-04-10 21:18:20: SobolGenerator, Failed jobs: 389 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/4 (55%/20), job_failed
2025-04-10 21:19:14: SobolGenerator, Failed jobs: 390 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/3 (50%/20), job_failed
2025-04-10 21:20:08: SobolGenerator, Failed jobs: 391 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/3 (45%/20), job_failed
2025-04-10 21:21:07: SobolGenerator, Failed jobs: 392 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/2 (40%/20), job_failed
2025-04-10 21:21:59: SobolGenerator, Failed jobs: 393 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/2 (35%/20), job_failed
2025-04-10 21:22:50: SobolGenerator, Failed jobs: 394 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/2 (30%/20), job_failed
2025-04-10 21:23:41: SobolGenerator, Failed jobs: 395 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/2 (25%/20), job_failed
2025-04-10 21:24:34: SobolGenerator, Failed jobs: 396 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/1 (20%/20), job_failed
2025-04-10 21:25:27: SobolGenerator, Failed jobs: 397 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/1 (15%/20), job_failed
2025-04-10 21:26:20: SobolGenerator, Failed jobs: 398 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 21:27:25: SobolGenerator, Failed jobs: 399 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 21:27:25: SobolGenerator, Failed jobs: 399 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), got 20, requested 20
2025-04-10 21:27:25: SobolGenerator, Failed jobs: 399 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 21:28:30: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 21:28:42: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 21:28:52: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 21:28:53: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 6s
2025-04-10 21:28:53: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 6s
2025-04-10 21:28:54: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 6s
2025-04-10 21:28:54: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 5s
2025-04-10 21:28:54: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 5s
2025-04-10 21:28:55: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 5s
2025-04-10 21:28:55: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 4s
2025-04-10 21:28:56: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 4s
2025-04-10 21:28:56: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 4s
2025-04-10 21:28:56: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 3s
2025-04-10 21:28:57: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 3s
2025-04-10 21:28:57: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 3s
2025-04-10 21:28:58: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 21:28:58: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 2s
2025-04-10 21:28:58: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 1s
2025-04-10 21:28:59: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 21:28:59: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 21:29:00: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 21:29:00: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 21:29:01: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 21:29:01: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 21:29:02: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 21:29:02: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 21:29:02: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 21:29:04: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 21:29:04: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 21:29:04: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 21:29:05: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 3 (15%/20), started new job
2025-04-10 21:29:05: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 3 (15%/20), starting parameter set
2025-04-10 21:29:06: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending 3 (15%/20), starting new job
2025-04-10 21:29:08: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/1 (20%/20), started new job
2025-04-10 21:29:08: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/1 (20%/20), starting parameter set
2025-04-10 21:29:08: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 3/1 (20%/20), starting new job
2025-04-10 21:29:09: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/1/1 (25%/20), started new job
2025-04-10 21:29:09: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/1/1 (25%/20), starting parameter set
2025-04-10 21:29:10: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/1/1 (25%/20), starting new job
2025-04-10 21:29:11: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), started new job
2025-04-10 21:29:11: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), starting parameter set
2025-04-10 21:29:11: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/1 (30%/20), starting new job
2025-04-10 21:29:12: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/2 (35%/20), started new job
2025-04-10 21:29:12: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/2 (35%/20), starting parameter set
2025-04-10 21:29:13: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/2 (35%/20), starting new job
2025-04-10 21:29:14: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/3 (40%/20), started new job
2025-04-10 21:29:14: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/3 (40%/20), starting parameter set
2025-04-10 21:29:14: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/3 (40%/20), starting new job
2025-04-10 21:29:15: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/4 (45%/20), started new job
2025-04-10 21:29:15: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/4 (45%/20), starting parameter set
2025-04-10 21:29:16: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/1/4 (45%/20), starting new job
2025-04-10 21:29:17: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/5/1 (50%/20), started new job
2025-04-10 21:29:17: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/5/1 (50%/20), starting parameter set
2025-04-10 21:29:17: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/5/1 (50%/20), starting new job
2025-04-10 21:29:18: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/5/2 (55%/20), started new job
2025-04-10 21:29:18: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/5/2 (55%/20), starting parameter set
2025-04-10 21:29:19: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/5/2 (55%/20), starting new job
2025-04-10 21:29:20: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/5/3 (60%/20), started new job
2025-04-10 21:29:20: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/5/3 (60%/20), starting parameter set
2025-04-10 21:29:20: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/5/3 (60%/20), starting new job
2025-04-10 21:29:21: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/8/1 (65%/20), started new job
2025-04-10 21:29:21: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/8/1 (65%/20), starting parameter set
2025-04-10 21:29:22: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/8/1 (65%/20), starting new job
2025-04-10 21:29:23: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/8/2 (70%/20), started new job
2025-04-10 21:29:23: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/8/2 (70%/20), starting parameter set
2025-04-10 21:29:23: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/8/2 (70%/20), starting new job
2025-04-10 21:29:24: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/8/3 (75%/20), started new job
2025-04-10 21:29:24: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/8/3 (75%/20), starting parameter set
2025-04-10 21:29:25: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/8/3 (75%/20), starting new job
2025-04-10 21:29:26: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/11/1 (80%/20), started new job
2025-04-10 21:29:26: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/11/1 (80%/20), starting parameter set
2025-04-10 21:29:26: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 4/11/1 (80%/20), starting new job
2025-04-10 21:29:27: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/5/2 (85%/20), started new job
2025-04-10 21:29:27: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/5/2 (85%/20), starting parameter set
2025-04-10 21:29:28: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/5/2 (85%/20), starting new job
2025-04-10 21:29:29: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/5/3 (90%/20), started new job
2025-04-10 21:29:29: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/5/3 (90%/20), starting parameter set
2025-04-10 21:29:29: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/5/3 (90%/20), starting new job
2025-04-10 21:29:30: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/5/4 (95%/20), started new job
2025-04-10 21:29:31: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/5/4 (95%/20), starting parameter set
2025-04-10 21:29:31: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/5/4 (95%/20), starting new job
2025-04-10 21:29:32: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 10/9/1 (100%/20), started new job
2025-04-10 21:30:30: SobolGenerator, Failed jobs: 400 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 3/4/13 (100%/20), job_failed
2025-04-10 21:31:28: SobolGenerator, Failed jobs: 401 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed/running 3/3/13 (95%/20), job_failed
2025-04-10 21:32:25: SobolGenerator, Failed jobs: 402 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 9/9 (90%/20), job_failed
2025-04-10 21:33:24: SobolGenerator, Failed jobs: 403 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/9 (85%/20), job_failed
2025-04-10 21:34:19: SobolGenerator, Failed jobs: 404 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 8/8 (80%/20), job_failed
2025-04-10 21:35:14: SobolGenerator, Failed jobs: 405 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/8 (75%/20), job_failed
2025-04-10 21:36:08: SobolGenerator, Failed jobs: 406 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/7 (70%/20), job_failed
2025-04-10 21:37:03: SobolGenerator, Failed jobs: 407 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 7/6 (65%/20), job_failed
2025-04-10 21:37:59: SobolGenerator, Failed jobs: 408 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/6 (60%/20), job_failed
2025-04-10 21:38:55: SobolGenerator, Failed jobs: 409 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 6/5 (55%/20), job_failed
2025-04-10 21:39:51: SobolGenerator, Failed jobs: 410 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/5 (50%/20), job_failed
2025-04-10 21:40:49: SobolGenerator, Failed jobs: 411 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/4 (45%/20), job_failed
2025-04-10 21:41:43: SobolGenerator, Failed jobs: 412 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 5/3 (40%/20), job_failed
2025-04-10 21:42:42: SobolGenerator, Failed jobs: 413 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/3 (35%/20), job_failed
2025-04-10 21:43:39: SobolGenerator, Failed jobs: 414 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 4/2 (30%/20), job_failed
2025-04-10 21:44:35: SobolGenerator, Failed jobs: 415 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 3/2 (25%/20), job_failed
2025-04-10 21:45:28: SobolGenerator, Failed jobs: 416 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/2 (20%/20), job_failed
2025-04-10 21:46:32: SobolGenerator, Failed jobs: 417 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 2/1 (15%/20), job_failed
2025-04-10 21:47:26: SobolGenerator, Failed jobs: 418 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory/completed 1/1 (10%/20), job_failed
2025-04-10 21:48:30: SobolGenerator, Failed jobs: 419 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 21:48:30: SobolGenerator, Failed jobs: 419 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), got 20, requested 20
2025-04-10 21:48:30: SobolGenerator, Failed jobs: 419 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), out_of_memory 1 (5%/20), job_failed
2025-04-10 21:49:35: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 21:49:48: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 21:49:59: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 21:49:59: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 7s
2025-04-10 21:50:00: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 6s
2025-04-10 21:50:00: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 6s
2025-04-10 21:50:01: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 6s
2025-04-10 21:50:01: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 5s
2025-04-10 21:50:01: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 5s
2025-04-10 21:50:02: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 5s
2025-04-10 21:50:02: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 4s
2025-04-10 21:50:03: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 4s
2025-04-10 21:50:03: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 4s
2025-04-10 21:50:04: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 3s
2025-04-10 21:50:04: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 3s
2025-04-10 21:50:05: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 21:50:05: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 2s
2025-04-10 21:50:06: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 2s
2025-04-10 21:50:06: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 21:50:06: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 21:50:07: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 21:50:07: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 21:50:08: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 21:50:08: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 21:50:10: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 21:50:10: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 21:50:10: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 21:50:11: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), started new job
2025-04-10 21:50:12: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting parameter set
2025-04-10 21:50:12: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting new job
2025-04-10 21:50:13: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), started new job
2025-04-10 21:50:14: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting parameter set
2025-04-10 21:50:14: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting new job
2025-04-10 21:50:15: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/1 (20%/20), started new job
2025-04-10 21:50:16: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/1 (20%/20), starting parameter set
2025-04-10 21:50:16: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/1 (20%/20), starting new job
2025-04-10 21:50:17: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/2 (25%/20), started new job
2025-04-10 21:50:17: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/2 (25%/20), starting parameter set
2025-04-10 21:50:18: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/2 (25%/20), starting new job
2025-04-10 21:50:19: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/3 (30%/20), started new job
2025-04-10 21:50:19: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/3 (30%/20), starting parameter set
2025-04-10 21:50:19: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 3/3 (30%/20), starting new job
2025-04-10 21:50:21: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/1 (35%/20), started new job
2025-04-10 21:50:21: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/1 (35%/20), starting parameter set
2025-04-10 21:50:21: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/1 (35%/20), starting new job
2025-04-10 21:50:22: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/2 (40%/20), started new job
2025-04-10 21:50:22: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/2 (40%/20), starting parameter set
2025-04-10 21:50:23: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/2 (40%/20), starting new job
2025-04-10 21:50:24: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/3 (45%/20), started new job
2025-04-10 21:50:24: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/3 (45%/20), starting parameter set
2025-04-10 21:50:24: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 6/3 (45%/20), starting new job
2025-04-10 21:50:25: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/1 (50%/20), started new job
2025-04-10 21:50:25: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/1 (50%/20), starting parameter set
2025-04-10 21:50:26: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/1 (50%/20), starting new job
2025-04-10 21:50:27: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/2 (55%/20), started new job
2025-04-10 21:50:27: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/2 (55%/20), starting parameter set
2025-04-10 21:50:27: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/2 (55%/20), starting new job
2025-04-10 21:50:28: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/3 (60%/20), started new job
2025-04-10 21:50:28: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/3 (60%/20), starting parameter set
2025-04-10 21:50:29: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/3 (60%/20), starting new job
2025-04-10 21:50:30: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/4 (65%/20), started new job
2025-04-10 21:50:30: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/4 (65%/20), starting parameter set
2025-04-10 21:50:30: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/3/4 (65%/20), starting new job
2025-04-10 21:50:31: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/7/1 (70%/20), started new job
2025-04-10 21:50:31: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/7/1 (70%/20), starting parameter set
2025-04-10 21:50:32: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/7/1 (70%/20), starting new job
2025-04-10 21:50:33: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/7/2 (75%/20), started new job
2025-04-10 21:50:33: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/7/2 (75%/20), starting parameter set
2025-04-10 21:50:33: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/7/2 (75%/20), starting new job
2025-04-10 21:50:34: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/7/3 (80%/20), started new job
2025-04-10 21:50:34: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/7/3 (80%/20), starting parameter set
2025-04-10 21:50:35: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/7/3 (80%/20), starting new job
2025-04-10 21:50:36: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/10/1 (85%/20), started new job
2025-04-10 21:50:36: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/10/1 (85%/20), starting parameter set
2025-04-10 21:50:36: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/10/1 (85%/20), starting new job
2025-04-10 21:50:37: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/10/2 (90%/20), started new job
2025-04-10 21:50:37: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/10/2 (90%/20), starting parameter set
2025-04-10 21:50:38: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 6/10/2 (90%/20), starting new job
2025-04-10 21:50:39: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 16/3 (95%/20), started new job
2025-04-10 21:50:39: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 16/3 (95%/20), starting parameter set
2025-04-10 21:50:39: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 16/3 (95%/20), starting new job
2025-04-10 21:50:40: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/unknown 19/1 (100%/20), started new job
2025-04-10 21:51:35: SobolGenerator, Failed jobs: 420 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 7/5/8 (100%/20), job_failed
2025-04-10 21:52:31: SobolGenerator, Failed jobs: 421 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/8 (95%/20), job_failed
2025-04-10 21:53:29: SobolGenerator, Failed jobs: 422 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/8 (90%/20), job_failed
2025-04-10 21:54:25: SobolGenerator, Failed jobs: 423 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/7 (85%/20), job_failed
2025-04-10 21:55:21: SobolGenerator, Failed jobs: 424 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/7 (80%/20), job_failed
2025-04-10 21:56:16: SobolGenerator, Failed jobs: 425 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/6 (75%/20), job_failed
2025-04-10 21:57:13: SobolGenerator, Failed jobs: 426 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/6 (70%/20), job_failed
2025-04-10 21:58:09: SobolGenerator, Failed jobs: 427 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/6 (65%/20), job_failed
2025-04-10 21:59:11: SobolGenerator, Failed jobs: 428 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/6 (60%/20), job_failed
2025-04-10 22:00:06: SobolGenerator, Failed jobs: 429 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/5 (55%/20), job_failed
2025-04-10 22:01:04: SobolGenerator, Failed jobs: 430 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/4 (50%/20), job_failed
2025-04-10 22:02:02: SobolGenerator, Failed jobs: 431 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/3 (45%/20), job_failed
2025-04-10 22:02:58: SobolGenerator, Failed jobs: 432 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/3 (40%/20), job_failed
2025-04-10 22:03:57: SobolGenerator, Failed jobs: 433 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/3 (35%/20), job_failed
2025-04-10 22:04:55: SobolGenerator, Failed jobs: 434 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/2 (30%/20), job_failed
2025-04-10 22:05:51: SobolGenerator, Failed jobs: 435 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/2 (25%/20), job_failed
2025-04-10 22:06:50: SobolGenerator, Failed jobs: 436 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/1 (20%/20), job_failed
2025-04-10 22:07:45: SobolGenerator, Failed jobs: 437 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/1 (15%/20), job_failed
2025-04-10 22:08:42: SobolGenerator, Failed jobs: 438 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 22:09:51: SobolGenerator, Failed jobs: 439 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 22:09:51: SobolGenerator, Failed jobs: 439 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), got 20, requested 20
2025-04-10 22:09:51: SobolGenerator, Failed jobs: 439 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 22:11:01: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 22:11:15: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 22:11:27: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 22:11:27: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 7s
2025-04-10 22:11:28: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 6s
2025-04-10 22:11:28: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 7s
2025-04-10 22:11:29: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 6s
2025-04-10 22:11:29: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 6s
2025-04-10 22:11:30: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 5s
2025-04-10 22:11:30: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 5s
2025-04-10 22:11:30: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 4s
2025-04-10 22:11:31: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 4s
2025-04-10 22:11:31: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 4s
2025-04-10 22:11:32: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 3s
2025-04-10 22:11:32: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 3s
2025-04-10 22:11:33: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 22:11:33: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 2s
2025-04-10 22:11:34: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 2s
2025-04-10 22:11:34: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 22:11:34: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 22:11:35: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 22:11:35: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 22:11:36: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 22:11:36: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 22:11:37: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 22:11:38: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 22:11:38: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 22:11:39: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), started new job
2025-04-10 22:11:39: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting parameter set
2025-04-10 22:11:39: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting new job
2025-04-10 22:11:40: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), started new job
2025-04-10 22:11:41: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting parameter set
2025-04-10 22:11:41: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting new job
2025-04-10 22:11:42: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/3 (20%/20), started new job
2025-04-10 22:11:42: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/3 (20%/20), starting parameter set
2025-04-10 22:11:42: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/3 (20%/20), starting new job
2025-04-10 22:11:43: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), started new job
2025-04-10 22:11:43: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), starting parameter set
2025-04-10 22:11:44: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/1 (25%/20), starting new job
2025-04-10 22:11:46: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), started new job
2025-04-10 22:11:46: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), starting parameter set
2025-04-10 22:11:46: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/3/2 (30%/20), starting new job
2025-04-10 22:11:47: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/1 (35%/20), started new job
2025-04-10 22:11:47: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/1 (35%/20), starting parameter set
2025-04-10 22:11:48: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/1 (35%/20), starting new job
2025-04-10 22:11:49: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/2 (40%/20), started new job
2025-04-10 22:11:49: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/2 (40%/20), starting parameter set
2025-04-10 22:11:49: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/2 (40%/20), starting new job
2025-04-10 22:11:50: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/3 (45%/20), started new job
2025-04-10 22:11:50: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/3 (45%/20), starting parameter set
2025-04-10 22:11:51: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/3 (45%/20), starting new job
2025-04-10 22:11:52: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/4 (50%/20), started new job
2025-04-10 22:11:52: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/4 (50%/20), starting parameter set
2025-04-10 22:11:52: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/4 (50%/20), starting new job
2025-04-10 22:11:53: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/1 (55%/20), started new job
2025-04-10 22:11:53: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/1 (55%/20), starting parameter set
2025-04-10 22:11:54: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/1 (55%/20), starting new job
2025-04-10 22:11:55: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/2 (60%/20), started new job
2025-04-10 22:11:55: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/2 (60%/20), starting parameter set
2025-04-10 22:11:55: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/2 (60%/20), starting new job
2025-04-10 22:11:56: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/3 (65%/20), started new job
2025-04-10 22:11:56: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/3 (65%/20), starting parameter set
2025-04-10 22:11:57: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/3 (65%/20), starting new job
2025-04-10 22:11:58: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/1 (70%/20), started new job
2025-04-10 22:11:58: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/1 (70%/20), starting parameter set
2025-04-10 22:11:58: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/1 (70%/20), starting new job
2025-04-10 22:11:59: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/2 (75%/20), started new job
2025-04-10 22:11:59: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/2 (75%/20), starting parameter set
2025-04-10 22:12:00: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/2 (75%/20), starting new job
2025-04-10 22:12:01: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/3 (80%/20), started new job
2025-04-10 22:12:01: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/3 (80%/20), starting parameter set
2025-04-10 22:12:01: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/3 (80%/20), starting new job
2025-04-10 22:12:02: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/4 (85%/20), started new job
2025-04-10 22:12:03: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/4 (85%/20), starting parameter set
2025-04-10 22:12:03: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/1/4 (85%/20), starting new job
2025-04-10 22:12:04: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/1 (90%/20), started new job
2025-04-10 22:12:04: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/1 (90%/20), starting parameter set
2025-04-10 22:12:04: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/1 (90%/20), starting new job
2025-04-10 22:12:05: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/2 (95%/20), started new job
2025-04-10 22:12:05: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/2 (95%/20), starting parameter set
2025-04-10 22:12:06: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/2 (95%/20), starting new job
2025-04-10 22:12:07: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/5/3 (100%/20), started new job
2025-04-10 22:13:03: SobolGenerator, Failed jobs: 440 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 10/4/6 (100%/20), job_failed
2025-04-10 22:14:04: SobolGenerator, Failed jobs: 441 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/8 (95%/20), job_failed
2025-04-10 22:15:05: SobolGenerator, Failed jobs: 442 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/8 (90%/20), job_failed
2025-04-10 22:16:05: SobolGenerator, Failed jobs: 443 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/7 (85%/20), job_failed
2025-04-10 22:17:05: SobolGenerator, Failed jobs: 444 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/6 (80%/20), job_failed
2025-04-10 22:18:09: SobolGenerator, Failed jobs: 445 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/6 (75%/20), job_failed
2025-04-10 22:19:11: SobolGenerator, Failed jobs: 446 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/5 (70%/20), job_failed
2025-04-10 22:20:14: SobolGenerator, Failed jobs: 447 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/5 (65%/20), job_failed
2025-04-10 22:21:18: SobolGenerator, Failed jobs: 448 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/4 (60%/20), job_failed
2025-04-10 22:22:23: SobolGenerator, Failed jobs: 449 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/4 (55%/20), job_failed
2025-04-10 22:23:28: SobolGenerator, Failed jobs: 450 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/3 (50%/20), job_failed
2025-04-10 22:24:30: SobolGenerator, Failed jobs: 451 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/3 (45%/20), job_failed
2025-04-10 22:25:32: SobolGenerator, Failed jobs: 452 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/3 (40%/20), job_failed
2025-04-10 22:26:34: SobolGenerator, Failed jobs: 453 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/2 (35%/20), job_failed
2025-04-10 22:27:35: SobolGenerator, Failed jobs: 454 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/2 (30%/20), job_failed
2025-04-10 22:28:37: SobolGenerator, Failed jobs: 455 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/1 (25%/20), job_failed
2025-04-10 22:29:36: SobolGenerator, Failed jobs: 456 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/1 (20%/20), job_failed
2025-04-10 22:30:38: SobolGenerator, Failed jobs: 457 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/1 (15%/20), job_failed
2025-04-10 22:31:39: SobolGenerator, Failed jobs: 458 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 22:32:54: SobolGenerator, Failed jobs: 459 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 22:32:54: SobolGenerator, Failed jobs: 459 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), got 20, requested 20
2025-04-10 22:32:54: SobolGenerator, Failed jobs: 459 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 22:34:07: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 22:34:21: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 22:34:35: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 22:34:35: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 8s
2025-04-10 22:34:36: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 7s
2025-04-10 22:34:36: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 6s
2025-04-10 22:34:36: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 6s
2025-04-10 22:34:37: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 5s
2025-04-10 22:34:37: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 5s
2025-04-10 22:34:38: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 5s
2025-04-10 22:34:38: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 4s
2025-04-10 22:34:38: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 4s
2025-04-10 22:34:39: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 4s
2025-04-10 22:34:39: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 3s
2025-04-10 22:34:40: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 3s
2025-04-10 22:34:40: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 2s
2025-04-10 22:34:41: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 2s
2025-04-10 22:34:41: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 2s
2025-04-10 22:34:42: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 22:34:42: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 22:34:43: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 22:34:43: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 22:34:43: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 22:34:44: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 22:34:45: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 22:34:45: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 22:34:45: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 22:34:46: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), started new job
2025-04-10 22:34:46: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting parameter set
2025-04-10 22:34:47: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 2 (10%/20), starting new job
2025-04-10 22:34:48: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), started new job
2025-04-10 22:34:48: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), starting parameter set
2025-04-10 22:34:48: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 2/1 (15%/20), starting new job
2025-04-10 22:34:49: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/1 (20%/20), started new job
2025-04-10 22:34:49: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/1 (20%/20), starting parameter set
2025-04-10 22:34:50: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/1 (20%/20), starting new job
2025-04-10 22:34:51: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/2 (25%/20), started new job
2025-04-10 22:34:51: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/2 (25%/20), starting parameter set
2025-04-10 22:34:51: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/2 (25%/20), starting new job
2025-04-10 22:34:53: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/3 (30%/20), started new job
2025-04-10 22:34:53: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/3 (30%/20), starting parameter set
2025-04-10 22:34:53: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 2/1/3 (30%/20), starting new job
2025-04-10 22:34:54: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/1 (35%/20), started new job
2025-04-10 22:34:54: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/1 (35%/20), starting parameter set
2025-04-10 22:34:54: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/1 (35%/20), starting new job
2025-04-10 22:34:55: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/2 (40%/20), started new job
2025-04-10 22:34:56: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/2 (40%/20), starting parameter set
2025-04-10 22:34:56: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/2 (40%/20), starting new job
2025-04-10 22:34:57: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/3 (45%/20), started new job
2025-04-10 22:34:57: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/3 (45%/20), starting parameter set
2025-04-10 22:34:57: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/3/3 (45%/20), starting new job
2025-04-10 22:34:58: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/6/1 (50%/20), started new job
2025-04-10 22:34:58: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/6/1 (50%/20), starting parameter set
2025-04-10 22:34:59: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/6/1 (50%/20), starting new job
2025-04-10 22:35:00: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/6/2 (55%/20), started new job
2025-04-10 22:35:00: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/6/2 (55%/20), starting parameter set
2025-04-10 22:35:00: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/6/2 (55%/20), starting new job
2025-04-10 22:35:01: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/6/3 (60%/20), started new job
2025-04-10 22:35:01: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/6/3 (60%/20), starting parameter set
2025-04-10 22:35:02: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/6/3 (60%/20), starting new job
2025-04-10 22:35:03: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/9/1 (65%/20), started new job
2025-04-10 22:35:03: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/9/1 (65%/20), starting parameter set
2025-04-10 22:35:04: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/9/1 (65%/20), starting new job
2025-04-10 22:35:05: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/9/2 (70%/20), started new job
2025-04-10 22:35:05: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/9/2 (70%/20), starting parameter set
2025-04-10 22:35:05: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/9/2 (70%/20), starting new job
2025-04-10 22:35:07: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/9/3 (75%/20), started new job
2025-04-10 22:35:07: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/9/3 (75%/20), starting parameter set
2025-04-10 22:35:07: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/9/3 (75%/20), starting new job
2025-04-10 22:35:08: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/12/1 (80%/20), started new job
2025-04-10 22:35:08: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/12/1 (80%/20), starting parameter set
2025-04-10 22:35:09: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 3/12/1 (80%/20), starting new job
2025-04-10 22:35:10: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/1 (85%/20), started new job
2025-04-10 22:35:10: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/1 (85%/20), starting parameter set
2025-04-10 22:35:10: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/1 (85%/20), starting new job
2025-04-10 22:35:11: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/2 (90%/20), started new job
2025-04-10 22:35:11: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/2 (90%/20), starting parameter set
2025-04-10 22:35:12: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 15/1/2 (90%/20), starting new job
2025-04-10 22:35:14: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 16/2/1 (95%/20), started new job
2025-04-10 22:35:14: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 16/2/1 (95%/20), starting parameter set
2025-04-10 22:35:14: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 16/2/1 (95%/20), starting new job
2025-04-10 22:35:15: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 16/2/2 (100%/20), started new job
2025-04-10 22:36:16: SobolGenerator, Failed jobs: 460 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory/running 10/5/5 (100%/20), job_failed
2025-04-10 22:37:21: SobolGenerator, Failed jobs: 461 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/8 (95%/20), job_failed
2025-04-10 22:38:24: SobolGenerator, Failed jobs: 462 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/8 (90%/20), job_failed
2025-04-10 22:39:31: SobolGenerator, Failed jobs: 463 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/7 (85%/20), job_failed
2025-04-10 22:40:35: SobolGenerator, Failed jobs: 464 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/6 (80%/20), job_failed
2025-04-10 22:41:38: SobolGenerator, Failed jobs: 465 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/6 (75%/20), job_failed
2025-04-10 22:42:42: SobolGenerator, Failed jobs: 466 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/6 (70%/20), job_failed
2025-04-10 22:43:49: SobolGenerator, Failed jobs: 467 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/6 (65%/20), job_failed
2025-04-10 22:44:53: SobolGenerator, Failed jobs: 468 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/5 (60%/20), job_failed
2025-04-10 22:45:59: SobolGenerator, Failed jobs: 469 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/5 (55%/20), job_failed
2025-04-10 22:47:06: SobolGenerator, Failed jobs: 470 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/4 (50%/20), job_failed
2025-04-10 22:48:18: SobolGenerator, Failed jobs: 471 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/4 (45%/20), job_failed
2025-04-10 22:49:24: SobolGenerator, Failed jobs: 472 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/4 (40%/20), job_failed
2025-04-10 22:50:33: SobolGenerator, Failed jobs: 473 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/4 (35%/20), job_failed
2025-04-10 22:51:42: SobolGenerator, Failed jobs: 474 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/4 (30%/20), job_failed
2025-04-10 22:52:47: SobolGenerator, Failed jobs: 475 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/3 (25%/20), job_failed
2025-04-10 22:53:51: SobolGenerator, Failed jobs: 476 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/2 (20%/20), job_failed
2025-04-10 22:54:56: SobolGenerator, Failed jobs: 477 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 2/1 (15%/20), job_failed
2025-04-10 22:56:00: SobolGenerator, Failed jobs: 478 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 22:57:20: SobolGenerator, Failed jobs: 479 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 22:57:20: SobolGenerator, Failed jobs: 479 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), got 20, requested 20
2025-04-10 22:57:20: SobolGenerator, Failed jobs: 479 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 22:58:39: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job
2025-04-10 22:58:54: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), trying to get 20 next steps (current done: 0, max: 500)
2025-04-10 22:59:09: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #1/20
2025-04-10 22:59:10: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #2/20 | ETA: 10s
2025-04-10 22:59:10: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #3/20 | ETA: 8s
2025-04-10 22:59:10: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #4/20 | ETA: 7s
2025-04-10 22:59:11: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #5/20 | ETA: 7s
2025-04-10 22:59:11: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #6/20 | ETA: 6s
2025-04-10 22:59:12: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #7/20 | ETA: 6s
2025-04-10 22:59:12: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #8/20 | ETA: 5s
2025-04-10 22:59:13: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #9/20 | ETA: 5s
2025-04-10 22:59:13: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #10/20 | ETA: 4s
2025-04-10 22:59:14: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #11/20 | ETA: 4s
2025-04-10 22:59:14: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #12/20 | ETA: 4s
2025-04-10 22:59:15: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #13/20 | ETA: 3s
2025-04-10 22:59:15: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #14/20 | ETA: 3s
2025-04-10 22:59:16: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #15/20 | ETA: 2s
2025-04-10 22:59:16: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #16/20 | ETA: 2s
2025-04-10 22:59:17: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #17/20 | ETA: 1s
2025-04-10 22:59:17: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #18/20 | ETA: 1s
2025-04-10 22:59:18: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #19/20 | ETA: 0s
2025-04-10 22:59:18: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), getting new hyperparameter set #20/20 | ETA: 0s
2025-04-10 22:59:18: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting parameter set
2025-04-10 22:59:19: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), starting new job
2025-04-10 22:59:20: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), started new job
2025-04-10 22:59:20: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting parameter set
2025-04-10 22:59:20: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), unknown 1 (5%/20), starting new job
2025-04-10 22:59:21: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), started new job
2025-04-10 22:59:22: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting parameter set
2025-04-10 22:59:22: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/1 (10%/20), starting new job
2025-04-10 22:59:23: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), started new job
2025-04-10 22:59:23: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting parameter set
2025-04-10 22:59:23: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), pending/unknown 1/2 (15%/20), starting new job
2025-04-10 22:59:24: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/1 (20%/20), started new job
2025-04-10 22:59:24: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/1 (20%/20), starting parameter set
2025-04-10 22:59:25: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/1 (20%/20), starting new job
2025-04-10 22:59:26: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/2 (25%/20), started new job
2025-04-10 22:59:26: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/2 (25%/20), starting parameter set
2025-04-10 22:59:26: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/2 (25%/20), starting new job
2025-04-10 22:59:27: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/3 (30%/20), started new job
2025-04-10 22:59:27: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/3 (30%/20), starting parameter set
2025-04-10 22:59:28: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/2/3 (30%/20), starting new job
2025-04-10 22:59:29: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/1 (35%/20), started new job
2025-04-10 22:59:29: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/1 (35%/20), starting parameter set
2025-04-10 22:59:29: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/1 (35%/20), starting new job
2025-04-10 22:59:30: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/2 (40%/20), started new job
2025-04-10 22:59:30: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/2 (40%/20), starting parameter set
2025-04-10 22:59:31: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/2 (40%/20), starting new job
2025-04-10 22:59:32: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/3 (45%/20), started new job
2025-04-10 22:59:32: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/3 (45%/20), starting parameter set
2025-04-10 22:59:32: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/5/3 (45%/20), starting new job
2025-04-10 22:59:33: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 1/9 (50%/20), started new job
2025-04-10 22:59:33: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 1/9 (50%/20), starting parameter set
2025-04-10 22:59:34: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending 1/9 (50%/20), starting new job
2025-04-10 22:59:35: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/1 (55%/20), started new job
2025-04-10 22:59:35: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/1 (55%/20), starting parameter set
2025-04-10 22:59:36: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/1 (55%/20), starting new job
2025-04-10 22:59:37: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/2 (60%/20), started new job
2025-04-10 22:59:37: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/2 (60%/20), starting parameter set
2025-04-10 22:59:37: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/9/2 (60%/20), starting new job
2025-04-10 22:59:39: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/11/1 (65%/20), started new job
2025-04-10 22:59:39: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/11/1 (65%/20), starting parameter set
2025-04-10 22:59:39: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 1/11/1 (65%/20), starting new job
2025-04-10 22:59:40: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 11/1/2 (70%/20), started new job
2025-04-10 22:59:40: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 11/1/2 (70%/20), starting parameter set
2025-04-10 22:59:41: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 11/1/2 (70%/20), starting new job
2025-04-10 22:59:42: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/1 (75%/20), started new job
2025-04-10 22:59:42: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/1 (75%/20), starting parameter set
2025-04-10 22:59:42: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/1 (75%/20), starting new job
2025-04-10 22:59:43: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/2 (80%/20), started new job
2025-04-10 22:59:43: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/2 (80%/20), starting parameter set
2025-04-10 22:59:44: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 12/2/2 (80%/20), starting new job
2025-04-10 22:59:45: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/1 (85%/20), started new job
2025-04-10 22:59:45: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/1 (85%/20), starting parameter set
2025-04-10 22:59:45: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/1 (85%/20), starting new job
2025-04-10 22:59:47: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/2 (90%/20), started new job
2025-04-10 22:59:47: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/2 (90%/20), starting parameter set
2025-04-10 22:59:47: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/2 (90%/20), starting new job
2025-04-10 22:59:49: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/3 (95%/20), started new job
2025-04-10 22:59:49: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/3 (95%/20), starting parameter set
2025-04-10 22:59:49: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/2/3 (95%/20), starting new job
2025-04-10 22:59:51: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), running/pending/unknown 14/5/1 (100%/20), started new job
2025-04-10 23:01:01: SobolGenerator, Failed jobs: 480 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 13/7 (100%/20), job_failed
2025-04-10 23:02:13: SobolGenerator, Failed jobs: 481 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 13/6 (95%/20), job_failed
2025-04-10 23:03:31: SobolGenerator, Failed jobs: 482 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 12/6 (90%/20), job_failed
2025-04-10 23:04:40: SobolGenerator, Failed jobs: 483 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 12/5 (85%/20), job_failed
2025-04-10 23:05:49: SobolGenerator, Failed jobs: 484 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 11/5 (80%/20), job_failed
2025-04-10 23:07:02: SobolGenerator, Failed jobs: 485 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/5 (75%/20), job_failed
2025-04-10 23:08:12: SobolGenerator, Failed jobs: 486 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 10/4 (70%/20), job_failed
2025-04-10 23:09:22: SobolGenerator, Failed jobs: 487 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 9/4 (65%/20), job_failed
2025-04-10 23:10:32: SobolGenerator, Failed jobs: 488 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 8/4 (60%/20), job_failed
2025-04-10 23:11:42: SobolGenerator, Failed jobs: 489 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/4 (55%/20), job_failed
2025-04-10 23:12:52: SobolGenerator, Failed jobs: 490 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 7/3 (50%/20), job_failed
2025-04-10 23:14:03: SobolGenerator, Failed jobs: 491 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/3 (45%/20), job_failed
2025-04-10 23:15:13: SobolGenerator, Failed jobs: 492 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/2 (40%/20), job_failed
2025-04-10 23:16:24: SobolGenerator, Failed jobs: 493 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 6/1 (35%/20), job_failed
2025-04-10 23:17:34: SobolGenerator, Failed jobs: 494 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 5/1 (30%/20), job_failed
2025-04-10 23:18:44: SobolGenerator, Failed jobs: 495 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 4/1 (25%/20), job_failed
2025-04-10 23:19:55: SobolGenerator, Failed jobs: 496 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed/out_of_memory 3/1 (20%/20), job_failed
2025-04-10 23:21:09: SobolGenerator, Failed jobs: 497 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 3 (15%/20), job_failed
2025-04-10 23:22:18: SobolGenerator, Failed jobs: 498 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 2 (10%/20), job_failed
2025-04-10 23:24:17: SobolGenerator, Failed jobs: 499 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), finishing jobs after starting them, finished 19 jobs
2025-04-10 23:24:17: SobolGenerator, Failed jobs: 499 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), got 20, requested 20
2025-04-10 23:24:17: SobolGenerator, Failed jobs: 499 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), completed 1 (5%/20), job_failed
2025-04-10 23:25:43: SobolGenerator, Failed jobs: 500 ('LOSS: <number>' not found, 'ACCURACY: <number>' not found, 'TIME: <number>' not found), finishing previous jobs (1), finished 1 job