2025-08-14 03:01:59 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 7 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, waiting for 3 jobs, finished 3 jobs
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
========================================================================
To cancel, press CTRLc, then run 'scancel 1787764'
โ Importing logging...โ Importing warnings...โ Importing argparse...โ Importing datetime...โ Importing dataclass...โ Importing socket...โ Importing stat...โ Importing pwd...โ Importing base64...โ Importing json...โ Importing yaml...โ Importing toml...โ Importing csv...โ Importing ast...โ Importing rich.table...โ Importing rich print...โ Importing rich.pretty...โ Importing rich.prompt...โ Importing types.FunctionType...โ Importing typing...โ Importing ThreadPoolExecutor...โ Importing submitit.LocalExecutor...โ Importing submitit.Job...โ Importing importlib.util...โ Importing platform...โ Importing inspect frame info...โ Importing pathlib.Path...โ Importing uuid...โ Importing cowsay...โ Importing shutil...โ Importing itertools.combinations...โ Importing os.listdir...โ Importing os.path...โ Importing PIL.Image...โ Importing sixel...โ Importing subprocess...โ Importing tqdm...โ ธImporting beartype...โ Importing statistics...โ Trying to import pyfiglet...โ Importing helpers...โ Parsing arguments...
[WARNING 08-14 02:57:52] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
โ ธImporting ax random seed...โ Importing torch...โ Importing numpy...โ Importing ax...โ Importing ax.core.generator_run...โ Importing Cont_X_trans and Y_trans from ax.adapter.registry...โ Importing ax.core.arm...โ Importing ax.core.objective...โ Importing ax.core.Metric...โ Importing ax.exceptions.core...โ Importing ax.exceptions.generation_strategy...โ Importing CORE_DECODER_REGISTRY...โ Trying ax.generation_strategy.generation_node...โ Importing GenerationStep, GenerationStrategy from generation_strategy...โ Importing GenerationNode from generation_node...โ Importing ExternalGenerationNode...โ Importing MaxTrials...โ Importing GeneratorSpec...โ Importing Models from ax.generation_strategy.registry...โ Importing get_pending_observation_features...โ Importing load_experiment...โ Importing save_experiment...โ Importing save_experiment_to_db...โ Importing TrialStatus...โ Importing Data...โ Importing Experiment...โ Importing parameter types...โ Importing TParameterization...โ Importing pandas...โ Importing AxClient and ObjectiveProperties...โ Importing RandomForestRegressor...โ Importing botorch...โ Importing submitit...โ Importing ax logger...โ Importing SQL-Storage-Stuff...
Run-UUID: 4ba849b3-b75c-441d-8b5f-079381ecb02d
____ _ ____ _ ___
/ __ \ (_)/ __ \ | | |__ \
| | | |_ __ ___ _ __ _| | | |_ __ | |_ ) |
| | | | '_ ` _ \| '_ \| | | | | '_ \| __| / /
| |__| | | | | | | | | | | |__| | |_) | |_ / /_
\____/|_| |_| |_|_| |_|_|\____/| .__/ \__|____|
| |
|_|
โ Writing worker creation log...
omniopt --live_share --send_anonymized_usage_stats --partition=alpha --experiment_name=test_orchestrator --mem_gb=5 --time=60 --worker_timeout=5 --max_eval=4 --num_parallel_jobs=4 --gpus=1 --run_program=Li8udGVzdHMvb3JjaGVzdHJhdG9yX3Rlc3RzLmJpbi8lKG5hbWUp --parameter name choice simple_ok,storage_error,timeout_failure,gpu_disconnected --num_random_steps=1 --model=BOTORCH_MODULAR --auto_exclude_defective_hosts --orchestrator_file .tests/example_orchestrator_config.yaml --seed 1234 --follow
โ Disabling logging...โ Setting run folder...โ Creating folder /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/test_orchestrator/0...โ Writing revert_to_random_when_seemingly_exhausted file ...โ Writing username state file...โ Writing result names file...โ Writing result min/max file...โ Saving state files...Run-folder: /home/h8/s3811141/test/randomtest_93382/OmniOpt/runs/test_orchestrator/0โ Printing run info...โ Initializing NVIDIA-Logs...โ Writing ui_url file if it is present...โ Writing live_share file if it is present...โ Writing job_start_time file...โ นWriting git informationโ Checking max_eval...Warning: --num_random_steps 1 is smaller than --num_parallel_jobs 4. It's recommended that --num_parallel_jobs is the same as or a multiple of --num_random_stepsโ Calculating number of steps...โ Adding excluded nodes...โ Handling random steps...โ Setting global generation strategyโ Initializing ax_client...โ Setting orchestrator...See https://imageseg.scads.de/omniax/share?user_id=s3811141&experiment_name=test_orchestrator&run_nr=1 for live-results.โ ผInitializing live share...โ ผInitializing live share...You have 2 CPUs available for the main process.Using CUDA device NVIDIA A100-SXM4-40GB.Generation strategy: SOBOL for 1 step and then BOTORCH_MODULAR for 3 steps.
Run-Program: ./.tests/orchestrator_tests.bin/%(name)
Experiment parameters
โโโโโโโโณโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โNameโType โValues โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โnameโchoiceโsimple_ok, storage_error, timeout_failure, gpu_disconnectedโ
โโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Result-Names
โโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โResult-NameโMin or max?โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โRESULT โ minโ
โโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
โ นWrite files and show overviewโ Starting generator workers
SOBOL, running 4โ4 (100%/4), job_failed : 25%|โโโโโโโโโโ| 1/4 [00:32<01:37, 32.64s/it]
RestartOnDifferentNode was triggered for node i8032. Adding node to defective hosts list and restarting on another host.
RestartOnDifferentNode was triggered for node i8032, but it was already in defective nodes. Job will only be resubmitted.
SOBOL, failed: 1 ('RESULT: ' not found), best RESULT: 9667, completed/pending 2/2โ4 (75%/4), job_failed : 50%|โโโโโโโโโโ| 2/4 [00:47<00:44, 22.31s/it]
ExcludeNode was triggered for node i8006
ExcludeNode was triggered for node i8006, but it was already in defective nodes and won't be added again
SOBOL, failed: 2 ('RESULT: ' not found), best RESULT: 0.123, completed 2โ2 (0%/4), job_failed : 50%|โโโโโโโโโโ| 2/4 [01:11<00:44, 22.31s/it]
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
RestartOnDifferentNode was triggered for node i8032, but it was already in defective nodes. Job will only be resubmitted.
RestartOnDifferentNode was triggered for node i8032, but it was already in defective nodes. Job will only be resubmitted.
RestartOnDifferentNode was triggered for node i8032, but it was already in defective nodes. Job will only be resubmitted.
RestartOnDifferentNode was triggered for node i8032, but it was already in defective nodes. Job will only be resubmitted.
SOBOL, failed: 4 ('RESULT: ' not found), best RESULT: 0.123, completed 4โ4 (0%/4), job_failed : 50%|โโโโโโโโโโ| 2/4 [01:52<00:44, 22.31s/it]
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
SOBOL, failed: 4 ('RESULT: ' not found), best RESULT: 0.123, completed 4โ4 (0%/4), job_failed : 75%|โโโโโโโโโโ| 3/4 [01:52<00:41, 41.57s/it]
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
RestartOnDifferentNode was triggered for node i8021. Adding node to defective hosts list and restarting on another host.
RestartOnDifferentNode was triggered for node i8032, but it was already in defective nodes. Job will only be resubmitted.
RestartOnDifferentNode was triggered for node i8021, but it was already in defective nodes. Job will only be resubmitted.
RestartOnDifferentNode was triggered for node i8032, but it was already in defective nodes. Job will only be resubmitted.
SOBOL, failed: 6 ('RESULT: ' not found), best RESULT: 0.123, completed/running 2/2โ4 (100%/4), waiting for 4 jobs, finished 4 jobs: 100%|โโโโโโโโโโ| 4/4 [02:36<00:00, 41.57s/it]
ERROR while trying to mark job as failure: Can only mark failed a trial that is currently running.
SOBOL, failed: 6 ('RESULT: ' not found), best RESULT: 0.123, completed/running 2/2โ4 (100%/4), waiting for 4 jobs, finished 4 jobs: 5it [02:36, 30.63s/it]
RestartOnDifferentNode was triggered for node i8032, but it was already in defective nodes. Job will only be resubmitted.
RestartOnDifferentNode was triggered for node i8032, but it was already in defective nodes. Job will only be resubmitted.
SOBOL, failed: 7 ('RESULT: ' not found), best RESULT: 0.123, waiting for 3 jobs, finished 3 jobs : 9it [03:47, 25.32s/it]
Best RESULT, min (total: 15, failed: 7)
โโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโ
โOO_Info_SLURM_JOB_IDโname โRESULTโ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ1787768 โsimple_okโ0.123 โ
โโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโ
Runtime Infos
โโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโ
โNumber of evaluationsโMin time โMax time โAverage timeโMedian timeโ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 16 โ 6.00 sec โ 7.00 sec โ 6.25 sec โ 6.00 sec โ
โโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโ
2025-08-14 02:58:11 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, Started OmniOpt2 run...
2025-08-14 02:58:11 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, getting new HP set
2025-08-14 02:58:11 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, requested 1 jobs, got 1, 0.05 s/job
2025-08-14 02:58:11 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, eval #1/1 start
2025-08-14 02:58:11 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, starting new job
2025-08-14 02:58:13 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, unknown 1∑1 (25%/4), started new job
2025-08-14 02:58:13 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, unknown 1∑1 (25%/4), getting new HP set
2025-08-14 02:58:13 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, unknown 1∑1 (25%/4), requested 1 jobs, got 1, 0.07 s/job
2025-08-14 02:58:13 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, unknown 1∑1 (25%/4), eval #1/1 start
2025-08-14 02:58:13 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, unknown 1∑1 (25%/4), starting new job
2025-08-14 02:58:13 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/1∑2 (25%/4), started new job
2025-08-14 02:58:13 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/1∑2 (25%/4), getting new HP set
2025-08-14 02:58:14 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/1∑2 (25%/4), requested 1 jobs, got 1, 0.08 s/job
2025-08-14 02:58:14 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/1∑2 (25%/4), eval #1/1 start
2025-08-14 02:58:14 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/1∑2 (25%/4), starting new job
2025-08-14 02:58:14 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/2∑3 (25%/4), started new job
2025-08-14 02:58:14 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/2∑3 (25%/4), getting new HP set
2025-08-14 02:58:14 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/2∑3 (25%/4), requested 1 jobs, got 1, 0.08 s/job
2025-08-14 02:58:14 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/2∑3 (25%/4), eval #1/1 start
2025-08-14 02:58:14 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/2∑3 (25%/4), starting new job
2025-08-14 02:58:14 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/3∑4 (25%/4), started new job
2025-08-14 02:58:16 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/3∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:16 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/3∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:17 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/3∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:17 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/3∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:18 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/3∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:18 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/3∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:19 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending/unknown 1/3∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:20 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:20 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:21 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:21 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:22 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:22 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:23 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:23 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:24 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:24 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:25 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:26 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (25%/4), waiting for 4 jobs
2025-08-14 02:58:27 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:28 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:29 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:29 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:30 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:30 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:31 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:31 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:32 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:32 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:33 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:34 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:34 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:35 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:35 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:36 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, pending 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:36 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:37 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:37 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:38 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:39 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:39 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:40 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:40 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:41 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:41 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:43 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:43 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), job_failed
2025-08-14 02:58:43 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, running 4∑4 (100%/4), new result: 9667.0
2025-08-14 02:58:57 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 1 ('RESULT: <FLOAT>' not found), best RESULT: 9667, completed/pending 2/2∑4 (100%/4), waiting for 4 jobs, finished 2 jobs
2025-08-14 02:58:57 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 1 ('RESULT: <FLOAT>' not found), best RESULT: 9667, completed/pending 2/2∑4 (100%/4), waiting for 4 jobs
2025-08-14 02:58:58 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 1 ('RESULT: <FLOAT>' not found), best RESULT: 9667, completed/pending 2/2∑4 (75%/4), job_failed
2025-08-14 02:58:58 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 1 ('RESULT: <FLOAT>' not found), best RESULT: 9667, completed/pending 2/2∑4 (75%/4), new result: 0.123
2025-08-14 02:59:19 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 2 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 2∑2 (0%/4), waiting for 4 jobs, finished 2 jobs
2025-08-14 02:59:20 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 2 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 2∑2 (0%/4), waiting for 2 jobs
2025-08-14 02:59:23 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 2 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 2∑2 (0%/4), job_failed
2025-08-14 02:59:23 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 2 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 2∑2 (0%/4), job_failed
2025-08-14 02:59:59 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 4 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 4∑4 (0%/4), waiting for 2 jobs, finished 2 jobs
2025-08-14 03:00:00 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 4 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 4∑4 (0%/4), waiting for 4 jobs
2025-08-14 03:00:03 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 4 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 4∑4 (0%/4), job_failed
2025-08-14 03:00:03 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 4 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 4∑4 (0%/4), new result: 22185.0
2025-08-14 03:00:03 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 4 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 4∑4 (0%/4), job_failed
2025-08-14 03:00:03 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 4 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 4∑4 (0%/4), new result: 16957.0
2025-08-14 03:00:47 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 6 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed/running 2/2∑4 (100%/4), waiting for 4 jobs, finished 4 jobs
2025-08-14 03:00:47 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 6 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed/running 2/2∑4 (100%/4), waiting for 4 jobs
2025-08-14 03:00:47 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 6 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed/running 2/2∑4 (100%/4), job_failed
2025-08-14 03:00:47 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 6 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed/running 2/2∑4 (100%/4), new result: 16933.0
2025-08-14 03:00:47 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 6 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed/running 2/2∑4 (100%/4), new result: 2477.0
2025-08-14 03:01:21 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 7 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 3∑3 (0%/4), waiting for 4 jobs, finished 3 jobs
2025-08-14 03:01:23 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 7 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 3∑3 (0%/4), waiting for 3 jobs
2025-08-14 03:01:25 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 7 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 3∑3 (0%/4), new result: 1128.0
2025-08-14 03:01:25 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 7 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 3∑3 (0%/4), new result: 8568.0
2025-08-14 03:01:25 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 7 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, completed 3∑3 (0%/4), new result: 12389.0
2025-08-14 03:01:59 (4b902e38-4500-45fe-9db6-d28cfa5fdee9): SOBOL, failed: 7 ('RESULT: <FLOAT>' not found), best RESULT: 0.123, waiting for 3 jobs, finished 3 jobs
