GUI page with all the settings of this job

Experiment overview

Best RESULT, min (total: 3)

Job Summary per Generation Node

Experiment parameters

Number of evaluations

Result names and types

Last progressbar status

Git-Version

Copy raw data to clipboard

Download »results.csv« as file

trial_index,submit_time,queue_time,worker_generator_uuid,start_time,end_time,run_time,program_string,exit_code,signal,hostname,OO_Info_int_param,OO_Info_int_param_two,OO_Info_float_param,OO_Info_choice_param,arm_name,trial_status,generation_node,RESULT,int_param,float_param,int_param_two,choice_param
0,1762176254,3,c4912eb4-53c8-4ba2-891c-4a23b247136d,1762176257,1762176263,6,./.tests/optimization_example  --int_param='-85' --float_param='-21.37047886848449707031' --choice_param='8' --int_param_two='-88' --nr_results=1,0,,c1,-85,-88,-21.3704788684844970703125,8,0_0,COMPLETED,SOBOL,-174024.541557532007573172450065612793,-85,-21.3704788684844970703125,-88,8
1,1762176268,3,c4912eb4-53c8-4ba2-891c-4a23b247136d,1762176271,1762176277,6,./.tests/optimization_example  --int_param='-38' --float_param='-94.14774686098098754883' --choice_param='1' --int_param_two='-40' --nr_results=1,0,,c1,-38,-40,-94.147746860980973337973409797996,1,1_0,COMPLETED,SOBOL,-137042.077076092013157904148101806641,-38,-94.147746860980987548828125,-40,1
2,1762176326,3,c4912eb4-53c8-4ba2-891c-4a23b247136d,1762176329,1762176336,7,./.tests/optimization_example  --int_param='-100' --float_param='10' --choice_param='8' --int_param_two='-25' --nr_results=1,0,,c1,-100,-25,10,8,2_0,COMPLETED,BOTORCH_MODULAR,-119790.919999999998253770172595977783,-100,10,-25,8

Copy raw data to clipboard

Download »results.csv« as file

Copy raw data to clipboard

Download »oo_errors.txt« as file

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

execute_nvidia_smi: An error occurred: Command '['nvidia-smi', '--query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used', '--format=csv,noheader']' returned non-zero exit status 255.

Copy raw data to clipboard

Download »oo_errors.txt« as file

Copy raw data to clipboard

Download »progressbar« as file

2025-11-03 14:24:13 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, Started OmniOpt2 run...
2025-11-03 14:24:14 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, getting new HP set (no sbatch)
2025-11-03 14:24:14 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, requested 1 jobs, got 1, 0.08 s/job
2025-11-03 14:24:14 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, eval #1/1 start                    
2025-11-03 14:24:14 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, starting new job                   
2025-11-03 14:24:14 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, started new job                    
2025-11-03 14:24:25 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, new result: RESULT: -174024.541558 
2025-11-03 14:24:27 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, best RESULT: -174024.541557532, finishing jobs, finished 1 job
2025-11-03 14:24:27 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, best RESULT: -174024.541557532, getting new HP set (no sbatch)
2025-11-03 14:24:27 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, best RESULT: -174024.541557532, requested 1 jobs, got 1, 0.11 s/job
2025-11-03 14:24:28 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, best RESULT: -174024.541557532, eval #1/1 start                    
2025-11-03 14:24:28 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, best RESULT: -174024.541557532, starting new job                   
2025-11-03 14:24:28 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, best RESULT: -174024.541557532, started new job                    
2025-11-03 14:24:39 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, best RESULT: -174024.541557532, new result: RESULT: -137042.077076 
2025-11-03 14:24:42 (c4912eb4-53c8-4ba2-891c-4a23b247136d): SOBOL, best RESULT: -174024.541557532, finishing jobs, finished 1 job     
2025-11-03 14:25:20 (c4912eb4-53c8-4ba2-891c-4a23b247136d): BOTORCH_MODULAR, best RESULT: -174024.541557532, getting new HP set (no sbatch)
2025-11-03 14:25:25 (c4912eb4-53c8-4ba2-891c-4a23b247136d): BOTORCH_MODULAR, best RESULT: -174024.541557532, requested 1 jobs, got 1, 37.66 s/job
2025-11-03 14:25:25 (c4912eb4-53c8-4ba2-891c-4a23b247136d): BOTORCH_MODULAR, best RESULT: -174024.541557532, eval #1/1 start                     
2025-11-03 14:25:26 (c4912eb4-53c8-4ba2-891c-4a23b247136d): BOTORCH_MODULAR, best RESULT: -174024.541557532, starting new job                    
2025-11-03 14:25:26 (c4912eb4-53c8-4ba2-891c-4a23b247136d): BOTORCH_MODULAR, best RESULT: -174024.541557532, started new job                     
2025-11-03 14:25:36 (c4912eb4-53c8-4ba2-891c-4a23b247136d): BOTORCH_MODULAR, best RESULT: -174024.541557532, new result: RESULT: -119790.920000  
2025-11-03 14:25:40 (c4912eb4-53c8-4ba2-891c-4a23b247136d): BOTORCH_MODULAR, best RESULT: -174024.541557532, finishing jobs, finished 1 job      

Copy raw data to clipboard

Download »progressbar« as file

Copy raw data to clipboard

Download »job_submit_durations.txt« as file

         Job submission durations          
┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┓
┃   Batch ┃ Seconds ┃ Jobs ┃ Time per job ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━┩
│       1 │   0.303 │    1 │        0.303 │
│       2 │   0.308 │    1 │        0.308 │
│       3 │   1.377 │    1 │        1.377 │
├─────────┼─────────┼──────┼──────────────┤
│ Average │   0.663 │      │              │
│  Median │   0.308 │      │              │
│   Total │   1.988 │      │              │
│     Max │   1.377 │      │              │
│     Min │   0.303 │      │              │
└─────────┴─────────┴──────┴──────────────┘

Copy raw data to clipboard

Download »job_submit_durations.txt« as file

Copy raw data to clipboard

Download »generation_times.txt« as file

           Model generation times            
┏━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┓
┃ Iteration ┃ Seconds ┃ Jobs ┃ Time per job ┃
┡━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━┩
│         1 │   0.083 │    1 │        0.083 │
│         2 │   0.112 │    1 │        0.112 │
│         3 │  37.665 │    1 │       37.665 │
├───────────┼─────────┼──────┼──────────────┤
│   Average │  12.620 │      │              │
│    Median │   0.112 │      │              │
│     Total │  37.860 │      │              │
│       Max │  37.665 │      │              │
│       Min │   0.083 │      │              │
└───────────┴─────────┴──────┴──────────────┘

Copy raw data to clipboard

Download »generation_times.txt« as file

Arguments Overview

Copy raw data to clipboard

Download »cpu_ram_usage.csv« as file

timestamp,ram_usage_mb,cpu_usage_percent
1762176248,790.43359375,10.6
1762176319,815.08203125,13.5

Copy raw data to clipboard

Download »cpu_ram_usage.csv« as file

RESULT (goal: minimize)

Best value: -174024.54155753
Achieved at:
- run_time = 6
- OO_Info_int_param = -85
- OO_Info_int_param_two = -88
- OO_Info_float_param = -21.370478868484
- OO_Info_choice_param = 8
- int_param = -85
- float_param = -21.370478868484
- int_param_two = -88
- choice_param = 8

Parameter statistics