Orchestrator
What is the Orchestrator?
Sometimes, partitions contain instable nodes that have, for example, hardware issues. The Orchestrator allows you to react to those circumstances by detecting certain configurable strings in the stdout and stderr of your programs. Given the strings are contained, certain actions are possible. For example, restarting on a different node, restarting in general, and just excluding the node. This allows to detect defective nodes and skip them in production after only a single test job that failed on them automatically.
Example orchestrator.yaml-file
errors:
- name: GPUDisconnected
match_strings:
- "AssertionError: ``AmpOptimizerWrapper`` is only available"
behavior: ExcludeNode
- name: Timeout
match_strings:
- "Timeout"
behavior: RestartOnDifferentNode
- name: ExampleRestart
match_strings:
- "StartAgain"
behavior: Restart
- name: StorageError
match_strings:
- "Read/Write failure"
behavior: ExcludeNodeAndRestartAll
This configuration file does the following. When a job ends, and in the output the string...
- ... AssertionError: ``AmpOptimizerWrapper`` is only available appears, it will exclude that node from all future executions inside the current and all continued jobs from it
- ... Timeout appears, it will exclude that node from all future executions inside the current and all continued jobs from it, and restart the job on the list of nodes on that partition excluding the one that ran this timeout job
- ... StartAgain appears, it will restart that job (may end up on the same node)
- ... Read/Write failure appears, it will exclude the node the job started on and restart it on a different node