Events are one the last major addition to the Polyaxonfile specification in this release, users can now run DAGs with full events support.
Most workflow orchestrators support an aggregated upstream condition, e.g.
all failed, or
all done, basically an orchestrator would schedule an operation or a task only after the upstream is finished. That was also the case for Polyaxon until this release.
Events allow starting an operation in response to any event generated by an upstream entity. In this release, the entity is an operation running in the same DAG context. In the future, the entity could be anything from a Git commit, a new S3 blob, to an internal alert, or a new registered model version.
Here’s an example of starting a Tensorboard as soon as a training operation starts running:
version: 1.1 kind: component name: experiment-with-tensorboard run: kind: dag operations: - name: experiment pathRef: './experiment.yml' params: learning_rate: value: 0.005 epochs: value: 10 - name: tensorboard hubRef: tensorboard termination: timeout: 7200 params: uuid: ref: ops.experiment value: globals.uuid events: - ref: ops.experiment kinds: [run_status_running]
This DAG will schedule two operations, a job for training a DL experiment and a Tensorboard to visualize the outputs of the experiment. Instead of waiting for the experiment to finish before starting a Tensorboard, or copying the UUID of the job to start the Tensorboard manually, this DAG will schedule the Tensorboard automatically as soon as the training starts running. If the training fails because of some compilation error, the Tensorboard will be marked as skipped.
Learn More about Polyaxon
This blog post just goes over a couple of features that we shipped since our last product update, there are several other features and fixes that are worth checking. To learn more about all the features, fixes, and enhancements, please visit the release notes.
Polyaxon continues to grow quickly and keeps improving and providing the simplest machine learning abstraction. We hope that these updates will improve your workflows and increase your productivity, and again, thank you for your continued feedback and support.
Subscribe to Polyaxon Blog
Get the latest posts delivered right to your inbox