Selinon by Fridolín Pokorný

Distributed dynamic task flow management with Python

Abstract

Selinon is a project that is based on popular Python project Celery. Celery is a distributed task queue that offers one to run tasks. Selinon gives one a power to define flows and dependencies in flows, schedule tasks based on results of workers, success or any external events, handle errors, trace flow state and actions in a distributed environment. Its main goal is to split task logic (code) and persistence logic from data and time dependencies between tasks in flows that are stated in simple YAML configuration files.

Description

Selinon is an advanced flow management above Celery project (an asynchronous distributed task queue) written in Python3, that allows you to dynamically schedule tasks based on results of previous tasks, group tasks to flows, schedule flows from other flows, store results of tasks in your storages and databases transparently, validate results against defined JSON schemas, track flow progress via the build-in tracing mechanism, complex per-task or per-flow failure handling with fallback tasks or fallback flows and run the whole system in a distributed environments orchestrated by OpenShift or Kubernetes.