![]() This separation of logic makes it easier to test, debug, and evolve your system. This has the beneficial side-effect of separating the orchestration logic (which we leave to Airflow) from execution logic (which we leave to Docker images that run in Kubernetes). This is achieved by providing a Docker image and corresponding configurations of the pod. The KubernetesPodOperator works with the Kubernetes Python Client to run a task by launching a pod, which allows the user to have full control over the run-time environment, resources, and security. #AIRFLOW KUBERNETES HOW TO#While a DAG (Directed Acyclic Graph) describes how to run a workflow of tasks, an Airflow Operator defines what gets done by a task. In this post we will try to go through the lessons and best practices we have learned from using Airflow with Kubernetes at Benevolent.Īirflow now offers Operators and Executors for running your workload on a Kubernetes cluster: the KubernetesPodOperator and the KubernetesExecutor.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |