How to avoid cascadian skip of tasks — With the growth of complexity of our Airflow DAGs, our workflows started to have multiple branches. Simple cases might be implemented with custom checks, more complex ones require utilizing the Airflow API. The latter has some not obvious behavior which I’m going to tell in this article and how to…

Or don’t break silos, but push out the silo mentality. — The problem of silos in organizations disturbs minds for ages. You can find a lot of thoughts about this topic on the Internet. They all have common thoughts and suggestions about overcoming the problem with silo teams. …

A list of useful code snippets and topics about Python general use scenarios which is worth to remember — If you start learning Python after being a C# developer for quite a long time, you might want to have the same toolset which you had in C#. Here I’d like to give a kind of a mapping from C# to Python, although I should admit it’s not an accurate…

Finalizing three weeks of research & development in our team, I’d like to tell more about criteria which led us to choose Airflow among other interesting platforms for workflow orchestration. — On the way to build a process of data transition, scanning, wrangling and cleaning it’s essential to have an easy learning path and be able to address orchestration of workflows and their releasing as separate procedures. Since this is a new component in our ecosystem we need to make sure…

Workaround data representation in Siddhi to apply MongoDB index with TTL — Siddhi doesn’t support date/time type and can represent date/time values in strings and big integers. In systems which use date/time values to apply TTL logic it’s not possible to use Siddhi without workarounds. For example, in MongoDB it’s possible to create indexes with TTL settings which makes entries to expire…

A lighthouse to improve of a team collaboration — The worst ever 2020 year (by version of The Time magazine) is coming to its end and we need more positive news. Let’s remember what is positive communication and positive thinking, from which traits communications benefit from and what traits have destructive influence. The further article will be based on…

You can use config-maps, attach volumes to containers in Docker or Kubernetes to substitute files, or you can use environment variables to extract configuration from applications. But these are dirty hacks in comparison to what you can do if you would have a service which would provide a configuration for…

When your DevOps are busy and developers are blocked, waiting for a new or updated CI/CD pipeline for a new project, you might start looking for a solution which would save your time and efforts on automation. In this article I’ll tell you how we established the CI/CD process for…

In this article I want to share a small technique which saves time on repetitive tasks of creating similar projects. In my project there are multiple types of projects: CLI tool, a microservice, a data pipeline, a job. A programming language might be any, but we’re using often C# and…