Show HN: New CI/CD Platform for Data Engineers https://ift.tt/q3VyHAC

Show HN: New CI/CD Platform for Data Engineers In the last year I've become interested in how Data Engineering is evolving to become more like Software Engineering, with the adoption of source control, branching/merging, unit testing, modular code etc. This has been made possible in part by dbt ( https://getdbt.io ), which is a really nice tool for defining and running data transformations. I had the idea to build an Open Source CI/CD platform around dbt, to automate the process and create a defined path to production for data changes. Though we could of course to do this with Jenkins or Github Actions, the thinking is that we can build a tool which avoids a lot of scripting when developing the pipeline, and provides a user experience that is optimised for Data Engineers, for instance exposing the underlying data as it is transformed and tested. Here is an introductory blog post with more thinking - https://ift.tt/pmJefHP I would be interested in any thoughts on the idea and approach. https://www.youtube.com/watch?v=-s-bULKecvY March 27, 2023 at 11:08PM