Running Lakeflow pipelines as Dagster assets: what the spike showed
At Protective Life I led a spike to test whether Lakeflow pipelines can run as Dagster assets. I guided three engineers and worked with the platform team.
What we found
Lakeflow pipelines can run as Dagster assets using two pipeline tags and no custom code.
What we tested
- Queueing
- Cancellation
- Failure behavior
I demoed the result live at sprint review, and the team then planned the pipeline's production rollout.
Back to the projects