I confess my naïveté about language servers -- would it not be feasible to write a dbt implementation of the Language Server Protocol in a reasonable amount of time? Surely dbt as a language can't be any more complex than, say, Terraform or other purpose-built languages. Are there technical reasons why a mixed SQL-Jinja-YAML cocktail wou…
I confess my naïveté about language servers -- would it not be feasible to write a dbt implementation of the Language Server Protocol in a reasonable amount of time? Surely dbt as a language can't be any more complex than, say, Terraform or other purpose-built languages. Are there technical reasons why a mixed SQL-Jinja-YAML cocktail would be intractable to parse?
Granted this doesn't address your point about file consolidation, which is a good one. Putting the templated SQL into the YAML feels incremental enough that it might be a feasible way forward.
Yea, you're right that you could get some of the benefits of a language server on the SQL-Jinja alone. SQL already has language servers, which can provide some limited functionality. Jinja has no concept of types though and very hard to debug. Don't forget there's also the Python interpreter in the mix somewhere too.
Not so sure about the yaml side. I don't think yaml is an expressive enough language, which is why so much of what we end up writing there when building metrics or tests can feel so clunky. Incremental improvements are definitely possible, but the whole point of this post was to remove constraints and think of what dbt might look like if we shed the weight of our past decisions.
Definitely. There's a lot of value to divergent thinking so I really appreciate your original post.
So my first thought is to toss out a platitude like "YAML isn't meant to be a very expressive language," but of course that's missing the point. Instead I want to learn about what would provide a good balance of expressivity and light weight. It makes me think of something like TypeScript -> JS transpilation, where you can impose type safety on yourself but in a manner that can easily be ignored for those who don't need (or don't think they need) it.
I'd say both LookML and Malloy offer some great alternatives to the dbt model, and it feels like they strike the right balance. Malloy is written in TypeScript, so a very similar model to what you're proposing too.
I confess my naïveté about language servers -- would it not be feasible to write a dbt implementation of the Language Server Protocol in a reasonable amount of time? Surely dbt as a language can't be any more complex than, say, Terraform or other purpose-built languages. Are there technical reasons why a mixed SQL-Jinja-YAML cocktail would be intractable to parse?
Granted this doesn't address your point about file consolidation, which is a good one. Putting the templated SQL into the YAML feels incremental enough that it might be a feasible way forward.
Yea, you're right that you could get some of the benefits of a language server on the SQL-Jinja alone. SQL already has language servers, which can provide some limited functionality. Jinja has no concept of types though and very hard to debug. Don't forget there's also the Python interpreter in the mix somewhere too.
Not so sure about the yaml side. I don't think yaml is an expressive enough language, which is why so much of what we end up writing there when building metrics or tests can feel so clunky. Incremental improvements are definitely possible, but the whole point of this post was to remove constraints and think of what dbt might look like if we shed the weight of our past decisions.
Definitely. There's a lot of value to divergent thinking so I really appreciate your original post.
So my first thought is to toss out a platitude like "YAML isn't meant to be a very expressive language," but of course that's missing the point. Instead I want to learn about what would provide a good balance of expressivity and light weight. It makes me think of something like TypeScript -> JS transpilation, where you can impose type safety on yourself but in a manner that can easily be ignored for those who don't need (or don't think they need) it.
I'd say both LookML and Malloy offer some great alternatives to the dbt model, and it feels like they strike the right balance. Malloy is written in TypeScript, so a very similar model to what you're proposing too.