I’ve done a lot of playing around with Delta Lake in my day, enough to have fallen in and out of love a few different times. I think for those of us who grew up in the land of Kimball and SQL Server, before the demi-gods of Snowflake and Databricks came a
and now seem to be a common part of the Data Engineering commentary. It’s probably here to stay. Say what you will about the thing and what it can and cannot replace, the future will decide.
Sometimes you have to take your own medicine, eat your own words, and swallow that bitter pill. I keep wondering if I’m going to have to do that with my DuckDB takes. I mean if you watch talking heads at Databricks Data and AI Summit it looks like DuckDB
Every once in a great while, I feel it’s good to pay some penance, to do a thing that isn’t fun at all, that you find appalling and horrible. I tell my children regularly that it’s a good thing to struggle; to do a “hard thing makes ya’ stronger
In a quest not to get too bored in the Data Engineering world, with Databricks and Snowflake being old hat these days it’s always good to go looking for interesting things to do. Interesting things yet obvious things, the simple things, things other peopl
Maybe it’s my age, I’m not totally sure, but it’s getting harder and harder for this old guy to keep up with all the new things that come trickling out of the quagmire that is the Data Engineering space these days. Some of the ideas that come into being s
Mar 04, 2024 If you know me, you know I like to stir the pot, the big boiling and smoldering cauldron of Data Tools pot. Yes, that’s the one, blackened and burned pot from years of conjurers pouring myriads of Modern Data Stack tools into it, which have s
Jan 29, 2024 In my never-ending quest to make angry I’m always on the lookout for Golden Calves and other Idols that I can smash to bits, and wait for the angry masses frothing at the mouth to hunt me down. Sorry, not sorry, after years of doing this, I k
Dec 18, 2022 Have you been as confused as me about My goal with this article is to cut through all the crud and get to the bottom of the issue. I keep seeing the name DuckDB keep popping up in my feeds here and there in my varied and disjoined internet tr
This is an interesting question indeed, is it not? What to use, what to use? Both DuckDB and Polars seem to be flying high at the moment, the new cool kids on the block. Everyone talking about them, and 1% of the people actually using them. Typical.
I’m well aware volumes of words have been written about simplicity in the context of Software Engineering, ye ole’ KISS concept that your grandmother taught you is as old as time. Yet the siren call of the modern data stack has lured many a poor soul down
In my never-ending quest to don’t take it personally, I do it for the good of all Data Engineers) I realized I’d forgotten something oh-so-important. I mean there are few more tedious things that we have to do day in and day out than date and datetime man
You know, just when you think things have finally settled down and our poor ears won’t have to hear another thing about The Great Catalog War, or The Great Lake House Format War it just keeps rolling in.
Sometimes I find myself lying in my sunroom, staring out the window in the blue sky above me while the sun plays on the maple tree, empty of most all but a few red leaves wondering what else I can do to make the already angry readers of my babbling even m
When it comes to the AI hype, I pretty much have tried to ignore the constant roll of never ending models and other hoopla that is mostly meaningless. Better to let the dust settle.
Thanks for the dedication to test the new shiny tool. We need people like you who is skeptical of enterprise blog posts and test yourself using your own platform. Im sure they are happy with the feedback and more people are aware of the new integration.
Unsure if this "means" anything, but I happened to notice that as of Mar 17, 2025, DuckDB seems to have recently passed Polars in number of downloads per day and per week.
But you have to say what he did wrong too you're an idiot" is just the first step to becoming them people. Nice try though.
re DuckDB: it has gotten a lot better at larger-than-memory queries. However, in this case, you’re trying to create the database in memory and then run a query on it. suggest two options: a) run your aggregation query directly without the CTAS by specifyi
An interesting article, thanks for it. However, I think one could expand your thoughts a little bit First of all, I assume that edge and serverless computing are two big areas where DuckDB could shine. Being able to easily examine a dataset inside an AWS
No solving world hunger, not building some fancy SaaS tool, nope, he was happily retired living inside a giant bespoke log mansion because there is apparently enough fighting that goes on in high places of power that kept him busy and rich, and probably s
Dec 05, 2024 Well, what is that old saying? Better late than never? Something like that. Weep, howl, and moan all ye Databricks and Snowflake padiwans, what you have greatly feared has come down upon you with a heavy hand. I can just see all the meetings
May 20, 2025 I will be the first to admit, in an unapologetic way, that working with Apache Iceberg is far from a pleasant experience once you move past the “playing around on my laptop” stage. The tight, inflexible relationship between Iceberg and a cata
We have opinions, what's yours? Tell me about what topics you want me to write about more. Do you want more indepth stuff with code? You want more high level stuff? No wait, maybe you want both? No, I know, you probably like it when I do data product revi
I’m usually not an advocate of being a dreamer, talking big talk but having it all be a pipe dream. I’m a dream killer most of the time. Maybe I’ve been around too long probably. Every tool in its place and every place needs a tool. It’s not that I’m agai
I know you are all gasping and covering your mouths in astonishment and disbelief, hardly able to contain yourself. Another data pipeline tool has arrived at your doorstep, promising to solve every known problem you have, and ones you don’t Part of my job
It’s been a while since I’ve kicked ye ole’ tires on something new. You know how much I love to pick and poke at things, I just can’t help it. But this one didn’t take any convincing on my part. I couldn’t even tell you where I ran into it. that is. Part