YAML
Summarizing dbt Project Metadata From YAML in One Report
Read the dbt manifest built from your properties YAML and report documentation coverage, untested models, owners, tags and source freshness in one script.
7 min read
Tag
etl
32 posts
XML
Flattening XML in Spark and Databricks With Native Reader
Spark 4.0 reads XML natively with rowTag. Tested on 100,000 orders: schema inference, explode for lines, the double precision trap and from_xml for strings.
7 min read
XML
Parsing XML in BigQuery With a JavaScript UDF or Upstream
BigQuery has no XML loader or functions. Two routes: a JavaScript UDF with fast-xml-parser, tested here in Node, or parsing before the load to Parquet.
6 min read
XML
Loading XML Into Snowflake With Native Support and FLATTEN
Snowflake loads XML into VARIANT with a file format of TYPE XML, and XMLGET plus FLATTEN turn it into rows. The statements, the data shape and the limits.
6 min read
CSV
Writing Data Quality Checks for CSV Files in dbt and DuckDB
Read a CSV in a dbt model with DuckDB and test it with unique, accepted_values, relationships and SQL tests: 6 of 8 failed on a faulty file.
12 min read
XMLGuide
Handling XML in Warehouses and Lakes the Practical Way
How to get XML into Snowflake, BigQuery, Spark, DuckDB and a lake: keep raw files, parse once with a typed schema, store Parquet and reconcile counts.
16 min read
XML
Parsing an Atom Feed Into a Database With Safe Upserts
Load an Atom feed into PostgreSQL by entry id with insert or update, read links, categories and html content correctly, and compare lxml with feedparser.
7 min read
XML
Loading a Product Feed in XML Into PostgreSQL With xmltable
Load a Google Merchant style product feed into PostgreSQL with xmltable and a namespace, split price from currency, and handle missing GTINs and empty prices.
7 min read
Excel
Validating Row Counts After an Excel Import Step by Step
Prove an Excel import lost nothing by counting rows at the source, the stage and the target, adding control totals, and reconciling rejects in one query.
8 min read
Excel
Staging Excel Data in a Table Before Cleaning It Up
Design a staging table for Excel loads with text columns and provenance, load it fast, profile it with SQL and keep or drop it on a retention rule.
8 min read
Excel
Loading Excel Files With DuckDB and the Excel Extension
Query an XLSX sheet with SQL in DuckDB through its excel extension, read every column as text, create a table and copy it to PostgreSQL or Parquet.
8 min read
Excel
Importing Excel Dates Correctly Into a Date Column
An Excel date column holds serial numbers, real dates or text. Classify each value in the stage, convert each kind by an explicit rule and reject the rest.
7 min read