Excel
Using the Excel Data Model for Larger Datasets in Practice
Load millions of rows into the Excel Data Model with Power Query, relate tables, build pivots on it, keep it small and know the limits of memory and bitness.
7 min read
Tag
large-files
34 posts
Excel
Benchmarking Excel Readers in Python With a Fair Test
Compare openpyxl, calamine, pandas engines and DuckDB for reading xlsx: build test files, time runs, record peak memory and avoid common benchmark mistakes.
7 min read
Excel
Reducing Workbook Size by Removing Unused Styles and Data
Find why an xlsx is large by looking inside the zip, then remove unused styles, empty formatted rows, pivot caches, images and links to shrink the file.
8 min read
Excel
Analyzing More Than a Million Rows Without Excel Limits
Analyze data beyond Excel's 1,048,576 row limit with DuckDB, SQLite, pandas in chunks or the Data Model, and bring only the summary back to Excel.
6 min read
Excel
Converting a Huge Excel File to CSV or Parquet Safely
Convert a very large Excel workbook to CSV or Parquet with a streaming script, DuckDB or LibreOffice, keeping types, dates, encoding and every sheet right.
8 min read
Excel
Why Excel Is Slow With Formulas Over Many Rows Explained
Why Excel slows down with many formulas: the calculation chain, volatile functions, whole column references and scanning lookups, and how to find the cost.
8 min read
Excel
Reading a 500 MB Excel File Without Running Out of Memory
Read a very large xlsx in Python without exhausting memory: why xlsx expands, openpyxl read only mode, calamine and DuckDB, and how to measure peak memory.
7 min read
ExcelGuide
A Guide to Excel Files That Are Too Big or Too Slow
Diagnose an Excel workbook that is huge, slow to calculate or too large to read in code, then fix it by size, formulas, the Data Model or a different format.
17 min read
XML
Summarizing a Large XML Export With a Streaming Parser
Total amounts by currency and top SKUs over 20,000 invoices in one streaming pass with Decimal, 21 MB of memory, and the same USD total as a tree-based check.
6 min read
XML
Mismatched Tag Errors in Large XML Files and Finding Them
Locate an Opening and ending tag mismatch in a 28.6 MB XML file four ways: the parser, per-tag counts, record-by-record parsing and bisection, with timings.
6 min read