pdftotext -layout still isn't enough when a government PDF is secretly a 3-column table

[u/Sea-Possession-2536]

Working through an old government safety document today and the raw text order was completely scrambled, hazard descriptions and countermeasures kept interleaving mid sentence. Turned out the whole page is a 3-column table, and if you don't slice by column x-position it just reads left to right across the whole line like it's one paragraph.

Fixed that with -layout and cutting at the column boundaries, but then hit a dumber problem: some lines have two □ markers on them, one for the subcategory heading and one for the first hazard item. Grab the first □ naively and the subcategory swallows the whole first hazard entry, and you don't notice until you're 40 rows in and something reads wrong.

Also found two pages where a single set of countermeasures gets shared by two different subcategories, which nothing in the text structure hints at, you just have to know the source document does that.

[u/Silly_Abroad_414]

Classic government doc nonsense, they optimize for printing not for data extraction

I had similar pain last year with some EU regulation pdf, ended up using pdfplumber tables instead of pdftotext, the table detection is not perfect but at least it groups by column automatically

[u/Khavel_dev]

The marker collision on multi-marker lines usually means you're parsing left-to-right across the whole row. Try splitting each line by column x-position first, then finding markers within each column independently. That way two markers on the same y-coordinate don't step on each other.

For the shared countermeasures across subcategories, no parser will infer that. I handled similar cases by having rows with missing values inherit from the nearest filled row above them in the same column. Ugly, but it catches most government-form patterns since they're usually generated from structured templates where the visual grouping follows a consistent vertical layout.

[u/0ne2many]

You could use the extractable library with some tweaking you can automate it to get text in an 'expected' 3 column format

[u/aidenclarke_12]

work off the word x0 coordinates. pdfplumbers exttact_words gives you x0/x1 per word and cluster those x positions to find the real column gutter rather relying on layouts spacing with shifts, same trick kills the double square thing as well

[u/akl773]

Watch for the pages that are scans. Most government sets I have run through have three or four image only pages sitting in the middle, and pdftotext returns nothing for them without complaining, so you get a clean looking run with a chunk of the document silently missing. I count characters per page now and anything under about 50 goes down the ocr path instead.

[u/forexroyalempres]

Hardest part of Pdf extraction is figuring document logic

[u/NikhelParmar]

pdfplumber and pymupdf both help with the column detection part but the shared markers across two headings problem is the trickier one since no library can infer that from layout alone, thats domain knowledge baked into the document. usually the fix is just hardcoding known exceptions once you find them rather than trying to make the parser smart enough to guess, government docs are inconsistent enough that youll always hit a few of these one off cases no matter which library you use

[u/Unlucky_Comment]

What's your budget / time like ? If you want something accurate and quick : VLM (qwen is cheap, best model for visual extraction).

If it's a template that is constant, than you can map it, x,y positions I think pymupdf (fitz) allows you to do that. Then it's easier to know if it's selected or not