Dealing with text in "newspaper archive formats"

AngusLooney — Tue, 14 May 2019 14:50:59 GMT

Has anyone had to process scanned/OCRed text in any of the following formats?

GALEN XML - https://github.com/alan-turing-institute/i_newspaper_rods/blob/epcc-master/newsrods/test/fixtures/2000_04_24.xml

METS and ALTO format à https://en.wikipedia.org/wiki/ALTO_(XML) and https://veridiansoftware.com/knowledge-base/metsalto/ . Here you have an example: https://chroniclingamerica.loc.gov/lccn/sn86069133/1910-10-31/ed-1/seq-1/ocr.xml .

These are kind of "digital archive" formats, focused on capturing and preserving historic documents, with a large focus on maintaining the "look" of the documents, and less on the content.

I envisage having to parse large corpuses of these kinds of documents, and processing them into lots of discrete paragraph sized fragments, tagged with metadata on the file they came from, and the tagging from the input files themselves.

It's not, from the looks of things, nicely behaved bodies of text that will extract neatly, unlike maybe mining emails or business reports.

Any experience, insights, anything would be greatly received. Thanks in advance.

topic Dealing with text in "newspaper archive formats" in SAS Data Science

Dealing with text in "newspaper archive formats"