LLM (semi) structured training and inference
Document AI / IE / Unstructured
Getting things out of documents
Benchmarks
- [2607.29677] ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction 1 , which is basically “given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata” and this is evaluated. Really cool, including classifying documents by how exactly are they problematic
Unstructured and semi-structured
Key
- EVAPORATE: [2304.09433] Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes 2
-
We propose and evaluate EVAPORATE […]. We identify two fundamentally different strategies for implementing this system: prompt the LLM to directly extract values from documents or prompt the LLM to synthesize code that performs the extraction.
- schema found automatically
-
Our key insight is to generate many candidate functions and ensemble their extractions using weak supervision. EVAPORATE-CODE+ not only outperforms the state-of-the art systems, but does so using a sublinear pass over the documents with the LLM.
-
To reduce variance, we synthesize many candidate functions, then estimate their quality and aggregate their extractions using weak supervision.
-
Benchmarks
- [2604.25359v1] The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models 3
- Neat overview of what’s wrong with other benchmarks
- Metrics for evaluation
- Many do JSON correctly but with wrong data
Surveys
- [2508.00217v1] Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges 4(2025)
- exceptional
- Large Language Models for Structured and Semi-Structured Data, Recommender Systems and Knowledge Base Engineering: A Survey of Recent Techniques and Architectures5 Section 5
- Lists many open topics
- Large Language Model for Table Processing: A Survey (2024)6 nice
Processing
- TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios7 LLama 8B finetuned for querying+editing tables
- SpreadsheetLLM: Encoding Spreadsheets for Large Language Models8 is BRILLIANT
- Spreadsheets != tables
- Uses an inverted index (
bob: C1,C4,A0), homogeneous columns (e.g. all dates), automatically find the useful table bit in the sheet. Genuinely a lot of cool ideas. (Also SpreadsheetCompressor to save tokens)
Table formats/representations
- Blog posts by the same company:
- Which Table Format Do LLMs Understand Best? (Results for 11 Formats), includes list of formats and sample data
- doesn’t convince me
- only gpt4.1-nano
- ‘Markdown-KV’ won, XML was second, CSV was the second worst
- Which Nested Data Format Do LLMs Understand Best? JSON vs. YAML vs. XML vs. Markdown
- JSON, CSV, Markdown, YAML
- 3 models tested, results varied
- Which Table Format Do LLMs Understand Best? (Results for 11 Formats), includes list of formats and sample data
Etc.
- Text-to-table is an established task.
- [2508.15910v1] Evaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets 9
- discussion on limited suitability of usual NLP metrics on table level
- does neat postprocessing to find and check the markdown tables returned by the LLM
- [2508.15910v1] Evaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets 9
Etc.
Venues for relevant literature
- This footnote from Section 2.1 of 10:
Consider, for example, the International Conference on Document Analysis and Recognition (ICDAR), the International Journal on Document Analysis and Recognition (IJDAR), the International Workshop on Document Analyis Systems (DAS), the International Workshop on Graphics Recognition (GREC), some of which host serial competitions, such as the Robust Reading Competition linking the document analysis and computer vision communities [Ka13; Ya17], cf. https://rrc.cvc.uab.es/.
- Robust Reading Competition does really neat competitions
“Robust Reading” refers to the research area dealing with the interpretation of written communication in unconstrained settings. // e.g. scans maps etc. + VQA
- Start | The ACM Symposium on Document Engineering
-
Document engineering is the computer science discipline that investigates systems for documents in any form and in all media. As with the relationship between software engineering and software, document engineering is concerned with principles, tools and processes that improve our ability to create, manage, and maintain documents.
- Proceedings, e.g. 2025: Proceedings of the 2025 ACM Symposium on Document Engineering | ACM Conferences
- Sessions on IR, classification, analysis
-
Less relevant but interesting topics
Tabular foundation models
TL;DR: pretrained on large tabular datasets, predict via in-context learning (dynamically adapting to a task without optimizing any parameters). Do classification regression etc.
- Tabular Foundation Models the WIP book on the basics.
- Models
Long tail of weird stuff
- TIL about TOON which tries to use fewer tokens than JSON/TOML/… (but CSV wins for small datasets)
TODO
- [2603.23047] Parametric Knowledge and Retrieval Behavior in RAG Fine-Tuning for Electronic Design Automation11
- Fine-tune a 7B RAG that beats lager model
- long quotes
- Metrics for retrieval incl. using triplet-based (response, context, query)
- neat intro and related work
-
<_(@zhangextractbenchbenchmark2026) “ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction” (2026) / Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, Simon Suo: http://arxiv.org/abs/2607.29677 /
10.48550/arXiv.2607.29677_> ↩︎ -
<_(@aroralanguagemodels2025) “Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes” (2025) / Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, Christopher Ré: http://arxiv.org/abs/2304.09433 /
10.48550/arXiv.2304.09433_> ↩︎ -
<_(@singhstructuredoutput2026) “The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models” (2026) / Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D. Khemlani, Vineet Agarwal: http://arxiv.org/abs/2604.25359 /
10.48550/arXiv.2604.25359_> ↩︎ -
<_(@wutabulardata2025) “Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges” (2025) / Xiaofeng Wu, Alan Ritter, Wei Xu: https://arxiv.org/abs/2508.00217 /
10.48550/ARXIV.2508.00217_> ↩︎ -
<_(@smajiclargelanguage2025) “Large Language Models for Structured and Semi-Structured Data, Recommender Systems and Knowledge Base Engineering: A Survey of Recent Techniques and Architectures” (2025) / Alma Smajić, Ratomir Karlović, Mieta Bobanović Dasko, Ivan Lorencin: https://www.mdpi.com/2079-9292/14/15/3153 /
10.3390/electronics14153153_> ↩︎ -
<_(@lulargelanguage2024) “Large Language Model for Table Processing: A Survey” (2024) / Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, Xiaoyong Du: https://arxiv.org/abs/2402.05121 /
10.48550/ARXIV.2402.05121_> ↩︎ -
<_(@zhangtablellmenabling2024) “TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios” (2024) / Xiaokang Zhang, Sijia Luo, Bohan Zhang, Zeyao Ma, Jing Zhang, Yang Li, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, Jifan Yu, Shu Zhao, Juanzi Li, Jie Tang: https://arxiv.org/abs/2403.19318 /
10.48550/ARXIV.2403.19318_> ↩︎ -
<_(@dongspreadsheetllmencoding2024) “SpreadsheetLLM: Encoding Spreadsheets for Large Language Models” (2024) / Haoyu Dong, Jianbo Zhao, Yuzhang Tian, Junyu Xiong, Shiyu Xia, Mengyu Zhou, Yun Lin, José Cambronero, Yeye He, Shi Han, Dongmei Zhang: https://arxiv.org/abs/2407.09025 /
10.48550/ARXIV.2407.09025_> ↩︎ -
<_(@oestreichevaluatingstructured2025) “Evaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets” (2025) / Julian Oestreich, Lydia Müller: https://arxiv.org/abs/2508.15910 /
10.48550/ARXIV.2508.15910_> ↩︎ -
<_(@perakisclassifyingfigures2023) “Classifying figures and illustrations in electronics datasheets: A comparative evaluation of recent computer vision models on a custom collection of 4000 technical documents” (2023) / Lymperis Perakis, Julian Balling, Frank Binder, Gerhard Heyer, Franz Kreupl: https://dl.gi.de/handle/20.500.12116/43114 /
10.18420/INF2023_186_> ↩︎ -
<_(@oestreichparametricknowledge2026) “Parametric Knowledge and Retrieval Behavior in RAG Fine-Tuning for Electronic Design Automation” (2026) / Julian Oestreich, Maximilian Bley, Frank Binder, Lydia Müller, Maksym Sydorenko, André Alcalde: http://arxiv.org/abs/2603.23047 /
10.48550/arXiv.2603.23047_> ↩︎