Skip to content
← All projects
Academic

LLM Web Parser

Modular web data pipeline for LLM ingestion. Features domain-specific parsers, gold-standard evaluation, and FastAPI microservices.

restpythoncrawl4aidockerfastapinlp
LLM Web Parser

Modular web data pipeline for LLM ingestion, built in the Computer Engineering Laboratory at Sapienza University of Rome under Prof. Roberto Navigli, together with Michele Pio Forlani and Salvatore Iavarone. It retrieves, cleans and structures web content so it can feed LLM training and retrieval workflows.

Highlights

  • AI-assisted extraction with Crawl4AI for LLM-friendly crawling and content retrieval.
  • Domain-specific parsers that strip boilerplate, scripts and styles, and normalise text.
  • Gold-standard evaluation to measure parser output quality.
  • FastAPI microservices with full Docker orchestration for reproducible deployment.
  • Structured output (plain text or JSON) ready for annotation tools and datasets.