This page contains a summary of how bibliographical data (bibliodata) is processed in LexBib.

Collection of bibliodata: Zotero

All bibliodata is stored in LexBib Zotero, which is a "group" on the Zotero platform. The group page is public, but item attachments (PDF, TXT) are restricted to registered group members. Member registration is restricted to members of the project.
The Zotero software includes web scraping so-called translators, which ingest bibliodata as single items or in batches. Zotero will also try to harvest the PDF. If it finds a PDF, it also produces a TXT version.
We transform different bibliodata representation formats, or tabular data, to RIS format, which is seamlessly ingested by Zotero, using own converters.

Manual curation

Completeness of publication metadata is manually checked. The editing team uses Zotero group synchronization.
Every item is annotated with the first author's location. An English Wikipedia page URL (as unambiguous identifier) is placed in the Zotero "extra" field. zotexport.py (see below) maps that to the corresponding LexBib place item.
Items are annotated with Zotero tags that contain shortcodes, which are interpreted by zotexport.py. The shortcodes point either to LexBib wikibase items (Q-ID), or to pre-defined values:
- :container Qxx points to a containing item (a journal issue, an edited volume)
- :event Qxx points to a corresponding event (a conference iteration, a workshop)
- :abstractLanguage en indicates that the abstract contained in the dataset is given in English (and not in the language of the article)
- :collection x points to an Elexifinder collection number
- :type Review classifies the item as review article
- :type Community classifies the item as piece of community communication (anniversaries, obituaries, etc.)
- :type Report classifies the item as event report

We use the GROBID tool. zotexport.py (see below) leaves a copy of all PDF in a folder, which is processed by GROBID; GROBID produces a TEI-XML representation of the PDF content. The article body (i.e. the part usually starting after the abstract and ending before the references section) is enclosed in a tag called <body>.
In cases where GROBID fails, we manually isolate the text body from the Zotero-produced TXT (tutorial).