Svarna: An Open Corpus Workbench for Modern Greek

Source: arXiv:2607.00970 · Published 2026-07-01 · By Stergios Chatzikyriakidis

TL;DR

Svarna is an open-source, web-based corpus workbench designed to unify and provide accessible search and analysis tools over modern Greek corpora. It integrates five independently searchable databases covering institutional, literary, dialectal, social media, and historical registers, totaling over 507 million words and approximately 29 million sentences. This platform addresses the fragmentation and accessibility issues in Greek language resources by consolidating freely available yet scattered datasets into a single, easy-to-use interface that requires no login or installation.

Svarna offers multiple linguistic analytic functions such as KWIC concordancing, register-based frequency analysis with normalization, mutual information-based collocation extraction, discourse marker profiling, register comparison through log-ratio statistics, regex search, and a novel optional LLM layer for pragmatic annotation and free-form research. Technically, it leverages SQLite FTS5 full-text indexes served via a FastAPI backend, deployed as Docker containers on Azure, with all code and database build scripts publicly available under the MIT license. Users can add new corpora and deploy customized instances. Although Svarna does not aim to replace mature commercial or institutional platforms, it fills a significant gap by enabling broad, immediate access to diverse Greek text corpora in a consistent environment. Demonstrations include formal vs informal register marker distributions, dialectal lexical variation, and polysemy-driven collocation differences.

Key results show that Svarna supports large-scale queries over a half-billion word database comprising multiple genres and dialects, providing normalized frequency tables, collocation networks, and discourse marker distributions automatically with a lightweight architecture. Its open design allows extensibility to additional Greek corpora and other languages. While certain limitations arise from opportunistic corpus inclusion and sample-based statistics, Svarna lays the foundation for improved Greek corpus linguistics research and widespread usability.

Key findings

• Svarna integrates five corpora spanning institutional, literary, dialectal, social media, and historical registers totaling 507 million words and 29 million sentences.
• The platform supports phrase, prefix, Boolean, and regex full-text searching using SQLite FTS5 with the unicode61 tokenizer tailored for Greek Unicode.
• Frequency analysis is normalized per million tokens by register to enable meaningful cross-register comparisons.
• Collocation extraction applies pointwise mutual information (PMI) at query time over concordance samples instead of precomputed indexes.
• Svarna includes a lexicon of 93 Greek discourse markers grouped by 11 functional categories with precomputed frequency by register.
• The register comparison tool uses log-ratio statistic to detect characteristic words distinguishing any two selected registers.
• An optional LLM interface allows pragmatic classification and free research queries on retrieved corpus text, using user-supplied API keys.
• The system's architecture is fully open with backend, frontend code, build and deployment scripts released under MIT license, supporting user-added corpora and self-hosting.

Threat model

n/a - This work focuses on corpus resource integration and linguistic search and analysis, and does not define or address any adversarial threat model for security or bot defense.

Methodology — deep read

1. Threat Model & Assumptions: Svarna is designed as an open research infrastructure rather than a security system, so adversarial considerations are not explicitly addressed. The system assumes users who want easy, immediate access to Greek corpora and will likely be naïve researchers or students without credentials or installations. There is no adversarial threat model.

2. Data: Svarna integrates five thematic databases:

• Corpus Database: institutional/general purpose Greek (Wikipedia, parliamentary minutes, web crawls, subtitles, treebanks) totaling roughly 473 million words.
• Literature Database: creative works including Project Gutenberg Greek texts, child-directed BabyLM corpus, interwar poetry, and a tesserae classical Greek corpus totaling 26.6 million words.
• Dialectal Database: nine Greek dialect groups from GRDD+ comprising 5.9 million words.
• Social Media Database: Greek Twitter data labeled for toxicity totaling 1.9 million words.
• Historical Database: text collections from CLARIN:EL Crete project totaling about 0.12 million words.

All raw corpora are publicly available though dispersed; Svarna standardizes and integrates them into self-contained SQLite FTS5 databases with sentence-level segmentation and metadata.

3. Architecture / Algorithm: Svarna uses a classic client-server architecture. Backend is a Python FastAPI app asynchronously serving queries to SQLite FTS5 virtual tables representing sentence-level indexed corpora. The FTS5 unicode61 tokenizer enables handling of Greek Unicode characters with diacritics.

The core data structure per corpus is a SQLite file with a single FTS5 virtual table named 'sentences'. Each row contains sentence text and JSON metadata (register labels, source info). Precomputed marker frequency tables store counts of 93 discourse markers by register.

Query-time text processing includes tokenization, frequency counting, n-gram extraction, collocation scoring via pointwise mutual information (PMI), and log-ratio calculations for register comparison. Unlike many tools, Svarna does not rely on precomputed collocation indexes, instead calculating these on a sample basis at query time.

The frontend is a single static HTML/JavaScript page (approx. 3500 lines) with vanilla JS and CSS, no frameworks. It communicates with backend via REST APIs (16 endpoints) for functions like concordancing, frequency, discourse markers, text analysis, regex search, register comparison, and LLM integration. Chart.js from CDN provides visualizations of frequencies and networks.

The LLM layer is optional, allowing users to supply their own API keys to services like Gemini or OpenRouter. This enables pragmatic annotation and research queries over retrieved text excerpts but is decoupled from core corpus functionality.

Deployment uses Docker containers on Azure Container Apps, where the corpus database files are mounted from Azure File Share volumes, allowing updates without rebuilding the container. Source code, build scripts, and deployment files are all publicly released under MIT license.

4. Training Regime: Not applicable as Svarna is not a machine learning model; no training required. It operates as an indexed search and analytic engine.

5. Evaluation Protocol: Evaluation is qualitative, demonstrated through usage examples like frequency distributions of discourse markers across registers, dialectal lexical variation analysis, and collocation patterns by register. No formal statistical testing or adversarial evaluation is reported.

6. Reproducibility: All code, database build scripts, and deployment configs are open-source on GitHub. The data is drawn from public corpora, and prebuilt databases will be shared via a data repository upon publication. Users can reproduce results, extend by adding their own corpora using provided build pipelines, and deploy their own instances.

Concrete Example: For example, a user searching for the adversative discourse marker ωστόσο (‘however’) can immediately get its frequency profile across registers (high in Wikipedia and parliamentary discourse; almost absent in subtitles), demonstrating formal register usage. Collocation networks for polysemous words like βαρύ (‘heavy’) also illustrate corpus insights possible with Svarna’s interface and MI collocation statistics.

Technical innovations

• Integration of five large, freely available modern Greek corpora across institutional, literary, dialectal, social media, and historical registers into a single searchable SQLite FTS5-based platform.
• On-the-fly query-time text processing and collocation extraction using mutual information over corpus samples rather than precomputing large indexes, enabling a lightweight and extensible architecture.
• Precomputed frequency profiles for a linguistically validated lexicon of 93 Greek discourse markers across multiple registers for immediate distributional analysis.
• An optional LLM annotation layer integrated as a user-key-enabled service to facilitate pragmatic classification and open-ended corpus research without incurring platform costs or restrictions.

Datasets

• Corpus Database — 473 million words — aggregated from Wikipedia, parliamentary minutes, CC-100 web crawls, OpenSubtitles, Europarl, Leipzig Corpus, Greek treebanks
• Literature Database — 26.6 million words — Project Gutenberg Greek texts, BabyLM child corpus, interwar poetry, Tesserae classical Greek corpus
• Dialectal Database — 5.9 million words — GRDD+ dialect groups for nine Greek dialects
• Social Media Database — 1.9 million words — Greek Twitter/X data labeled for toxicity
• Historical Database — 0.12 million words — CLARIN:EL Crete collections

Baselines vs proposed

• No direct quantitative baseline comparisons with established corpus tools reported; qualitative functional comparisons note Svarna is more lightweight and accessible than Sketch Engine or CQPweb but lacks precomputed word sketches and advanced parsing.
• Collocation extraction uses mutual information scores computed on sample matches vs Sketch Engine’s precomputed models — enabling faster index building at the expense of some accuracy.
• Frequency normalization and register distribution comparisons are demonstrated on Greek-specific corpora; no cross-language baselines.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.00970.

Fig 2: KWIC (Concordancer) interface. The search term is highlighted in the center column,

Fig 3: Frequency tab. The million-unit normalized frequency of the search term is displayed as a

Fig 4: The discourse-markers tab. The lexicon of 93 markers is browsable by functional category

Fig 5: The text-analysis tab, here showing the most frequent 4-grams in the corpus. The sub-tabs

Fig 6: LLM layer applied to actual classification.

Limitations

• Corpus composition is opportunistic and unbalanced, with overrepresentation of institutional texts (Wikipedia, parliamentary minutes) and underrepresentation of informal conversation or personal correspondence.
• Collocation frequencies and statistics are computed at query time over sample matches rather than full corpus, making them approximate and potentially less accurate for very common words.
• Currently, adding new corpora requires technical knowledge to run local build scripts and deploy self-hosted instances; no in-app upload interface exists yet.
• No formal evaluation metrics or statistical validation of search or collocation accuracy against gold standards.
• No adversarial or robustness analysis performed given the system’s focus on accessibility rather than security.
• LLM integration depends on user-supplied external API keys, so it is not a built-in, internally controlled feature.

Open questions / follow-ons

• How can Svarna’s corpus composition be made more balanced to better represent informal, conversational, and spoken Greek registers?
• What methods could optimize collocation extraction accuracy and speed by precomputing some statistics without sacrificing extensibility?
• How can in-app corpus uploading and indexing pipelines be designed to lower technical barriers for non-expert users to add data?
• What is the efficacy and reliability of the optional LLM integration for pragmatic annotation and exploratory research over Greek corpus data?

Why it matters for bot defense

Although Svarna is not a security or bot-detection system, it offers valuable lessons for practitioners interested in linguistic resource infrastructure for underrepresented languages. Bot defense systems leveraging language analysis might benefit from insights on managing multi-register, dialectally diverse corpora with scalable, lightweight text indexing and querying. The design approach of on-demand statistical computation over large text collections balances performance and extensibility—a consideration relevant for real-time language-based bot-detection heuristics. Additionally, the open-source, containerized deployment model demonstrates how to build accessible language tools without prohibitive institutional barriers, serving as a model for inclusive development of linguistic assets supporting CAPTCHA challenges or content moderation in Greek or similar languages.

Cite

@article{arxiv2607_00970,
  title={ Svarna: An Open Corpus Workbench for Modern Greek },
  author={ Stergios Chatzikyriakidis },
  journal={arXiv preprint arXiv:2607.00970},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.00970}
}

Read the full paper

• arXiv:2607.00970 (abstract)
• arXiv:2607.00970 (PDF)