Pipelines¶

Ready-to-use pipelines for detecting intertextual parallels in Latin literature.

Each pipeline loads its own models and exposes a single run() method that accepts two Document objects and returns scored results.

TwoStagePipeline¶

Two-stage pipeline: embedding retrieval + classification.

Combines a fast embedding-based retrieval step with a more expensive classification step to efficiently identify intertextual parallels in large corpora.

Pipeline steps:

Retrieval - Encode all segments with a sentence-transformer model and retrieve the top_k most similar source segments for each query segment using cosine similarity.
Classification - Feed each query-candidate pair into a fine-tuned sequence-classification model. The positive-class probability is used as the judgment score.

PARAMETER	DESCRIPTION
`classification_name`	HuggingFace model identifier for the sequence-classification model. TYPE: `str` DEFAULT: `'julian-schelb/xlm-roberta-large-class-lat-intertext-v1'`
`embedding_model_name`	HuggingFace model identifier for the sentence-transformer. TYPE: `str` DEFAULT: `'julian-schelb/multilingual-e5-large-emb-lat-intertext-v1'`
`device`	Torch device string (`"cpu"`, `"cuda"`, …). TYPE: `str \| int \| None` DEFAULT: `None`
`pos_class_idx`	Index of the positive class in the classifier output. TYPE: `int` DEFAULT: `1`
`label_names`	Optional class labels for binary or multiclass models. TYPE: `Sequence[str] \| Mapping[int \| str, str] \| None` DEFAULT: `None`
`positive_class_ids`	Optional class ids that count as intertextual links. TYPE: `Sequence[int] \| None` DEFAULT: `None`
`positive_labels`	Optional labels that count as intertextual links. TYPE: `Sequence[str] \| None` DEFAULT: `None`
`negative_labels`	Optional labels that count as non-links. TYPE: `Sequence[str] \| None` DEFAULT: `None`
`emit_class_metadata`	Whether to attach predicted labels and class probabilities to results. TYPE: `bool \| None` DEFAULT: `None`

Example

from locisimiles.pipeline import ClassificationPipelineWithCandidategeneration
from locisimiles.document import Document

# Load documents
query = Document("query.csv")
source = Document("source.csv")

# Define pipeline
pipeline = ClassificationPipelineWithCandidategeneration(device="cpu")

# Run pipeline
results = pipeline.run(query=query, source=source, top_k=10)

device `property` ¶

device: str

Device used by the classification judge.

ClassificationPipeline¶

Classification pipeline for exhaustive pairwise comparison.

For each query segment every source segment is considered as a candidate. Each query-source pair is then fed to a fine-tuned sequence-classification model that outputs the probability of the pair being an intertextual match.

Pipeline steps:

Candidate generation - Create all possible query-source pairs.
Classification - Score each pair with a HuggingFace sequence-classification model. The positive-class probability is used as the judgment score.

PARAMETER	DESCRIPTION
`classification_name`	HuggingFace model identifier for the sequence-classification model. TYPE: `str` DEFAULT: `'julian-schelb/xlm-roberta-large-class-lat-intertext-v1'`
`device`	Torch device string (`"cpu"`, `"cuda"`, ...). TYPE: `str \| int \| None` DEFAULT: `None`
`pos_class_idx`	Index of the positive class in the classifier output. TYPE: `int` DEFAULT: `1`
`label_names`	Optional class labels for binary or multiclass models. TYPE: `Sequence[str] \| Mapping[int \| str, str] \| None` DEFAULT: `None`
`positive_class_ids`	Optional class ids that count as intertextual links. TYPE: `Sequence[int] \| None` DEFAULT: `None`
`positive_labels`	Optional labels that count as intertextual links. TYPE: `Sequence[str] \| None` DEFAULT: `None`
`negative_labels`	Optional labels that count as non-links. TYPE: `Sequence[str] \| None` DEFAULT: `None`
`emit_class_metadata`	Whether to attach predicted labels and class probabilities to results. TYPE: `bool \| None` DEFAULT: `None`

Example

from locisimiles.pipeline import ClassificationPipeline
from locisimiles.document import Document

# Load documents
query = Document("query.csv")
source = Document("source.csv")

# Define pipeline
pipeline = ClassificationPipeline(device="cpu")

# Run pipeline
results = pipeline.run(query=query, source=source)

device `property` ¶

device: str

Device used by the classification judge.

RetrievalPipeline¶

locisimiles.pipeline.retrieval.RetrievalPipeline ¶

RetrievalPipeline(
    *,
    embedding_model_name: str = "julian-schelb/multilingual-e5-large-emb-lat-intertext-v1",
    device: str | int | None = None,
    top_k: int = 10,
    similarity_threshold: Optional[float] = None,
)

Retrieval pipeline based on embedding similarity.

Uses a sentence-transformer model to encode query and source segments into dense vectors and ranks candidates by cosine similarity.

Pipeline steps:

Encoding - Encode all query and source segments with a sentence-transformer model.
Retrieval - For each query, retrieve all source segments ranked by cosine similarity.
Thresholding - Mark the top_k most similar candidates as positive, or all candidates above similarity_threshold if provided.

PARAMETER	DESCRIPTION
`embedding_model_name`	HuggingFace model identifier for the sentence-transformer. TYPE: `str` DEFAULT: `'julian-schelb/multilingual-e5-large-emb-lat-intertext-v1'`
`device`	Torch device string (`"cpu"`, `"cuda"`, …). TYPE: `str \| int \| None` DEFAULT: `None`
`top_k`	Number of top candidates to mark as positive. TYPE: `int` DEFAULT: `10`
`similarity_threshold`	If provided, uses threshold instead of top-k. TYPE: `Optional[float]` DEFAULT: `None`

Example

from locisimiles.pipeline import RetrievalPipeline
from locisimiles.document import Document

# Load documents
query = Document("query.csv")
source = Document("source.csv")

# Define pipeline
pipeline = RetrievalPipeline(device="cpu")

# Run pipeline
results = pipeline.run(query=query, source=source, top_k=10)

device `property` ¶

device: str

Device used by the embedding generator.

RuleBasedPipeline¶

locisimiles.pipeline.rule_based.RuleBasedPipeline ¶

RuleBasedPipeline(
    *,
    min_shared_words: int = 2,
    min_complura: int = 4,
    max_distance: int = 3,
    similarity_threshold: float = 0.3,
    stopwords: Optional[Set[str]] = None,
    use_htrg: bool = False,
    use_similarity: bool = False,
    pos_model: str = "enelpol/evalatin2022-pos-open",
    spacy_model: str = "la_core_web_lg",
    device: Optional[str] = None,
)

Rule-based pipeline for lexical intertextuality detection.

Identifies potential quotations, allusions, and textual reuse between Latin documents using a multi-stage rule-based approach.

Pipeline steps:

Text preprocessing - Normalise prefix assimilations (e.g. adt- → att-, inm- → imm-), quotation marks, and whitespace connectors. Apply genre-specific phrasing rules for prose or poetry.
Text matching - Tokenise both documents and find shared non-stopword tokens between every query-source segment pair.
Distance criterion - Discard matches where shared words are too far apart (controlled by max_distance).
Scissa filter - Compare punctuation patterns around shared words to strengthen evidence of deliberate textual reuse.
HTRG filter (optional) - Part-of-speech analysis using a HuggingFace token-classification model. Requires torch.
Similarity filter (optional) - Word-embedding similarity check using spaCy vectors. Requires spacy.

PARAMETER	DESCRIPTION
`min_shared_words`	Minimum number of shared non-stopwords required. TYPE: `int` DEFAULT: `2`
`min_complura`	Minimum adjacent tokens for complura detection. TYPE: `int` DEFAULT: `4`
`max_distance`	Maximum distance between shared words. TYPE: `int` DEFAULT: `3`
`similarity_threshold`	Threshold for semantic similarity filter. TYPE: `float` DEFAULT: `0.3`
`stopwords`	Set of stopwords to exclude. Uses defaults if `None`. TYPE: `Optional[Set[str]]` DEFAULT: `None`
`use_htrg`	Whether to apply HTRG (POS-based) filter. Requires torch. TYPE: `bool` DEFAULT: `False`
`use_similarity`	Whether to apply similarity filter. Requires spacy. TYPE: `bool` DEFAULT: `False`
`pos_model`	HuggingFace model name for POS tagging. TYPE: `str` DEFAULT: `'enelpol/evalatin2022-pos-open'`
`spacy_model`	spaCy model name for embeddings. TYPE: `str` DEFAULT: `'la_core_web_lg'`
`device`	Device for neural models (`"cuda"`, `"cpu"`, or `None` for auto). TYPE: `Optional[str]` DEFAULT: `None`

Example

from locisimiles.pipeline import RuleBasedPipeline
from locisimiles.document import Document

# Load documents
query = Document("query.csv")
source = Document("source.csv")

# Define pipeline
pipeline = RuleBasedPipeline()

# Run pipeline
results = pipeline.run(query=query, source=source)

min_shared_words `property` ¶

min_shared_words: int

Minimum number of shared non-stopwords required.

min_complura `property` ¶

min_complura: int

Minimum adjacent tokens for complura detection.

max_distance `property` ¶

max_distance: int

Maximum distance between shared words.

similarity_threshold `property` ¶

similarity_threshold: float

Threshold for semantic similarity filter.

stopwords `property` ¶

stopwords: Set[str]

Set of stopwords to exclude.

use_htrg `property` ¶

use_htrg: bool

Whether HTRG (POS-based) filter is enabled.

use_similarity `property` ¶

use_similarity: bool

Whether similarity filter is enabled.

device `property` ¶

device: str

Device for neural models.

load_stopwords ¶

load_stopwords(filepath: Union[str, Path]) -> None

Load stopwords from a file (one word per line).

PARAMETER	DESCRIPTION
`filepath`	Path to stopwords file. TYPE: `Union[str, Path]`

Word2VecRetrievalPipeline¶

Burns-style Word2Vec bigram retrieval + threshold judge.

Burns-style Word2Vec retrieval pipeline.

PARAMETER	DESCRIPTION
`model_path`	Path to a local gensim `.model` file. TYPE: `str \| Path` DEFAULT: `DEFAULT_WORD2VEC_MODEL_PATH`
`top_k`	Number of candidates to mark as positive via threshold judge. TYPE: `int` DEFAULT: `10`
`similarity_threshold`	Optional score threshold for positive labels. TYPE: `Optional[float]` DEFAULT: `None`
`interval`	Maximum token gap for bigrams. TYPE: `int` DEFAULT: `0`
`order_free`	Whether to treat bigrams as order-insensitive. TYPE: `bool` DEFAULT: `False`

LatinBertRetrievalPipeline¶

Gong-style contextual Latin BERT token retrieval + threshold judge.

Retrieval-only pipeline using contextual token similarity.

PARAMETER	DESCRIPTION
`model_name`	HuggingFace model identifier. TYPE: `str` DEFAULT: `DEFAULT_CONTEXTUAL_BERT_MODEL_NAME`
`model_path`	Optional local path to a model directory. TYPE: `str \| Path \| None` DEFAULT: `None`
`device`	Torch device. TYPE: `str \| int \| None` DEFAULT: `None`
`top_k`	Number of candidates marked positive by default. TYPE: `int` DEFAULT: `10`
`similarity_threshold`	Optional score threshold overriding top-k labels. TYPE: `Optional[float]` DEFAULT: `None`
`max_length`	Maximum tokenized length per segment. TYPE: `int` DEFAULT: `256`
`min_token_length`	Minimum token length retained for scoring. TYPE: `int` DEFAULT: `2`
`use_stopword_filter`	Whether to filter common Latin stopwords. TYPE: `bool` DEFAULT: `True`

LatinBertTwoStagePipeline¶

Contextual Latin BERT retrieval + classification.

Two-stage pipeline with contextual Latin BERT retrieval.

Stage 1 performs contextual token-level retrieval. Stage 2 applies a sequence classifier for reranking/labeling.

TfidfRetrievalPipeline¶

TF-IDF lexical retrieval + threshold judge.

TF-IDF lexical retrieval pipeline.

PARAMETER	DESCRIPTION
`top_k`	Number of candidates to mark as positive via threshold judge. TYPE: `int` DEFAULT: `10`
`similarity_threshold`	Optional score threshold for positive labels. TYPE: `Optional[float]` DEFAULT: `None`
`lemmatize`	Whether to lemmatize tokens with CLTK before vectorizing. TYPE: `bool` DEFAULT: `True`
`lowercase`	Whether to lowercase tokens before vectorizing. TYPE: `bool` DEFAULT: `True`
`ngram_range`	`(min_n, max_n)` token n-gram range. TYPE: `tuple[int, int]` DEFAULT: `(1, 1)`
`max_features`	Maximum vocabulary size. TYPE: `int` DEFAULT: `50000`

BM25RetrievalPipeline¶

Okapi BM25 lexical retrieval + threshold judge — the benchmark's best retriever.

BM25 (Okapi) lexical retrieval pipeline — the benchmark's best retriever.

PARAMETER	DESCRIPTION
`top_k`	Number of candidates to mark as positive via threshold judge. TYPE: `int` DEFAULT: `10`
`similarity_threshold`	Optional score threshold for positive labels. TYPE: `Optional[float]` DEFAULT: `None`
`lemmatize`	Whether to lemmatize tokens with CLTK before indexing. TYPE: `bool` DEFAULT: `True`
`lowercase`	Whether to lowercase tokens before indexing. TYPE: `bool` DEFAULT: `True`
`k1`	BM25 term-frequency saturation parameter. TYPE: `float` DEFAULT: `1.5`
`b`	BM25 length-normalization parameter. TYPE: `float` DEFAULT: `0.75`

BM25TwoStagePipeline¶

BM25 retrieval + classification — the benchmark's "best combined" configuration.

Two-stage pipeline with BM25 retrieval.

Stage 1 ranks candidates by Okapi BM25 score over (optionally lemmatized) Latin text. Stage 2 applies a sequence classifier for reranking/labeling.

BM25LexicalTwoStagePipeline¶

BM25 retrieval + trained LogReg/GBDT lexical classifier — the benchmark's "best non-neural" configuration. No neural model required end to end.

Two-stage pipeline with BM25 retrieval and a lexical classifier reranker.

Stage 1 ranks candidates by Okapi BM25 score over (optionally lemmatized) Latin text. Stage 2 applies a trained LogReg/GBDT classifier (TF-IDF/Jaccard/overlap features) for reranking/labeling — no neural model required end to end.

Pipelines¶

TwoStagePipeline¶

device property ¶

ClassificationPipeline¶

device property ¶

RetrievalPipeline¶

locisimiles.pipeline.retrieval.RetrievalPipeline ¶

device property ¶

RuleBasedPipeline¶

locisimiles.pipeline.rule_based.RuleBasedPipeline ¶

min_shared_words property ¶

min_complura property ¶

max_distance property ¶

similarity_threshold property ¶

stopwords property ¶

use_htrg property ¶

use_similarity property ¶

device property ¶

load_stopwords ¶

Word2VecRetrievalPipeline¶

LatinBertRetrievalPipeline¶

LatinBertTwoStagePipeline¶

TfidfRetrievalPipeline¶

BM25RetrievalPipeline¶

BM25TwoStagePipeline¶

BM25LexicalTwoStagePipeline¶

device `property` ¶

device `property` ¶

device `property` ¶

min_shared_words `property` ¶

min_complura `property` ¶

max_distance `property` ¶

similarity_threshold `property` ¶

stopwords `property` ¶

use_htrg `property` ¶

use_similarity `property` ¶

device `property` ¶