RAG-SyntecIN PROGRESS

The Syntec collective agreement (IDCC 1486) from Légifrance: 172 documents split into 8,447 chunks, versioned in a hashed manifest.

Solo

1. Ingestion and chunking: 500-character chunks with 100 characters of overlap.
2. A Chroma index with top-3 dense retrieval, embeddings from a local model (Ollama) or OpenAI.
3. Generation with a local model or gpt-4o-mini behind the same interface.
4. A FastAPI /query endpoint, Docker Compose and 30 pytest tests.
5. A labelled set of 55 questions in four classes, plus 20 held out, and the recall@k, precision@k, MRR and false-refusal metrics.

Refuse before generating

If the nearest chunk is farther than a calibrated distance, the API refuses without calling the LLM.

Versioned corpus

A hashed manifest ties every chunk to a document version.

Local or hosted models

The same pipeline runs on Ollama or on OpenAI.

Threshold calibration only: in-topic questions stayed below a distance of 0.701, off-topic ones above 0.780; the threshold sits at 0.74.

• No retrieval evaluation has run yet: the question set and the metrics exist, the runner does not.
• Dense retrieval only: hybrid search and reranking are planned, not built.
• The refusal threshold rests on six questions.