non-profit · solo-maintained · opening to contributors
About twelve million people write Tunisian Derja every day — in texts, in comments, in the middle of French sentences — and there is almost no proper record of it. OpenDerja_TN is a platform where Tunisians write, correct and confirm their own regional Derja. What comes out is a citable corpus, a lexicon, and a translator built directly on top of it. Text first; voice by design in a later phase.
A grandmother in the northwest says a word nobody else uses. When she stops, it's gone — nobody wrote it down. That happens constantly, and nobody is on it.
What it is
The platform runs on text from day one. The consent model and the schema for voice exist from the first migration — not bolted on years later when it can't be collected retroactively.
Coverage per region is a number that's tracked, published and recruited against — not something hoped to work out. What's online is Tunis-skewed; targeted elicitation is the counterweight.
Academic corpora stop the day the paper ships. This one is built to keep growing, and every confirmed contribution improves the tools the same day.
Uses the existing CODA-TUN and NOTA spelling conventions instead of inventing another standard.
The corpus is exposed as a tool people have a reason to open — a translator, a lexicon, a compare view — not only a file researchers have a reason to cite.
Why you'd use it
The corpus is the foundation; these are what you actually come for.
Derja → MSA / French / English in any script, and — the part nobody else offers — MSA / French / English → Derja with a region selector. “Say this in southern Derja” is a thing no existing tool can do.
A versioned, openly licensed, regionally tagged dataset with honest coverage stats, a DOI, and per-source licensing tiers. It returns what Tunisians actually wrote, or nothing — which is why it's citable.
Open data you can download or query: parallel pairs, region-tagged sentences, a lexicon with regional variants and word origins. No API key, no cost, nothing in the request path that can be down.
One box: a word, a sentence, a paragraph. Region as four buttons, consent as one checkbox, no account. Three seconds and it's in the record.
What makes it different
It's not the first Tunisian corpus (Tunisiya, TARIC, TSAC and others exist), not the first standardisation of Tunisian Arabic (CODA-TUN did that), not the first Arabic-dialect crowdsourcing platform. What no existing platform combines in one place: text now and voice by design later; community confirmation instead of one authority's word; region and era tagged together, so old Derja sits next to today's four regions; a working translator, not just a file; open data anyone can pull; continuously maintained as one open service.
| Region | Covers |
|---|---|
| northwest | Kef, Jendouba, Siliana, Kasserine, the western highlands |
| north | Tunis, Bizerte, Nabeul, Ariana, Manouba, Ben Arous |
| sahel | Sousse, Monastir, Mahdia, Sfax, the coastal centre |
| south | Gabès, Medenine, Tataouine, Tozeur, Kebili, Douz, Gafsa |
| Axis | Values | Note |
|---|---|---|
| scope | pan-Tunisian · regional | Whether it's marked at all, not how widely it's used |
| region | a set of: northwest · north · sahel · south | One or more. Empty when pan-Tunisian — dialect is a continuum, so region is a set, never one value. |
| era | contemporary · historical | “Old Derja” lives here, not in region |
| setting | urban · rural · unknown | Cuts across region, doesn't follow it |
| register | neutral · formal · vulgar · archaic | Vulgar is filtered by default with an honest opt-in toggle, never dropped from exports |
“Northwest + rural + historical” is a real thing you can ask for — and that exact combination is the material nobody has collected, which is most of the reason this exists.
The core loop
The whole system is built around this loop — whether the text is typed in by a contributor or pulled from a public source by a scraper.
Someone contributes it, or a scraper pulls it from an open, dump-based source (Wikipedia, Common Crawl filtered to .tn, Tatoeba, Tunisian forums under their own ToS).
Is it actually Derja? Too short or too long? Is it a near-duplicate of something already in the corpus (MinHash/LSH)?
If it looks new and valid. A duplicate triggers “we already have this — does your region say it differently?” instead of a silent rejection.
Review it, tag its region, confirm it, translate it, link its words to lexicon entries.
Nothing one person writes is automatically “true.” A tag or translation stays a proposal until enough independent people agree.
Enough agreement → it becomes the trusted answer. Disagreement → a region-routed reviewer decides. Only the agreed answer is ever shown publicly.
The translator
Word and phrase lookup runs through the lexicon: exact match, then normalised match, then trigram similarity for typos. Sentences are retrieved by trigram similarity over attested corpus items with their human translations.
Into Derja with a region selector is the demo that makes people send the link on.
Every accepted annotation improves it immediately — no retraining, no snapshot to promote, no regression risk. A retrieval translator can only get better.
No attestation, no guess. It shows what it found, the nearest approximate neighbours, what's missing by region (“attested in north and sahel. No southern attestation.”), and a one-click prompt for exactly that gap.
A live counter: this translator knows N words and M sentences across four regions, up from last month. On day one that number is embarrassingly small and it stays visible anyway.
It's a corpus lookup, not machine translation. It returns what Tunisians have actually written, or it returns nothing. That's a weaker product than a fluent model and a more honest one — and the honesty is why a researcher can cite it. Anything generated is a later phase, always labelled, never mixed in with attested results.
Architecture
Splitting it this way means each piece can be understood, tested and changed on its own.
| Plane | Language | What it does |
|---|---|---|
| control-plane/ | TypeScript | backend (NestJS API), worker (background jobs), scrapers. Almost all the logic — and it stays TypeScript permanently; the hard problems here are the rules, not raw speed. |
| data-plane/ | Rust | CPU-bound work measured to need it. Today: near-duplicate detection. Everything else is a stub until profiling says otherwise — a piece gets rewritten in Rust only once it's proven a bottleneck. |
| intelligence-plane/ | Python (FastAPI) | ML inference and training, phase-gated. A health-check stub for now — nothing in the request path ever waits on it, on purpose. |
| frontend/ | TypeScript (Next.js) | The only piece a visitor touches. Never talks to the database directly — always through the backend API, so there's exactly one place the rules are enforced. |
Full picture with diagrams, for developers and non-developers: docs/ARCHITECTURE.md. Every endpoint with request/response shapes: docs/API_REFERENCE.md.
Where things actually stand
The project keeps an honest ledger of what's been driven end-to-end versus built-but-unverified versus not built yet. The short version:
Auth (password, OAuth, 2FA, anonymous contributor sessions), the task engine, consensus, admin tooling, a scraper pipeline, a Rust dedup service. Real unit coverage. The next milestone is running it against a live deployment — that's real work, not a formality.
Every page in the spec exists as a stub that builds and returns 200, not a working page. Seven shared annotation components don't exist yet. docs/API_REFERENCE.md already has every endpoint worked out.
The Rust plane does one measured job. The Python plane is a health-check stub. Each stays that way until there's a measured reason — not a phase timer.
If you're picking up a PR here expecting a finished product, you'll be disappointed. If you're expecting a real, working backend with obvious frontend gaps to fill — that's accurate.
Contribute
Highest-value work today. Turn route stubs into real pages against a documented API, and build the shared annotation components.
Surgical fixes — bugs, missing test coverage, filling a documented gap (see db/DATABASE_NOTES.md).
Not yet — the contribution flow needs to be live and reviewed first. Opens once the frontend catches up.
Always useful, no experience threshold.
One project-wide rule
No code comments, anywhere, no exceptions. If you genuinely believe one is needed, say so in the PR description and let the maintainer decide the wording. A PR that adds comments gets asked to remove them.
Planning something substantial? Open an issue first — it's cheaper to redirect a plan than a finished PR. Review times depend on one person's availability; a slow response isn't a signal your contribution isn't wanted.
Get started
git clone https://github.com/firaslamouchi21/Open_Derja_Tn.git && cd Open_Derja_Tn pnpm install && cp .env.example .env # postgres, pgbouncer, redis, stubs pnpm infra:up # pnpm --filter @open-derja/db run migrate:deploy # :3000 pnpm --filter @open-derja/backend run start:dev # :3001 pnpm --filter @open-derja/frontend run dev Replace every change-me value in .env — AUTH_SECRET especially (openssl rand -base64 32). CI is disabled pending the public launch, so run the test suites locally (pnpm test:unit, cargo test) before opening a PR.
License
Apache 2.0 — the whole repository. Permissive on purpose: adoption matters more than control.
Licensed separately, and still being decided between share-alike and non-commercial. Not about revenue — there isn't any — but about stopping someone walling off what contributors built for free and selling it back to them.
At submission, one explicit checkbox: this material can go into an openly licensed dataset, including commercial use — covering the annotated record, not just the raw text. A separate voice clause ships with it from day one, years before recording exists, because it can't be collected retroactively.
Contact
Want to contribute, have a question about the architecture, or work on Tunisian NLP and want to compare notes — this goes straight to Firas's inbox and you'll get an auto-reply to confirm.