Qurandia: matching two datasets whose numbers disagree, by fingerprinting the text

Qurandia was built as an open and free knowledge resource for people who want to read, understand and learn the Qur’an: translations, word-by-word analysis, comparative tafsir, revelation contexts, the hadith corpus, a root dictionary, a thematic index, search and a learning academy, all under one roof.

The interesting part is not the interface, it is the data underneath. A platform like this is never a copy of a single source; it has to overlay datasets produced in complete ignorance of one another. And those datasets do not recognise each other.

Trusting the fingerprint of the text, not the number

The hardest case showed up on the hadith side. The Arabic text and the English translations come from one dataset, the Turkish translations from an entirely different one. Both address hadiths by number — but the numbers do not count the same thing: one uses a collection-wide sequential index, the other follows a different numbering scheme. Matched directly, the two datasets agree 0–14% of the time. Trusting the number means silently attaching the wrong translation to the wrong hadith.

The approach I tried drops the number entirely and uses a fingerprint of the Arabic text. The text is NFC-normalised, orthographic variants collapse to a single letter (أ إ آ ٱ → ا, ى → ي, ؤ → و, ئ → ي, ة → ه), and everything outside the \x{0621}-\x{064A} range — diacritics, stop marks, whitespace, punctuation — is stripped. What remains is a bare letter sequence identical in both datasets. Match rates land at 91–99% depending on the collection.

The same discipline runs through the whole import pipeline: every command is idempotent (upsert on a natural unique key), every run writes an import_runs row, and tests never touch the network — they all work from fixtures.

Audio drew a boundary too: 6,236 mp3 files are self-hosted rather than leaning on a third party, and the player lives in a persistent dock that survives wire:navigate transitions — recitation does not stop when the page changes.

Where it stands, in numbers

114 surahs, 6,236 ayahs, 18,708 translations across three sources, 77,429 words and 1,651 roots, 8,117 tafsir entries, 197 revelation-context notes in both languages, and 34,178 hadiths across six collections. Turkish hadith translation coverage sits at 83% — the remaining gap is on the source side, not ours.

Search runs on Postgres full-text search plus pg_trgm; there is no Scout and no Meilisearch, because neither was needed. There is no third-party UI or JS library either: waveform, charts, icons, dates — all hand-built SVG and Alpine. The sitemap reaches roughly 85,000 URLs, indexed so each locale gets one entry.

The quality gates hold: 127/127 Pest tests green against a real Postgres, Larastan level 8 and Pint clean. Mobile Lighthouse on /quran/al-fatihah scores 100 for accessibility, 100 for best practices and 92 for SEO.

The remaining few percent are still by hand

The infrastructure is up and the platform is live. What continues is the data, not the code.

Fingerprint matching sits at 91–99% depending on the collection, which means the remaining few percent are still on the table: records that do not line up even after orthographic variants are normalised — the point where automatic matching ends. Turkish hadith translation coverage sits at 83% too, and that gap is on the source side; software cannot close it, only another source or manual translation can.

So the open end is measurable and boring: pushing coverage towards a hundred percent. Every unmatched hadith is a measure of how far reducing text to a fingerprint actually gets you.