Fixing the ayah the owner named broke the ayah next to it

The similar-ayat view on /quran/compare colours the words two passages share. In under an hour, four highlight rules shipped, each to answer one ayah the owner pointed at. Each rule was checked on that ayah and nowhere else. Two of them moved highlights nobody looked at, and the owner found each one a message later. The fourth would have too, had a diff over every pair not caught it first. This lesson is about why a rule that runs over the whole text can't be judged on one ayah, and about the two checks that finally held it still.

What the page does

The matcher (src/lib/quran/similar.ts, specified in docs/plans/quran-similar-ayat.md) reads the Quranic Arabic Corpus. It turns every noun, verb and adjective into its root and every other word with a lemma (كلا، إن، لا، ما) into that lemma. Then it compares every run of one to three ayat with every run elsewhere. A pair that shares enough rare roots, in roughly the same order, is kept. The whole Qur'an yields 5,082 pairs, written once to src/app/quran/compare/similar.json.

Finding the pairs is one problem. Colouring them is another, and this lesson is about the second. For each kept pair a second function, marks, decides which words to colour:

• Amber: the other side has this word letter for letter.
• Blue: the other side has a different word from the same root or particle, as إنه in المدثر ٥٤ against إنها in عبس ١١. That is the difference a memoriser has to learn, so it gets its own colour.

Note the shape of that function. It takes no ayah from the reader. It runs over all 5,082 pairs and every word in them, about 57,000 coloured words in all. A change to it is a change to every one of those words at once.

The page, view by view

Before the story, the whole page. /quran/compare answers one question at several scales: where does the same wording, or nearly the same wording, occur again? Four engines sit behind it, and each view is a window onto one or two of them.

• The repetition sidebar ranks surahs and surah pairs by how many repeated words they share. It loads the whole dataset once from /api/quran/repetition, so switching surahs costs no request.
• The surah view (?s=55) shows the phrases a surah repeats inside itself, the passages it shares with other surahs, and every similar-ayat pair that touches it.
• The pair view (?a=56&b=69) compares two surahs, or two ranges of ayat with af, at, bf and bt, so one surah can be compared with itself, as in Al-Rahman 46–61 against 62–77. Its default is the list of similar ayat (المتشابه). A switch turns to the older ayah-by-ayah alignment (view=ayat, آية بآية). Both answer which ayat of the two belong together, by different rules.
• The themes view (?mode=themes) lines up the topics the two surahs share.

Two kinds of word

The engines do not all see the same words, and most of the page's behaviour follows from which kind each one sees.

A normalized word is the printed word with its marks removed. src/lib/quran/normalize.ts strips the diacritics and Quranic annotation signs, writes every alif form (أ إ آ ٱ) as ا, ى as ي, ة as ه and ک as ك, and drops the basmala that opens every surah except the first and the ninth. وَإِنَّهُ becomes وانه. Two words are the same when their letters are the same. The alignment and both repetition finders work on these.

A token is what the morphology corpus says the word is built from. Here the same وانه is two things, a particle and a pronoun, and لقسم is a noun whose root is ق س م. The similar-ayat matcher works on tokens, which is how it can tell that أقسم and لقسم are related while their letters differ, and that وانه and انها share a particle while sharing only three letters.

Where a token comes from: roots and lemmas

The tokens come from the Quranic Arabic Corpus, stored in the repository as data/quran/quranic-corpus-morphology-0.4.txt. Each line is one segment of one word: a prefix, a stem or a suffix. Its location is (surah:ayah:word:segment), the text is in Buckwalter transliteration (an ASCII spelling of Arabic, with < for إ and ~ for the shadda), and a tag and a list of features follow. These are the first segments of الواقعة ٧٦, وَإِنَّهُ لَقَسَمٌ:

(56:76:1:1)  wa      CONJ  PREFIX|w:CONJ+
(56:76:1:2)  <in~a   ACC   STEM|POS:ACC|LEM:<in~|SP:<in~
(56:76:1:3)  hu,     PRON  SUFFIX|PRON:3MS
(56:76:2:1)  la      EMPH  PREFIX|l:EMPH+
(56:76:2:2)  qasamN  N     STEM|POS:N|LEM:qasam|ROOT:qsm|M|INDEF|NOM

Word 1 is three segments, the conjunction و, the particle إنّ and the pronoun ه. Word 2 is the emphatic ل and the noun قسم. A root (ROOT:qsm) is the consonantal skeleton a family of words is built on: ق س م gives قسم, أقسم and مقسم alike. A lemma (LEM:qasam) is the dictionary headword of one word, so قسم and أقسم have different lemmas and the same root.

src/lib/quran/similar.ts's readAyat turns each segment into at most one token:

1. A noun, verb, adjective or proper noun (N, V, ADJ, PN) that has a root becomes its root, r:qsm. Roots are what a memoriser confuses: تذكرة and ذكره, قسم and أقسم.
2. Any other segment with a lemma becomes its lemma, p:<in~, unless its tag is a preposition, conjunction, the article, a pronoun or the resumption particle (P, CONJ, DET, PRON, REM). So كلا, بل, إن, من and ما survive as particles, and و, ف, ال and ـه do not.
3. Everything else gives no token.

It also rebuilds each printed word from its segments, with the marks stripped, so a highlight can point at a whole word. A leading و or ف is left off that rebuilt form, which is why وإنه and إنه compare as the same word. The fromBuckwalter function in src/lib/quran/lemmas.ts turns a root back into Arabic letters for display, and throws on a character it does not know rather than guess.

Finding similar ayat

The matcher's job is the متشابه a memoriser trips on: المدثر ٥٤–٥٥ «كلا إنه تذكرة فمن شاء ذكره» against عبس ١١–١٢ «كلا إنها تذكرة فمن شاء ذكره», which differ in one letter and share no long run of identical words. The rules are in docs/plans/quran-similar-ayat.md, and the code is findSimilar.

Weights. A token's weight is its rarity, log(N / df), where N is the number of ayat in the Qur'an and df the number of ayat holding the token. A root in a handful of ayat weighs a lot; the root of الله weighs little. A particle's weight is halved, so a shared كلا helps a match but cannot make one.

Candidates. Comparing each of the 6,236 ayat with every other would be about 19 million pairs, each scored several ways. Instead, each root rarer than a cut-off (weight 2.5 or more, roughly one ayah in twelve) lists the ayat that hold it. Every two ayat on the same list gain that root's weight. A pair becomes a seed only when the weight it shares, divided by the size of the two ayat, already reaches 0.3 for the ayah alone or for it joined with the next ayah on either side. Only seeds are scored, which keeps the whole build to seconds.

Windows. A pattern one surah says in one ayah another says in two, so around each seed the matcher tries every window of 1 to 3 consecutive ayat inside one surah on each side. It skips a pair of windows that overlap each other, a pair inside one surah whose letters are identical (the repetition list already shows those), and a window whose first or last ayah shares nothing with the other side, so a match is never padded with a neighbour.

Score. For two windows with token lists x and y:

score = 0.5 × shared weight / d          (cosine: which tokens both have)
      + 0.5 × weighted LCS(x, y) / d     (order: the same tokens in the same order)
      + 0.1 if both open with the same two tokens
d     = √(weight of x's tokens × weight of y's tokens)

The first half asks which tokens the two windows share, the second whether they share them in the same order, through a longest common subsequence where each matched token counts its weight. The opening bonus is there because a memoriser goes wrong at the start of an ayah.

Keeping. A pair stays when its score is at least 0.5 and the tokens it shares weigh at least 6.5, which rules out a match on one common root, unless the score reaches 0.8, as with الواقعة ١ and الحاقة ١٥, which share only the root و ق ع but say the same thing. The survivors are sorted by score, and a pair is dropped when a better one already covers an overlapping window on both sides.

The result is 5,082 pairs, written once by npm run quran:similar to src/app/quran/compare/similar.json. Scoring the whole text takes too long for a request, and the text never changes, so the file changes only when the rules do. That is also why the story below could diff it before and after.

Aligning two surahs ayah by ayah

The view=ayat alignment, alignSurahs in src/lib/quran/compare.ts, is older and answers a different question: given two surahs, which stretches run in parallel? It reads normalized words from the database on each request and needs no generated file. Its rules are in docs/plans/quran-compare-page.md.

1. Each ayah pair scores the square of its longest run of shared words. A run of one word scores nothing. Squaring makes one run of six (36) worth more than three runs of two (12).
2. Anchors come from an order-preserving dynamic program, the same shape as a sequence alignment: it picks the highest-scoring set of ayah pairs that never cross, so if 56:75 pairs with 69:38, nothing before 56:75 can pair with anything after 69:38. A refrain repeated at a fixed offset gets a bonus of 32 for an anchor with a run of four or more that sits within five ayat of an earlier anchor on the same diagonal. Without it, Al-Rahman 46–61 against 62–77 shifted by two to chase one longer phrase; with it, the sixteen pairs come out in step.
3. A second link is added for a pair with a run of four or more next to an anchor on one side, so 57:1 and 57:2 can both pair with 64:1. The ayah that has two counterparts is shown once, with its counterparts stacked beside it, each link in its own colour.
4. Blocks. Consecutive anchors join one block when both surahs step the same number of ayat between them, five or fewer. A block stays with two anchors, or one anchor whose run is four or more. Every ayah between a block's first and last anchor is paired in step with its partner, and a pair that shares nothing says so.
5. The last five ayat of each surah form a block aligned by count, which claims nothing from the text, and the ayat between blocks collapse into one row that can be opened.

Aligning البقرة with آل عمران, 286 by 200 ayat, takes about 140 ms, almost all of it the table of longest runs.

Repeated phrases, inside a surah and across surahs

The sidebar and the surah view come from src/lib/quran/repetition.ts, which runs two finders on normalized words and merges what they find into one record per phrase (docs/plans/quran-repetition-merge.md).

Inside one surah, findPhrases in src/lib/quran/shifts.ts indexes every run of three words that does not cross an ayah boundary, then extends each pair of occurrences as far as the words keep matching. A phrase seen at two places counts when it is four words or longer and the two places are in different ayat. A phrase seen three times or more counts when it is longer than three words, or when it is three words seen five times or more. فبأي آلاء ربكما تكذبان in Al-Rahman is one phrase with 31 places. Across the Qur'an that gives 528 phrases, and 53 surahs have none.

Across surahs, findPassages in src/lib/quran/passages.ts starts from every run of ten identical words that occurs in two surahs, then extends it backwards and forwards. It tolerates a difference: when the words stop matching, it may skip up to four words on either side, but only if the next three words then match again. A ten-word run is paired at no more than its first 150 occurrences, since a run that common is ordinary diction rather than a shared passage. The finder returns 112 passages, which group into 89 phrases, since one phrase can recur in up to five surahs.

The two finders feed one 114 × 114 table: a cell counts the repeated words two surahs share, and the diagonal counts a surah's repetition of itself. The endpoint computes all of it from the database in about 150 ms and caches the response, since the text never changes.

Shared themes

The themes view does no text matching at all. Each surah's topics are stored as QuranTopic rows with an ayah range and an Arabic label, and a topic may carry a themeId. Topic slugs are per surah, so two surahs share a topic only when both carry the same theme id (src/lib/quran/topicEngine.ts). The comparison lines those up, which is why it can pair passages that share no word.

The engines side by side

The same pair of passages can turn up in more than one row of that table, by different rules, and each view says which rule it used.

Four rules in an hour

The pair under discussion was الواقعة ٧٥–٧٧ against الحاقة ٣٨–٤٠:

الواقعة ٧٥  فَلَا أُقْسِمُ بِمَوَاقِعِ النُّجُومِ        الحاقة ٣٨  فَلَا أُقْسِمُ بِمَا تُبْصِرُونَ
الواقعة ٧٦  وَإِنَّهُ لَقَسَمٌ لَوْ تَعْلَمُونَ عَظِيمٌ    الحاقة ٣٩  وَمَا لَا تُبْصِرُونَ
الواقعة ٧٧  إِنَّهُ لَقُرْآنٌ كَرِيمٌ                الحاقة ٤٠  إِنَّهُ لَقَوْلُ رَسُولٍ كَرِيمٍ

Here is the sequence, one owner message and one rule at a time.

1. "Why are وإنه and لا blue?" They were the same words as إنه and فلا on the other side, but the attached و and ف made the letters differ, so they fell through to "same root". Rule: ignore a leading و or ف when comparing words. Checked: both turned amber. Unchecked: وإنه in ٧٦ was now amber because of the إنه in الحاقة ٤٠, a different ayah that says something else.
2. "Why is لا amber?" Because الواقعة ٧٥ has فلا, now read as لا. The match is real and meaningless: one is an oath formula, the other a negation. Rule: stop colouring a particle that stands alone, away from any other coloured word. Checked: لا in ٣٩ went plain. Unchecked: إنه opens both ٧٧ and ٤٠, and it stands alone too, because لقرآن and لقول don't match. It lost its colour.
3. "إنه in ٧٧ and ٤٠ fails to match." Rule: an exception, so a lone particle stays coloured when the paired ayah has it at the same position. Checked: إنه came back. Unchecked: وإنه in ٧٦ was still amber, kept alive by its neighbour لقسم.
4. "Why is وإنه amber, and why is لقسم blue when ٣٨ shows no blue?" لقسم is blue because of the root ق س م in أقسم, but أقسم in ٣٨ is amber, since it matches أقسم in ٧٥ exactly. A reader sees a blue word with no blue partner. This is where the method changed, and the next section is about that.

The pattern is plain in hindsight. Every rule was proved on the example in the message and on nothing else. Every rule was correct for that example. The rule itself was a statement about tens of thousands of words, and one example is a sample of one.

Why the named ayah can't vouch for the rule

The owner's message names the place where a reader noticed something wrong. It does not name where the rule acts. A rule like "ignore a leading و" acts everywhere a word starts with و, and in Arabic that is a large share of the text. The one ayah checked was the one place guaranteed to come out right, because the rule was written to fix it.

There was a second trap. Each fix left the earlier fixes in place, so every rule ran on top of the last. Rule 3 was an exception to rule 2, which narrowed rule 1. By the third round, nobody could say from reading the code what a given word would get. The code's intent had stopped being visible in the code, so the only way left to know what it did was to run it and look at all of it.

The diff over every pair

Before the fourth change, the old similar.json was copied aside. After the change, a short script compared the two, pair by pair. For each pair it found which marks were lost and which were gained, and it totalled amber and blue across the file. It also printed a dozen changed pairs to read.

pairs 5082 -> 5082 | changed pairs 953 lost marks 79 gained 1469
amber 41629 -> 42675 | blue 15513 -> 15857
2,25,25|4,57,57 - +A25:15:0,A25:20:1,A25:22:0

That first run was itself a surprise. The new rule paired the ayat of two equal windows in order (٧٥↔٣٨، ٧٦↔٣٩، ٧٧↔٤٠) and compared a particle only with its paired ayah. That fixed وإنه. But the diff showed 1,469 new marks, and reading the samples showed why. In long ayat like البقرة ٢٥, a particle such as من or في now matched anywhere in the paired ayah, which is the same stray match as the لا complaint, only in long ayat that nobody had opened. One more narrowing, to the same position in the paired ayah, and the diff came back small enough to read row by row:

pairs 5082 -> 5082 | changed pairs 93 lost marks 79 gained 63

Ninety-three changed pairs is a list a person can scan in a few minutes. 953 is not, and the difference is the signal. A rule meant to move one class of word should move a small, explainable set of pairs. When the count is far larger than the class, the rule is doing something else too.

Invariants as tests

The blue-without-a-partner complaint was not about one ayah at all. It was about a property every pair should have: if a word is coloured because of something on the other side, that something should be coloured too. Stated that way, it can be checked over every pair, not sampled. The test in src/lib/quran/similar.test.ts loops over all 5,082 pairs and fails if any coloured word's root or particle is missing from the other side's coloured words. The fix is paired in similar.ts, which drops such marks. Together with a tooltip naming each word's partners, it answers the complaint for the whole file, not only for ٣٨.

The pre-PR review then found the mirror image of the problem in a test. "Keeps a lone two-word match" used two one-ayah ranges. Two ranges of equal length took a new shortcut that pairs ayat in order, so the test never reached the filter it was named for, and it would have passed with that filter reverted. The fix was to give it ranges of unequal length. To show the new version can fail, the filter was reverted by hand for one run. The test failed, and the filter was put back. A test that has never been seen to fail has not been shown to test anything.

The move worth stealing

When a change runs over a whole corpus (a highlight rule, a matcher threshold, a normaliser), three checks belong with it, in this order:

1. Diff the output before and after, over everything, and print the counts and a sample. Ask whether the count of changes matches the size of the class the rule targets.
2. Write the property, not the example, as a test that loops over the whole output. "No mark lacks a partner" covers 5,082 pairs; "٧٦ has no mark at word 0" covers one.
3. Break the code once and watch the test fail. Revert the line the test is about, run it, and put the line back.

The owner's example still matters. It is the first test case, the one that proves the rule handles the reported problem. It is not evidence that the rule handles anything else.

Read next

Primary sources: docs/plans/quran-similar-ayat.md for the matcher and the colouring rules as they stand; src/lib/quran/similar.ts (marks and paired); the short record Fixing the highlight the owner pointed at broke the one next to it; Lesson 0003, whose check that never ran is a cousin of the test that could not fail.