Skip to content

syncalong.textnorm

Shared text normalization used by both the lyrics parser and the transcriber, so the two sides of the aligner normalize identically.

textnorm

Shared text normalization for lyric and transcript words.

Both sides of the aligner must normalize identically so fuzzy matching compares like with like — keep this the single source of truth.

normalize 

normalize(text: str) -> str

Lowercase, strip punctuation and accents, and collapse whitespace.

Apostrophes are deleted so contractions stay one token ("don't" → "dont"); every other punctuation mark becomes a word boundary so hyphenated compounds split the way Whisper transcribes them ("déjà-vu" → "deja vu"). Both sides of the aligner must call this so fuzzy matching compares like with like.

Parameters:

Name Type Description Default
text str

Arbitrary input text.

 required

Returns:

Type Description
str

The normalised text.
Source code in src/syncalong/textnorm.py 
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
def normalize(text: str) -> str:
    """Lowercase, strip punctuation and accents, and collapse whitespace.

    Apostrophes are deleted so contractions stay one token ("don't" →
    "dont"); every other punctuation mark becomes a word boundary so
    hyphenated compounds split the way Whisper transcribes them ("déjà-vu" →
    "deja vu"). Both sides of the aligner must call this so fuzzy matching
    compares like with like.

    Args:
        text: Arbitrary input text.

    Returns:
        The normalised text.
    """
    text = unicodedata.normalize("NFKD", text)
    # Drop combining accent marks — deleting, not spacing, so accented
    # letters mid-word don't split the word.
    text = "".join(c for c in text if not unicodedata.combining(c))
    text = text.lower()
    text = text.replace("'", "").replace("’", "")
    text = re.sub(r"[^\w\s]", " ", text)
    return re.sub(r"\s+", " ", text).strip()