QiQSSKJrJr SSKJrJr SSKJrJr SSK J r "SS\ 5r \ "5Rr SrSrS rS rSS jrSS jrg ))tee zip_longest)escapeunescape)Paralleldelayed)tqdmc b\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSr\\\\\ \ \ \ \\\/ rSrg)CJKChars z An object that enumerates the code points of the CJK characters as listed on http://en.wikipedia.org/wiki/Basic_Multilingual_Plane#Basic_Multilingual_Plane )ii)i.iϤ)i@i)ii)ii)i0iO)iei)ioio)ipi)ii/)ipi)iiN)__name__ __module__ __qualname____firstlineno____doc__ Hangul_Jamo CJK_RadicalsPhags_PaHangul_SyllablesCJK_Compatibility_IdeographsCJK_Compatibility_FormsKatakana_Hangul_Halfwidth#Ideographic_Symbols_And_PunctuationTangutKana_SupplementNushuSupplementary_Ideographic_Planeranges__static_attributes__r a/var/www/html/backend/Backoffice_Marketplace/venv/lib/python3.13/site-packages/sacremoses/util.pyr r s K*"LH&$2 -!/+'F'O E'# $! ' Fr!r cN[U5n[Hup#X:dM X:s $ g)u This checks for CJK character. >>> CJKChars().ranges [(4352, 4607), (11904, 42191), (43072, 43135), (44032, 55215), (63744, 64255), (65072, 65103), (65381, 65500), (94208, 101119), (110592, 110895), (110960, 111359), (131072, 196607)] >>> is_cjk('㏾') True >>> is_cjk('﹟') False :param character: The character that needs to be checked. :type character: char :return: bool F)ord_CJKChars_ranges) charactercharstartends r"is_cjkr*as+ y>D&  :< ' r!c "[USSSSSS.S9$)aO This function transforms the input text into an "escaped" version suitable for well-formed XML formatting. Note that the default xml.sax.saxutils.escape() function don't escape some characters that Moses does so we have to manually add them to the entities dictionary. >>> input_str = ''')| & < > ' " ] [''' >>> expected_output = ''')| & < > ' " ] [''' >>> escape(input_str) == expected_output True >>> xml_escape(input_str) ')| & < > ' " ] [' :param text: The text that needs to be escaped. :type text: str :rtype: str '"|[])'"|[]entities)rtexts r" xml_escaper:ws(&      r!c "[USSSSSS.S9$)a1 This function transforms the "escaped" version suitable for well-formed XML formatting into humanly-readable string. Note that the default xml.sax.saxutils.unescape() function don't unescape some characters that Moses does so we have to manually add them to the entities dictionary. >>> from xml.sax.saxutils import unescape >>> s = ')| & < > ' " ] [' >>> expected = ''')| & < > ' " ] [''' >>> xml_unescape(s) == expected True :param text: The text that needs to be unescaped. :type text: str :rtype: str r1r2r3r4r5)r,r-r.r/r0r6)rr8s r" xml_unescaper<s($      r!cJ[U5up[US5 [X5$)zd From https://docs.python.org/3/library/itertools.html#recipes s -> (s0,s1), (s1,s2), (s2, s3), ... N)rnextzip)iterableabs r"pairwiserCs" x=DADM q9r!Nc6[U5/U-n[USU06$)zaCollect data into fixed-length chunks or blocks from https://stackoverflow.com/a/16789869/610569 fillvalue)iterr)r@nrEargss r"grouperrIs% N a D  2 22r!c^U(a [U5OUnUS::a [TU5$[US9"U4SjU55$)N)n_jobsc3F># UHn[T5"U5v M g7fN)r).0linefuncs r" )parallelize_preprocess..s%OhdgdmD&9&9hs!)r mapr)rQiterator processes progress_bars` r"parallelize_preprocessrXs:!-tH~8HA~4"" 9 %%Oh%O OOr!rN)F) itertoolsrrxml.sax.saxutilsrrjoblibrrr objectr rr%r*r:r<rCrIrXr r!r"r]sQ'-$PvPf:$$,><3Pr!