Python code coverage for Lib/re.py

#	count	content
1	n/a	#
2	n/a	# Secret Labs' Regular Expression Engine
3	n/a	#
4	n/a	# re-compatible interface for the sre matching engine
5	n/a	#
6	n/a	# Copyright (c) 1998-2001 by Secret Labs AB. All rights reserved.
7	n/a	#
8	n/a	# This version of the SRE library can be redistributed under CNRI's
9	n/a	# Python 1.6 license. For any other use, please contact Secret Labs
10	n/a	# AB (info@pythonware.com).
11	n/a	#
12	n/a	# Portions of this engine have been developed in cooperation with
13	n/a	# CNRI. Hewlett-Packard provided funding for 1.6 integration and
14	n/a	# other compatibility work.
15	n/a	#
16	n/a
17	n/a	r"""Support for regular expressions (RE).
18	n/a
19	n/a	This module provides regular expression matching operations similar to
20	n/a	those found in Perl. It supports both 8-bit and Unicode strings; both
21	n/a	the pattern and the strings being processed can contain null bytes and
22	n/a	characters outside the US ASCII range.
23	n/a
24	n/a	Regular expressions can contain both special and ordinary characters.
25	n/a	Most ordinary characters, like "A", "a", or "0", are the simplest
26	n/a	regular expressions; they simply match themselves. You can
27	n/a	concatenate ordinary characters, so last matches the string 'last'.
28	n/a
29	n/a	The special characters are:
30	n/a	"." Matches any character except a newline.
31	n/a	"^" Matches the start of the string.
32	n/a	"$" Matches the end of the string or just before the newline at
33	n/a	the end of the string.
34	n/a	"*" Matches 0 or more (greedy) repetitions of the preceding RE.
35	n/a	Greedy means that it will match as many repetitions as possible.
36	n/a	"+" Matches 1 or more (greedy) repetitions of the preceding RE.
37	n/a	"?" Matches 0 or 1 (greedy) of the preceding RE.
38	n/a	*?,+?,?? Non-greedy versions of the previous three special characters.
39	n/a	{m,n} Matches from m to n repetitions of the preceding RE.
40	n/a	{m,n}? Non-greedy version of the above.
41	n/a	"\\" Either escapes special characters or signals a special sequence.
42	n/a	[] Indicates a set of characters.
43	n/a	A "^" as the first character indicates a complementing set.
44	n/a	"\|" A\|B, creates an RE that will match either A or B.
45	n/a	(...) Matches the RE inside the parentheses.
46	n/a	The contents can be retrieved or matched later in the string.
47	n/a	(?aiLmsux) Set the A, I, L, M, S, U, or X flag for the RE (see below).
48	n/a	(?:...) Non-grouping version of regular parentheses.
49	n/a	(?P<name>...) The substring matched by the group is accessible by name.
50	n/a	(?P=name) Matches the text matched earlier by the group named name.
51	n/a	(?#...) A comment; ignored.
52	n/a	(?=...) Matches if ... matches next, but doesn't consume the string.
53	n/a	(?!...) Matches if ... doesn't match next.
54	n/a	(?<=...) Matches if preceded by ... (must be fixed length).
55	n/a	(?<!...) Matches if not preceded by ... (must be fixed length).
56	n/a	(?(id/name)yes\|no) Matches yes pattern if the group with id/name matched,
57	n/a	the (optional) no pattern otherwise.
58	n/a
59	n/a	The special sequences consist of "\\" and a character from the list
60	n/a	below. If the ordinary character is not on the list, then the
61	n/a	resulting RE will match the second character.
62	n/a	\number Matches the contents of the group of the same number.
63	n/a	\A Matches only at the start of the string.
64	n/a	\Z Matches only at the end of the string.
65	n/a	\b Matches the empty string, but only at the start or end of a word.
66	n/a	\B Matches the empty string, but not at the start or end of a word.
67	n/a	\d Matches any decimal digit; equivalent to the set [0-9] in
68	n/a	bytes patterns or string patterns with the ASCII flag.
69	n/a	In string patterns without the ASCII flag, it will match the whole
70	n/a	range of Unicode digits.
71	n/a	\D Matches any non-digit character; equivalent to [^\d].
72	n/a	\s Matches any whitespace character; equivalent to [ \t\n\r\f\v] in
73	n/a	bytes patterns or string patterns with the ASCII flag.
74	n/a	In string patterns without the ASCII flag, it will match the whole
75	n/a	range of Unicode whitespace characters.
76	n/a	\S Matches any non-whitespace character; equivalent to [^\s].
77	n/a	\w Matches any alphanumeric character; equivalent to [a-zA-Z0-9_]
78	n/a	in bytes patterns or string patterns with the ASCII flag.
79	n/a	In string patterns without the ASCII flag, it will match the
80	n/a	range of Unicode alphanumeric characters (letters plus digits
81	n/a	plus underscore).
82	n/a	With LOCALE, it will match the set [0-9_] plus characters defined
83	n/a	as letters for the current locale.
84	n/a	\W Matches the complement of \w.
85	n/a	\\ Matches a literal backslash.
86	n/a
87	n/a	This module exports the following functions:
88	n/a	match Match a regular expression pattern to the beginning of a string.
89	n/a	fullmatch Match a regular expression pattern to all of a string.
90	n/a	search Search a string for the presence of a pattern.
91	n/a	sub Substitute occurrences of a pattern found in a string.
92	n/a	subn Same as sub, but also return the number of substitutions made.
93	n/a	split Split a string by the occurrences of a pattern.
94	n/a	findall Find all occurrences of a pattern in a string.
95	n/a	finditer Return an iterator yielding a match object for each match.
96	n/a	compile Compile a pattern into a RegexObject.
97	n/a	purge Clear the regular expression cache.
98	n/a	escape Backslash all non-alphanumerics in a string.
99	n/a
100	n/a	Some of the functions in this module takes flags as optional parameters:
101	n/a	A ASCII For string patterns, make \w, \W, \b, \B, \d, \D
102	n/a	match the corresponding ASCII character categories
103	n/a	(rather than the whole Unicode categories, which is the
104	n/a	default).
105	n/a	For bytes patterns, this flag is the only available
106	n/a	behaviour and needn't be specified.
107	n/a	I IGNORECASE Perform case-insensitive matching.
108	n/a	L LOCALE Make \w, \W, \b, \B, dependent on the current locale.
109	n/a	M MULTILINE "^" matches the beginning of lines (after a newline)
110	n/a	as well as the string.
111	n/a	"$" matches the end of lines (before a newline) as well
112	n/a	as the end of the string.
113	n/a	S DOTALL "." matches any character at all, including the newline.
114	n/a	X VERBOSE Ignore whitespace and comments for nicer looking RE's.
115	n/a	U UNICODE For compatibility only. Ignored for string patterns (it
116	n/a	is the default), and forbidden for bytes patterns.
117	n/a
118	n/a	This module also defines an exception 'error'.
119	n/a
120	n/a	"""
121	n/a
122	n/a	import enum
123	n/a	import sre_compile
124	n/a	import sre_parse
125	n/a	import functools
126	n/a	try:
127	n/a	import _locale
128	n/a	except ImportError:
129	n/a	_locale = None
130	n/a
131	n/a	# public symbols
132	n/a	__all__ = [
133	n/a	"match", "fullmatch", "search", "sub", "subn", "split",
134	n/a	"findall", "finditer", "compile", "purge", "template", "escape",
135	n/a	"error", "A", "I", "L", "M", "S", "X", "U",
136	n/a	"ASCII", "IGNORECASE", "LOCALE", "MULTILINE", "DOTALL", "VERBOSE",
137	n/a	"UNICODE",
138	n/a	]
139	n/a
140	n/a	__version__ = "2.2.1"
141	n/a
142	n/a	class RegexFlag(enum.IntFlag):
143	n/a	ASCII = sre_compile.SRE_FLAG_ASCII # assume ascii "locale"
144	n/a	IGNORECASE = sre_compile.SRE_FLAG_IGNORECASE # ignore case
145	n/a	LOCALE = sre_compile.SRE_FLAG_LOCALE # assume current 8-bit locale
146	n/a	UNICODE = sre_compile.SRE_FLAG_UNICODE # assume unicode "locale"
147	n/a	MULTILINE = sre_compile.SRE_FLAG_MULTILINE # make anchors look for newline
148	n/a	DOTALL = sre_compile.SRE_FLAG_DOTALL # make dot match newline
149	n/a	VERBOSE = sre_compile.SRE_FLAG_VERBOSE # ignore whitespace and comments
150	n/a	A = ASCII
151	n/a	I = IGNORECASE
152	n/a	L = LOCALE
153	n/a	U = UNICODE
154	n/a	M = MULTILINE
155	n/a	S = DOTALL
156	n/a	X = VERBOSE
157	n/a	# sre extensions (experimental, don't rely on these)
158	n/a	TEMPLATE = sre_compile.SRE_FLAG_TEMPLATE # disable backtracking
159	n/a	T = TEMPLATE
160	n/a	DEBUG = sre_compile.SRE_FLAG_DEBUG # dump pattern after compilation
161	n/a	globals().update(RegexFlag.__members__)
162	n/a
163	n/a	# sre exception
164	n/a	error = sre_compile.error
165	n/a
166	n/a	# --------------------------------------------------------------------
167	n/a	# public interface
168	n/a
169	n/a	def match(pattern, string, flags=0):
170	n/a	"""Try to apply the pattern at the start of the string, returning
171	n/a	a match object, or None if no match was found."""
172	n/a	return _compile(pattern, flags).match(string)
173	n/a
174	n/a	def fullmatch(pattern, string, flags=0):
175	n/a	"""Try to apply the pattern to all of the string, returning
176	n/a	a match object, or None if no match was found."""
177	n/a	return _compile(pattern, flags).fullmatch(string)
178	n/a
179	n/a	def search(pattern, string, flags=0):
180	n/a	"""Scan through string looking for a match to the pattern, returning
181	n/a	a match object, or None if no match was found."""
182	n/a	return _compile(pattern, flags).search(string)
183	n/a
184	n/a	def sub(pattern, repl, string, count=0, flags=0):
185	n/a	"""Return the string obtained by replacing the leftmost
186	n/a	non-overlapping occurrences of the pattern in string by the
187	n/a	replacement repl. repl can be either a string or a callable;
188	n/a	if a string, backslash escapes in it are processed. If it is
189	n/a	a callable, it's passed the match object and must return
190	n/a	a replacement string to be used."""
191	n/a	return _compile(pattern, flags).sub(repl, string, count)
192	n/a
193	n/a	def subn(pattern, repl, string, count=0, flags=0):
194	n/a	"""Return a 2-tuple containing (new_string, number).
195	n/a	new_string is the string obtained by replacing the leftmost
196	n/a	non-overlapping occurrences of the pattern in the source
197	n/a	string by the replacement repl. number is the number of
198	n/a	substitutions that were made. repl can be either a string or a
199	n/a	callable; if a string, backslash escapes in it are processed.
200	n/a	If it is a callable, it's passed the match object and must
201	n/a	return a replacement string to be used."""
202	n/a	return _compile(pattern, flags).subn(repl, string, count)
203	n/a
204	n/a	def split(pattern, string, maxsplit=0, flags=0):
205	n/a	"""Split the source string by the occurrences of the pattern,
206	n/a	returning a list containing the resulting substrings. If
207	n/a	capturing parentheses are used in pattern, then the text of all
208	n/a	groups in the pattern are also returned as part of the resulting
209	n/a	list. If maxsplit is nonzero, at most maxsplit splits occur,
210	n/a	and the remainder of the string is returned as the final element
211	n/a	of the list."""
212	n/a	return _compile(pattern, flags).split(string, maxsplit)
213	n/a
214	n/a	def findall(pattern, string, flags=0):
215	n/a	"""Return a list of all non-overlapping matches in the string.
216	n/a
217	n/a	If one or more capturing groups are present in the pattern, return
218	n/a	a list of groups; this will be a list of tuples if the pattern
219	n/a	has more than one group.
220	n/a
221	n/a	Empty matches are included in the result."""
222	n/a	return _compile(pattern, flags).findall(string)
223	n/a
224	n/a	def finditer(pattern, string, flags=0):
225	n/a	"""Return an iterator over all non-overlapping matches in the
226	n/a	string. For each match, the iterator returns a match object.
227	n/a
228	n/a	Empty matches are included in the result."""
229	n/a	return _compile(pattern, flags).finditer(string)
230	n/a
231	n/a	def compile(pattern, flags=0):
232	n/a	"Compile a regular expression pattern, returning a pattern object."
233	n/a	return _compile(pattern, flags)
234	n/a
235	n/a	def purge():
236	n/a	"Clear the regular expression caches"
237	n/a	_cache.clear()
238	n/a	_compile_repl.cache_clear()
239	n/a
240	n/a	def template(pattern, flags=0):
241	n/a	"Compile a template pattern, returning a pattern object"
242	n/a	return _compile(pattern, flags\|T)
243	n/a
244	n/a	_alphanum_str = frozenset(
245	n/a	"_abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890")
246	n/a	_alphanum_bytes = frozenset(
247	n/a	b"_abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890")
248	n/a
249	n/a	def escape(pattern):
250	n/a	"""
251	n/a	Escape all the characters in pattern except ASCII letters, numbers and '_'.
252	n/a	"""
253	n/a	if isinstance(pattern, str):
254	n/a	alphanum = _alphanum_str
255	n/a	s = list(pattern)
256	n/a	for i, c in enumerate(pattern):
257	n/a	if c not in alphanum:
258	n/a	if c == "\000":
259	n/a	s[i] = "\\000"
260	n/a	else:
261	n/a	s[i] = "\\" + c
262	n/a	return "".join(s)
263	n/a	else:
264	n/a	alphanum = _alphanum_bytes
265	n/a	s = []
266	n/a	esc = ord(b"\\")
267	n/a	for c in pattern:
268	n/a	if c in alphanum:
269	n/a	s.append(c)
270	n/a	else:
271	n/a	if c == 0:
272	n/a	s.extend(b"\\000")
273	n/a	else:
274	n/a	s.append(esc)
275	n/a	s.append(c)
276	n/a	return bytes(s)
277	n/a
278	n/a	# --------------------------------------------------------------------
279	n/a	# internals
280	n/a
281	n/a	_cache = {}
282	n/a
283	n/a	_pattern_type = type(sre_compile.compile("", 0))
284	n/a
285	n/a	_MAXCACHE = 512
286	n/a	def _compile(pattern, flags):
287	n/a	# internal: compile pattern
288	n/a	try:
289	n/a	p, loc = _cache[type(pattern), pattern, flags]
290	n/a	if loc is None or loc == _locale.setlocale(_locale.LC_CTYPE):
291	n/a	return p
292	n/a	except KeyError:
293	n/a	pass
294	n/a	if isinstance(pattern, _pattern_type):
295	n/a	if flags:
296	n/a	raise ValueError(
297	n/a	"cannot process flags argument with a compiled pattern")
298	n/a	return pattern
299	n/a	if not sre_compile.isstring(pattern):
300	n/a	raise TypeError("first argument must be string or compiled pattern")
301	n/a	p = sre_compile.compile(pattern, flags)
302	n/a	if not (flags & DEBUG):
303	n/a	if len(_cache) >= _MAXCACHE:
304	n/a	_cache.clear()
305	n/a	if p.flags & LOCALE:
306	n/a	if not _locale:
307	n/a	return p
308	n/a	loc = _locale.setlocale(_locale.LC_CTYPE)
309	n/a	else:
310	n/a	loc = None
311	n/a	_cache[type(pattern), pattern, flags] = p, loc
312	n/a	return p
313	n/a
314	n/a	@functools.lru_cache(_MAXCACHE)
315	n/a	def _compile_repl(repl, pattern):
316	n/a	# internal: compile replacement pattern
317	n/a	return sre_parse.parse_template(repl, pattern)
318	n/a
319	n/a	def _expand(pattern, match, template):
320	n/a	# internal: match.expand implementation hook
321	n/a	template = sre_parse.parse_template(template, pattern)
322	n/a	return sre_parse.expand_template(template, match)
323	n/a
324	n/a	def _subx(pattern, template):
325	n/a	# internal: pattern.sub/subn implementation helper
326	n/a	template = _compile_repl(template, pattern)
327	n/a	if not template[0] and len(template[1]) == 1:
328	n/a	# literal replacement
329	n/a	return template[1][0]
330	n/a	def filter(match, template=template):
331	n/a	return sre_parse.expand_template(template, match)
332	n/a	return filter
333	n/a
334	n/a	# register myself for pickling
335	n/a
336	n/a	import copyreg
337	n/a
338	n/a	def _pickle(p):
339	n/a	return _compile, (p.pattern, p.flags)
340	n/a
341	n/a	copyreg.pickle(_pattern_type, _pickle, _compile)
342	n/a
343	n/a	# --------------------------------------------------------------------
344	n/a	# experimental stuff (see python-dev discussions for details)
345	n/a
346	n/a	class Scanner:
347	n/a	def __init__(self, lexicon, flags=0):
348	n/a	from sre_constants import BRANCH, SUBPATTERN
349	n/a	self.lexicon = lexicon
350	n/a	# combine phrases into a compound pattern
351	n/a	p = []
352	n/a	s = sre_parse.Pattern()
353	n/a	s.flags = flags
354	n/a	for phrase, action in lexicon:
355	n/a	gid = s.opengroup()
356	n/a	p.append(sre_parse.SubPattern(s, [
357	n/a	(SUBPATTERN, (gid, 0, 0, sre_parse.parse(phrase, flags))),
358	n/a	]))
359	n/a	s.closegroup(gid, p[-1])
360	n/a	p = sre_parse.SubPattern(s, [(BRANCH, (None, p))])
361	n/a	self.scanner = sre_compile.compile(p)
362	n/a	def scan(self, string):
363	n/a	result = []
364	n/a	append = result.append
365	n/a	match = self.scanner.scanner(string).match
366	n/a	i = 0
367	n/a	while True:
368	n/a	m = match()
369	n/a	if not m:
370	n/a	break
371	n/a	j = m.end()
372	n/a	if i == j:
373	n/a	break
374	n/a	action = self.lexicon[m.lastindex-1][1]
375	n/a	if callable(action):
376	n/a	self.match = m
377	n/a	action = action(self, m.group())
378	n/a	if action is not None:
379	n/a	append(action)
380	n/a	i = j
381	n/a	return result, string[i:]