Python code coverage for Tools/scripts/fixdiv.py

#	count	content
1	n/a	#! /usr/bin/env python3
2	n/a
3	n/a	"""fixdiv - tool to fix division operators.
4	n/a
5	n/a	To use this tool, first run `python -Qwarnall yourscript.py 2>warnings'.
6	n/a	This runs the script `yourscript.py' while writing warning messages
7	n/a	about all uses of the classic division operator to the file
8	n/a	`warnings'. The warnings look like this:
9	n/a
10	n/a	<file>:<line>: DeprecationWarning: classic <type> division
11	n/a
12	n/a	The warnings are written to stderr, so you must use `2>' for the I/O
13	n/a	redirect. I know of no way to redirect stderr on Windows in a DOS
14	n/a	box, so you will have to modify the script to set sys.stderr to some
15	n/a	kind of log file if you want to do this on Windows.
16	n/a
17	n/a	The warnings are not limited to the script; modules imported by the
18	n/a	script may also trigger warnings. In fact a useful technique is to
19	n/a	write a test script specifically intended to exercise all code in a
20	n/a	particular module or set of modules.
21	n/a
22	n/a	Then run `python fixdiv.py warnings'. This first reads the warnings,
23	n/a	looking for classic division warnings, and sorts them by file name and
24	n/a	line number. Then, for each file that received at least one warning,
25	n/a	it parses the file and tries to match the warnings up to the division
26	n/a	operators found in the source code. If it is successful, it writes
27	n/a	its findings to stdout, preceded by a line of dashes and a line of the
28	n/a	form:
29	n/a
30	n/a	Index: <file>
31	n/a
32	n/a	If the only findings found are suggestions to change a / operator into
33	n/a	a // operator, the output is acceptable input for the Unix 'patch'
34	n/a	program.
35	n/a
36	n/a	Here are the possible messages on stdout (N stands for a line number):
37	n/a
38	n/a	- A plain-diff-style change ('NcN', a line marked by '<', a line
39	n/a	containing '---', and a line marked by '>'):
40	n/a
41	n/a	A / operator was found that should be changed to //. This is the
42	n/a	recommendation when only int and/or long arguments were seen.
43	n/a
44	n/a	- 'True division / operator at line N' and a line marked by '=':
45	n/a
46	n/a	A / operator was found that can remain unchanged. This is the
47	n/a	recommendation when only float and/or complex arguments were seen.
48	n/a
49	n/a	- 'Ambiguous / operator (..., ...) at line N', line marked by '?':
50	n/a
51	n/a	A / operator was found for which int or long as well as float or
52	n/a	complex arguments were seen. This is highly unlikely; if it occurs,
53	n/a	you may have to restructure the code to keep the classic semantics,
54	n/a	or maybe you don't care about the classic semantics.
55	n/a
56	n/a	- 'No conclusive evidence on line N', line marked by '*':
57	n/a
58	n/a	A / operator was found for which no warnings were seen. This could
59	n/a	be code that was never executed, or code that was only executed
60	n/a	with user-defined objects as arguments. You will have to
61	n/a	investigate further. Note that // can be overloaded separately from
62	n/a	/, using __floordiv__. True division can also be separately
63	n/a	overloaded, using __truediv__. Classic division should be the same
64	n/a	as either of those. (XXX should I add a warning for division on
65	n/a	user-defined objects, to disambiguate this case from code that was
66	n/a	never executed?)
67	n/a
68	n/a	- 'Phantom ... warnings for line N', line marked by '*':
69	n/a
70	n/a	A warning was seen for a line not containing a / operator. The most
71	n/a	likely cause is a warning about code executed by 'exec' or eval()
72	n/a	(see note below), or an indirect invocation of the / operator, for
73	n/a	example via the div() function in the operator module. It could
74	n/a	also be caused by a change to the file between the time the test
75	n/a	script was run to collect warnings and the time fixdiv was run.
76	n/a
77	n/a	- 'More than one / operator in line N'; or
78	n/a	'More than one / operator per statement in lines N-N':
79	n/a
80	n/a	The scanner found more than one / operator on a single line, or in a
81	n/a	statement split across multiple lines. Because the warnings
82	n/a	framework doesn't (and can't) show the offset within the line, and
83	n/a	the code generator doesn't always give the correct line number for
84	n/a	operations in a multi-line statement, we can't be sure whether all
85	n/a	operators in the statement were executed. To be on the safe side,
86	n/a	by default a warning is issued about this case. In practice, these
87	n/a	cases are usually safe, and the -m option suppresses these warning.
88	n/a
89	n/a	- 'Can't find the / operator in line N', line marked by '*':
90	n/a
91	n/a	This really shouldn't happen. It means that the tokenize module
92	n/a	reported a '/' operator but the line it returns didn't contain a '/'
93	n/a	character at the indicated position.
94	n/a
95	n/a	- 'Bad warning for line N: XYZ', line marked by '*':
96	n/a
97	n/a	This really shouldn't happen. It means that a 'classic XYZ
98	n/a	division' warning was read with XYZ being something other than
99	n/a	'int', 'long', 'float', or 'complex'.
100	n/a
101	n/a	Notes:
102	n/a
103	n/a	- The augmented assignment operator /= is handled the same way as the
104	n/a	/ operator.
105	n/a
106	n/a	- This tool never looks at the // operator; no warnings are ever
107	n/a	generated for use of this operator.
108	n/a
109	n/a	- This tool never looks at the / operator when a future division
110	n/a	statement is in effect; no warnings are generated in this case, and
111	n/a	because the tool only looks at files for which at least one classic
112	n/a	division warning was seen, it will never look at files containing a
113	n/a	future division statement.
114	n/a
115	n/a	- Warnings may be issued for code not read from a file, but executed
116	n/a	using the exec() or eval() functions. These may have
117	n/a	<string> in the filename position, in which case the fixdiv script
118	n/a	will attempt and fail to open a file named '<string>' and issue a
119	n/a	warning about this failure; or these may be reported as 'Phantom'
120	n/a	warnings (see above). You're on your own to deal with these. You
121	n/a	could make all recommended changes and add a future division
122	n/a	statement to all affected files, and then re-run the test script; it
123	n/a	should not issue any warnings. If there are any, and you have a
124	n/a	hard time tracking down where they are generated, you can use the
125	n/a	-Werror option to force an error instead of a first warning,
126	n/a	generating a traceback.
127	n/a
128	n/a	- The tool should be run from the same directory as that from which
129	n/a	the original script was run, otherwise it won't be able to open
130	n/a	files given by relative pathnames.
131	n/a	"""
132	n/a
133	n/a	import sys
134	n/a	import getopt
135	n/a	import re
136	n/a	import tokenize
137	n/a
138	n/a	multi_ok = 0
139	n/a
140	n/a	def main():
141	n/a	try:
142	n/a	opts, args = getopt.getopt(sys.argv[1:], "hm")
143	n/a	except getopt.error as msg:
144	n/a	usage(msg)
145	n/a	return 2
146	n/a	for o, a in opts:
147	n/a	if o == "-h":
148	n/a	print(__doc__)
149	n/a	return
150	n/a	if o == "-m":
151	n/a	global multi_ok
152	n/a	multi_ok = 1
153	n/a	if not args:
154	n/a	usage("at least one file argument is required")
155	n/a	return 2
156	n/a	if args[1:]:
157	n/a	sys.stderr.write("%s: extra file arguments ignored\n", sys.argv[0])
158	n/a	warnings = readwarnings(args[0])
159	n/a	if warnings is None:
160	n/a	return 1
161	n/a	files = list(warnings.keys())
162	n/a	if not files:
163	n/a	print("No classic division warnings read from", args[0])
164	n/a	return
165	n/a	files.sort()
166	n/a	exit = None
167	n/a	for filename in files:
168	n/a	x = process(filename, warnings[filename])
169	n/a	exit = exit or x
170	n/a	return exit
171	n/a
172	n/a	def usage(msg):
173	n/a	sys.stderr.write("%s: %s\n" % (sys.argv[0], msg))
174	n/a	sys.stderr.write("Usage: %s [-m] warnings\n" % sys.argv[0])
175	n/a	sys.stderr.write("Try `%s -h' for more information.\n" % sys.argv[0])
176	n/a
177	n/a	PATTERN = (r"^(.+?):(\d+): DeprecationWarning: "
178	n/a	r"classic (int\|long\|float\|complex) division$")
179	n/a
180	n/a	def readwarnings(warningsfile):
181	n/a	prog = re.compile(PATTERN)
182	n/a	try:
183	n/a	f = open(warningsfile)
184	n/a	except IOError as msg:
185	n/a	sys.stderr.write("can't open: %s\n" % msg)
186	n/a	return
187	n/a	warnings = {}
188	n/a	while 1:
189	n/a	line = f.readline()
190	n/a	if not line:
191	n/a	break
192	n/a	m = prog.match(line)
193	n/a	if not m:
194	n/a	if line.find("division") >= 0:
195	n/a	sys.stderr.write("Warning: ignored input " + line)
196	n/a	continue
197	n/a	filename, lineno, what = m.groups()
198	n/a	list = warnings.get(filename)
199	n/a	if list is None:
200	n/a	warnings[filename] = list = []
201	n/a	list.append((int(lineno), sys.intern(what)))
202	n/a	f.close()
203	n/a	return warnings
204	n/a
205	n/a	def process(filename, list):
206	n/a	print("-"*70)
207	n/a	assert list # if this fails, readwarnings() is broken
208	n/a	try:
209	n/a	fp = open(filename)
210	n/a	except IOError as msg:
211	n/a	sys.stderr.write("can't open: %s\n" % msg)
212	n/a	return 1
213	n/a	print("Index:", filename)
214	n/a	f = FileContext(fp)
215	n/a	list.sort()
216	n/a	index = 0 # list[:index] has been processed, list[index:] is still to do
217	n/a	g = tokenize.generate_tokens(f.readline)
218	n/a	while 1:
219	n/a	startlineno, endlineno, slashes = lineinfo = scanline(g)
220	n/a	if startlineno is None:
221	n/a	break
222	n/a	assert startlineno <= endlineno is not None
223	n/a	orphans = []
224	n/a	while index < len(list) and list[index][0] < startlineno:
225	n/a	orphans.append(list[index])
226	n/a	index += 1
227	n/a	if orphans:
228	n/a	reportphantomwarnings(orphans, f)
229	n/a	warnings = []
230	n/a	while index < len(list) and list[index][0] <= endlineno:
231	n/a	warnings.append(list[index])
232	n/a	index += 1
233	n/a	if not slashes and not warnings:
234	n/a	pass
235	n/a	elif slashes and not warnings:
236	n/a	report(slashes, "No conclusive evidence")
237	n/a	elif warnings and not slashes:
238	n/a	reportphantomwarnings(warnings, f)
239	n/a	else:
240	n/a	if len(slashes) > 1:
241	n/a	if not multi_ok:
242	n/a	rows = []
243	n/a	lastrow = None
244	n/a	for (row, col), line in slashes:
245	n/a	if row == lastrow:
246	n/a	continue
247	n/a	rows.append(row)
248	n/a	lastrow = row
249	n/a	assert rows
250	n/a	if len(rows) == 1:
251	n/a	print("*** More than one / operator in line", rows[0])
252	n/a	else:
253	n/a	print("*** More than one / operator per statement", end=' ')
254	n/a	print("in lines %d-%d" % (rows[0], rows[-1]))
255	n/a	intlong = []
256	n/a	floatcomplex = []
257	n/a	bad = []
258	n/a	for lineno, what in warnings:
259	n/a	if what in ("int", "long"):
260	n/a	intlong.append(what)
261	n/a	elif what in ("float", "complex"):
262	n/a	floatcomplex.append(what)
263	n/a	else:
264	n/a	bad.append(what)
265	n/a	lastrow = None
266	n/a	for (row, col), line in slashes:
267	n/a	if row == lastrow:
268	n/a	continue
269	n/a	lastrow = row
270	n/a	line = chop(line)
271	n/a	if line[col:col+1] != "/":
272	n/a	print("*** Can't find the / operator in line %d:" % row)
273	n/a	print("*", line)
274	n/a	continue
275	n/a	if bad:
276	n/a	print("*** Bad warning for line %d:" % row, bad)
277	n/a	print("*", line)
278	n/a	elif intlong and not floatcomplex:
279	n/a	print("%dc%d" % (row, row))
280	n/a	print("<", line)
281	n/a	print("---")
282	n/a	print(">", line[:col] + "/" + line[col:])
283	n/a	elif floatcomplex and not intlong:
284	n/a	print("True division / operator at line %d:" % row)
285	n/a	print("=", line)
286	n/a	elif intlong and floatcomplex:
287	n/a	print("*** Ambiguous / operator (%s, %s) at line %d:" % (
288	n/a	"\|".join(intlong), "\|".join(floatcomplex), row))
289	n/a	print("?", line)
290	n/a	fp.close()
291	n/a
292	n/a	def reportphantomwarnings(warnings, f):
293	n/a	blocks = []
294	n/a	lastrow = None
295	n/a	lastblock = None
296	n/a	for row, what in warnings:
297	n/a	if row != lastrow:
298	n/a	lastblock = [row]
299	n/a	blocks.append(lastblock)
300	n/a	lastblock.append(what)
301	n/a	for block in blocks:
302	n/a	row = block[0]
303	n/a	whats = "/".join(block[1:])
304	n/a	print("*** Phantom %s warnings for line %d:" % (whats, row))
305	n/a	f.report(row, mark="*")
306	n/a
307	n/a	def report(slashes, message):
308	n/a	lastrow = None
309	n/a	for (row, col), line in slashes:
310	n/a	if row != lastrow:
311	n/a	print("*** %s on line %d:" % (message, row))
312	n/a	print("*", chop(line))
313	n/a	lastrow = row
314	n/a
315	n/a	class FileContext:
316	n/a	def __init__(self, fp, window=5, lineno=1):
317	n/a	self.fp = fp
318	n/a	self.window = 5
319	n/a	self.lineno = 1
320	n/a	self.eoflookahead = 0
321	n/a	self.lookahead = []
322	n/a	self.buffer = []
323	n/a	def fill(self):
324	n/a	while len(self.lookahead) < self.window and not self.eoflookahead:
325	n/a	line = self.fp.readline()
326	n/a	if not line:
327	n/a	self.eoflookahead = 1
328	n/a	break
329	n/a	self.lookahead.append(line)
330	n/a	def readline(self):
331	n/a	self.fill()
332	n/a	if not self.lookahead:
333	n/a	return ""
334	n/a	line = self.lookahead.pop(0)
335	n/a	self.buffer.append(line)
336	n/a	self.lineno += 1
337	n/a	return line
338	n/a	def __getitem__(self, index):
339	n/a	self.fill()
340	n/a	bufstart = self.lineno - len(self.buffer)
341	n/a	lookend = self.lineno + len(self.lookahead)
342	n/a	if bufstart <= index < self.lineno:
343	n/a	return self.buffer[index - bufstart]
344	n/a	if self.lineno <= index < lookend:
345	n/a	return self.lookahead[index - self.lineno]
346	n/a	raise KeyError
347	n/a	def report(self, first, last=None, mark="*"):
348	n/a	if last is None:
349	n/a	last = first
350	n/a	for i in range(first, last+1):
351	n/a	try:
352	n/a	line = self[first]
353	n/a	except KeyError:
354	n/a	line = "<missing line>"
355	n/a	print(mark, chop(line))
356	n/a
357	n/a	def scanline(g):
358	n/a	slashes = []
359	n/a	startlineno = None
360	n/a	endlineno = None
361	n/a	for type, token, start, end, line in g:
362	n/a	endlineno = end[0]
363	n/a	if startlineno is None:
364	n/a	startlineno = endlineno
365	n/a	if token in ("/", "/="):
366	n/a	slashes.append((start, line))
367	n/a	if type == tokenize.NEWLINE:
368	n/a	break
369	n/a	return startlineno, endlineno, slashes
370	n/a
371	n/a	def chop(line):
372	n/a	if line.endswith("\n"):
373	n/a	return line[:-1]
374	n/a	else:
375	n/a	return line
376	n/a
377	n/a	if __name__ == "__main__":
378	n/a	sys.exit(main())