ยปCore Development>Code coverage>Tools/scripts/fixdiv.py

Python code coverage for Tools/scripts/fixdiv.py

#countcontent
1n/a#! /usr/bin/env python3
2n/a
3n/a"""fixdiv - tool to fix division operators.
4n/a
5n/aTo use this tool, first run `python -Qwarnall yourscript.py 2>warnings'.
6n/aThis runs the script `yourscript.py' while writing warning messages
7n/aabout all uses of the classic division operator to the file
8n/a`warnings'. The warnings look like this:
9n/a
10n/a <file>:<line>: DeprecationWarning: classic <type> division
11n/a
12n/aThe warnings are written to stderr, so you must use `2>' for the I/O
13n/aredirect. I know of no way to redirect stderr on Windows in a DOS
14n/abox, so you will have to modify the script to set sys.stderr to some
15n/akind of log file if you want to do this on Windows.
16n/a
17n/aThe warnings are not limited to the script; modules imported by the
18n/ascript may also trigger warnings. In fact a useful technique is to
19n/awrite a test script specifically intended to exercise all code in a
20n/aparticular module or set of modules.
21n/a
22n/aThen run `python fixdiv.py warnings'. This first reads the warnings,
23n/alooking for classic division warnings, and sorts them by file name and
24n/aline number. Then, for each file that received at least one warning,
25n/ait parses the file and tries to match the warnings up to the division
26n/aoperators found in the source code. If it is successful, it writes
27n/aits findings to stdout, preceded by a line of dashes and a line of the
28n/aform:
29n/a
30n/a Index: <file>
31n/a
32n/aIf the only findings found are suggestions to change a / operator into
33n/aa // operator, the output is acceptable input for the Unix 'patch'
34n/aprogram.
35n/a
36n/aHere are the possible messages on stdout (N stands for a line number):
37n/a
38n/a- A plain-diff-style change ('NcN', a line marked by '<', a line
39n/a containing '---', and a line marked by '>'):
40n/a
41n/a A / operator was found that should be changed to //. This is the
42n/a recommendation when only int and/or long arguments were seen.
43n/a
44n/a- 'True division / operator at line N' and a line marked by '=':
45n/a
46n/a A / operator was found that can remain unchanged. This is the
47n/a recommendation when only float and/or complex arguments were seen.
48n/a
49n/a- 'Ambiguous / operator (..., ...) at line N', line marked by '?':
50n/a
51n/a A / operator was found for which int or long as well as float or
52n/a complex arguments were seen. This is highly unlikely; if it occurs,
53n/a you may have to restructure the code to keep the classic semantics,
54n/a or maybe you don't care about the classic semantics.
55n/a
56n/a- 'No conclusive evidence on line N', line marked by '*':
57n/a
58n/a A / operator was found for which no warnings were seen. This could
59n/a be code that was never executed, or code that was only executed
60n/a with user-defined objects as arguments. You will have to
61n/a investigate further. Note that // can be overloaded separately from
62n/a /, using __floordiv__. True division can also be separately
63n/a overloaded, using __truediv__. Classic division should be the same
64n/a as either of those. (XXX should I add a warning for division on
65n/a user-defined objects, to disambiguate this case from code that was
66n/a never executed?)
67n/a
68n/a- 'Phantom ... warnings for line N', line marked by '*':
69n/a
70n/a A warning was seen for a line not containing a / operator. The most
71n/a likely cause is a warning about code executed by 'exec' or eval()
72n/a (see note below), or an indirect invocation of the / operator, for
73n/a example via the div() function in the operator module. It could
74n/a also be caused by a change to the file between the time the test
75n/a script was run to collect warnings and the time fixdiv was run.
76n/a
77n/a- 'More than one / operator in line N'; or
78n/a 'More than one / operator per statement in lines N-N':
79n/a
80n/a The scanner found more than one / operator on a single line, or in a
81n/a statement split across multiple lines. Because the warnings
82n/a framework doesn't (and can't) show the offset within the line, and
83n/a the code generator doesn't always give the correct line number for
84n/a operations in a multi-line statement, we can't be sure whether all
85n/a operators in the statement were executed. To be on the safe side,
86n/a by default a warning is issued about this case. In practice, these
87n/a cases are usually safe, and the -m option suppresses these warning.
88n/a
89n/a- 'Can't find the / operator in line N', line marked by '*':
90n/a
91n/a This really shouldn't happen. It means that the tokenize module
92n/a reported a '/' operator but the line it returns didn't contain a '/'
93n/a character at the indicated position.
94n/a
95n/a- 'Bad warning for line N: XYZ', line marked by '*':
96n/a
97n/a This really shouldn't happen. It means that a 'classic XYZ
98n/a division' warning was read with XYZ being something other than
99n/a 'int', 'long', 'float', or 'complex'.
100n/a
101n/aNotes:
102n/a
103n/a- The augmented assignment operator /= is handled the same way as the
104n/a / operator.
105n/a
106n/a- This tool never looks at the // operator; no warnings are ever
107n/a generated for use of this operator.
108n/a
109n/a- This tool never looks at the / operator when a future division
110n/a statement is in effect; no warnings are generated in this case, and
111n/a because the tool only looks at files for which at least one classic
112n/a division warning was seen, it will never look at files containing a
113n/a future division statement.
114n/a
115n/a- Warnings may be issued for code not read from a file, but executed
116n/a using the exec() or eval() functions. These may have
117n/a <string> in the filename position, in which case the fixdiv script
118n/a will attempt and fail to open a file named '<string>' and issue a
119n/a warning about this failure; or these may be reported as 'Phantom'
120n/a warnings (see above). You're on your own to deal with these. You
121n/a could make all recommended changes and add a future division
122n/a statement to all affected files, and then re-run the test script; it
123n/a should not issue any warnings. If there are any, and you have a
124n/a hard time tracking down where they are generated, you can use the
125n/a -Werror option to force an error instead of a first warning,
126n/a generating a traceback.
127n/a
128n/a- The tool should be run from the same directory as that from which
129n/a the original script was run, otherwise it won't be able to open
130n/a files given by relative pathnames.
131n/a"""
132n/a
133n/aimport sys
134n/aimport getopt
135n/aimport re
136n/aimport tokenize
137n/a
138n/amulti_ok = 0
139n/a
140n/adef main():
141n/a try:
142n/a opts, args = getopt.getopt(sys.argv[1:], "hm")
143n/a except getopt.error as msg:
144n/a usage(msg)
145n/a return 2
146n/a for o, a in opts:
147n/a if o == "-h":
148n/a print(__doc__)
149n/a return
150n/a if o == "-m":
151n/a global multi_ok
152n/a multi_ok = 1
153n/a if not args:
154n/a usage("at least one file argument is required")
155n/a return 2
156n/a if args[1:]:
157n/a sys.stderr.write("%s: extra file arguments ignored\n", sys.argv[0])
158n/a warnings = readwarnings(args[0])
159n/a if warnings is None:
160n/a return 1
161n/a files = list(warnings.keys())
162n/a if not files:
163n/a print("No classic division warnings read from", args[0])
164n/a return
165n/a files.sort()
166n/a exit = None
167n/a for filename in files:
168n/a x = process(filename, warnings[filename])
169n/a exit = exit or x
170n/a return exit
171n/a
172n/adef usage(msg):
173n/a sys.stderr.write("%s: %s\n" % (sys.argv[0], msg))
174n/a sys.stderr.write("Usage: %s [-m] warnings\n" % sys.argv[0])
175n/a sys.stderr.write("Try `%s -h' for more information.\n" % sys.argv[0])
176n/a
177n/aPATTERN = (r"^(.+?):(\d+): DeprecationWarning: "
178n/a r"classic (int|long|float|complex) division$")
179n/a
180n/adef readwarnings(warningsfile):
181n/a prog = re.compile(PATTERN)
182n/a try:
183n/a f = open(warningsfile)
184n/a except IOError as msg:
185n/a sys.stderr.write("can't open: %s\n" % msg)
186n/a return
187n/a warnings = {}
188n/a while 1:
189n/a line = f.readline()
190n/a if not line:
191n/a break
192n/a m = prog.match(line)
193n/a if not m:
194n/a if line.find("division") >= 0:
195n/a sys.stderr.write("Warning: ignored input " + line)
196n/a continue
197n/a filename, lineno, what = m.groups()
198n/a list = warnings.get(filename)
199n/a if list is None:
200n/a warnings[filename] = list = []
201n/a list.append((int(lineno), sys.intern(what)))
202n/a f.close()
203n/a return warnings
204n/a
205n/adef process(filename, list):
206n/a print("-"*70)
207n/a assert list # if this fails, readwarnings() is broken
208n/a try:
209n/a fp = open(filename)
210n/a except IOError as msg:
211n/a sys.stderr.write("can't open: %s\n" % msg)
212n/a return 1
213n/a print("Index:", filename)
214n/a f = FileContext(fp)
215n/a list.sort()
216n/a index = 0 # list[:index] has been processed, list[index:] is still to do
217n/a g = tokenize.generate_tokens(f.readline)
218n/a while 1:
219n/a startlineno, endlineno, slashes = lineinfo = scanline(g)
220n/a if startlineno is None:
221n/a break
222n/a assert startlineno <= endlineno is not None
223n/a orphans = []
224n/a while index < len(list) and list[index][0] < startlineno:
225n/a orphans.append(list[index])
226n/a index += 1
227n/a if orphans:
228n/a reportphantomwarnings(orphans, f)
229n/a warnings = []
230n/a while index < len(list) and list[index][0] <= endlineno:
231n/a warnings.append(list[index])
232n/a index += 1
233n/a if not slashes and not warnings:
234n/a pass
235n/a elif slashes and not warnings:
236n/a report(slashes, "No conclusive evidence")
237n/a elif warnings and not slashes:
238n/a reportphantomwarnings(warnings, f)
239n/a else:
240n/a if len(slashes) > 1:
241n/a if not multi_ok:
242n/a rows = []
243n/a lastrow = None
244n/a for (row, col), line in slashes:
245n/a if row == lastrow:
246n/a continue
247n/a rows.append(row)
248n/a lastrow = row
249n/a assert rows
250n/a if len(rows) == 1:
251n/a print("*** More than one / operator in line", rows[0])
252n/a else:
253n/a print("*** More than one / operator per statement", end=' ')
254n/a print("in lines %d-%d" % (rows[0], rows[-1]))
255n/a intlong = []
256n/a floatcomplex = []
257n/a bad = []
258n/a for lineno, what in warnings:
259n/a if what in ("int", "long"):
260n/a intlong.append(what)
261n/a elif what in ("float", "complex"):
262n/a floatcomplex.append(what)
263n/a else:
264n/a bad.append(what)
265n/a lastrow = None
266n/a for (row, col), line in slashes:
267n/a if row == lastrow:
268n/a continue
269n/a lastrow = row
270n/a line = chop(line)
271n/a if line[col:col+1] != "/":
272n/a print("*** Can't find the / operator in line %d:" % row)
273n/a print("*", line)
274n/a continue
275n/a if bad:
276n/a print("*** Bad warning for line %d:" % row, bad)
277n/a print("*", line)
278n/a elif intlong and not floatcomplex:
279n/a print("%dc%d" % (row, row))
280n/a print("<", line)
281n/a print("---")
282n/a print(">", line[:col] + "/" + line[col:])
283n/a elif floatcomplex and not intlong:
284n/a print("True division / operator at line %d:" % row)
285n/a print("=", line)
286n/a elif intlong and floatcomplex:
287n/a print("*** Ambiguous / operator (%s, %s) at line %d:" % (
288n/a "|".join(intlong), "|".join(floatcomplex), row))
289n/a print("?", line)
290n/a fp.close()
291n/a
292n/adef reportphantomwarnings(warnings, f):
293n/a blocks = []
294n/a lastrow = None
295n/a lastblock = None
296n/a for row, what in warnings:
297n/a if row != lastrow:
298n/a lastblock = [row]
299n/a blocks.append(lastblock)
300n/a lastblock.append(what)
301n/a for block in blocks:
302n/a row = block[0]
303n/a whats = "/".join(block[1:])
304n/a print("*** Phantom %s warnings for line %d:" % (whats, row))
305n/a f.report(row, mark="*")
306n/a
307n/adef report(slashes, message):
308n/a lastrow = None
309n/a for (row, col), line in slashes:
310n/a if row != lastrow:
311n/a print("*** %s on line %d:" % (message, row))
312n/a print("*", chop(line))
313n/a lastrow = row
314n/a
315n/aclass FileContext:
316n/a def __init__(self, fp, window=5, lineno=1):
317n/a self.fp = fp
318n/a self.window = 5
319n/a self.lineno = 1
320n/a self.eoflookahead = 0
321n/a self.lookahead = []
322n/a self.buffer = []
323n/a def fill(self):
324n/a while len(self.lookahead) < self.window and not self.eoflookahead:
325n/a line = self.fp.readline()
326n/a if not line:
327n/a self.eoflookahead = 1
328n/a break
329n/a self.lookahead.append(line)
330n/a def readline(self):
331n/a self.fill()
332n/a if not self.lookahead:
333n/a return ""
334n/a line = self.lookahead.pop(0)
335n/a self.buffer.append(line)
336n/a self.lineno += 1
337n/a return line
338n/a def __getitem__(self, index):
339n/a self.fill()
340n/a bufstart = self.lineno - len(self.buffer)
341n/a lookend = self.lineno + len(self.lookahead)
342n/a if bufstart <= index < self.lineno:
343n/a return self.buffer[index - bufstart]
344n/a if self.lineno <= index < lookend:
345n/a return self.lookahead[index - self.lineno]
346n/a raise KeyError
347n/a def report(self, first, last=None, mark="*"):
348n/a if last is None:
349n/a last = first
350n/a for i in range(first, last+1):
351n/a try:
352n/a line = self[first]
353n/a except KeyError:
354n/a line = "<missing line>"
355n/a print(mark, chop(line))
356n/a
357n/adef scanline(g):
358n/a slashes = []
359n/a startlineno = None
360n/a endlineno = None
361n/a for type, token, start, end, line in g:
362n/a endlineno = end[0]
363n/a if startlineno is None:
364n/a startlineno = endlineno
365n/a if token in ("/", "/="):
366n/a slashes.append((start, line))
367n/a if type == tokenize.NEWLINE:
368n/a break
369n/a return startlineno, endlineno, slashes
370n/a
371n/adef chop(line):
372n/a if line.endswith("\n"):
373n/a return line[:-1]
374n/a else:
375n/a return line
376n/a
377n/aif __name__ == "__main__":
378n/a sys.exit(main())