How can you profile a Python script?

How can you profile a Python script?

Project Euler and other coding contests often have a maximum time to run or people boast of how fast their particular solution runs. With python, sometimes the approaches are somewhat kludgey - i.e., adding timing code to __main__.

What is a good way to profile how long a python program takes to run?

Answer by Chris Lawlor for How can you profile a Python script?

Python includes a profiler called cProfile. It not only gives the total running time, but also times each function separately, and tells you how many times each function was called, making it easy to determine where you should make optimizations.

You can call it from within your code, or from the interpreter, like this:

import cProfile  cProfile.run('foo()')  

Even more usefully, you can invoke the cProfile when running a script:

python -m cProfile myscript.py  

To make it even easier, I made a little batch file called 'profile.bat':

python -m cProfile %1  

So all I have to do is run:

profile euler048.py  

And I get this:

1007 function calls in 0.061 CPU seconds    Ordered by: standard name  ncalls  tottime  percall  cumtime  percall filename:lineno(function)      1    0.000    0.000    0.061    0.061 :1()   1000    0.051    0.000    0.051    0.000 euler048.py:2()      1    0.005    0.005    0.061    0.061 euler048.py:2()      1    0.000    0.000    0.061    0.061 {execfile}      1    0.002    0.002    0.053    0.053 {map}      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler objects}      1    0.000    0.000    0.000    0.000 {range}      1    0.003    0.003    0.003    0.003 {sum}  

EDIT: Updated link to a good video resource from PyCon 2013: http://lanyrd.com/2013/pycon/scdywg/

Answer by Walter for How can you profile a Python script?

In Virtaal's source there's a very useful class and decorator that can make it profiling (even for specific methods/functions) very easy. The output can then be viewed very comfortably in KCacheGrind.

Answer by Joe Shaw for How can you profile a Python script?

It's worth pointing out that using the profiler only works (by default) on the main thread, and you won't get any information from other threads if you use them. This can be a bit of a gotcha as it is completely unmentioned in the profiler documentation.

If you also want to profile threads, you'll want to look at the threading.setprofile() function in the docs.

You could also create your own threading.Thread subclass to do it:

class ProfiledThread(threading.Thread):      # Overrides threading.Thread.run()      def run(self):          profiler = cProfile.Profile()          try:              return profiler.runcall(threading.Thread.run, self)          finally:              profiler.dump_stats('myprofile-%d.profile' % (self.ident,))  

and use that ProfiledThread class instead of the standard one. It might give you more flexibility, but I'm not sure it's worth it, especially if you are using third-party code which wouldn't use your class.

Answer by brent.payne for How can you profile a Python script?

The python wiki is a great page for profiling resources: http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Profiling_Code

as is the python docs: http://docs.python.org/library/profile.html

as shown by Chris Lawlor cProfile is a great tool and can easily be used to print to the screen:

python -m cProfile -s time mine.py   

or to file:

python -m cProfile -o output.file mine.py   

PS> If you are using Ubuntu, make sure to install python-profile

sudo apt-get install python-profiler   

If you output to file you can get nice visualizations using the following tools

PyCallGraph : a tool to create call graph images
install:

 sudo pip install pycallgraph  

run:

 pycallgraph mine.py args  

view:

 gimp pycallgraph.png  

You can use whatever you like to view the png file, I used gimp
Unfortunately I often get

dot: graph is too large for cairo-renderer bitmaps. Scaling by 0.257079 to fit

which makes my images unusably small. So I generally create svg files:

pycallgraph -f svg -o pycallgraph.svg mine.py   

PS> make sure to install graphviz (which provides the dot program):

sudo pip install graphviz  

Alternative Graphing using gprof2dot via @maxy / @quodlibetor :

sudo pip install gprof2dot  python -m cProfile -o profile.pstats mine.py  gprof2dot -f pstats profile.pstats | dot -Tsvg -o mine.svg  

Answer by Ian Langmore for How can you profile a Python script?

A nice profiling module is the line_profiler (called using the script kernprof.py). It can be downloaded here.

My understanding is that cProfile only gives information about total time spent in each function. So individual lines of code are not timed. This is an issue in scientific computing since often one single line can take a lot of time. Also, as I remember, cProfile didn't catch the time I was spending in say numpy.dot.

Answer by PypeBros for How can you profile a Python script?

Following Joe Shaw's answer about multi-threaded code not to work as expected, I figured that the runcall method in cProfile is merely doing self.enable() and self.disable() calls around the profiled function call, so you can simply do that yourself and have whatever code you want in-between with minimal interference with existing code.

Answer by Gerald Kaszuba for How can you profile a Python script?

A while ago I made pycallgraph which generates a visualisation from your Python code. Edit: I've updated the example to work with the latest release.

After a pip install pycallgraph and installing GraphViz you can run it from the command line:

pycallgraph graphviz -- ./mypythonscript.py  

Or, you can profile particular parts of your code:

from pycallgraph import PyCallGraph  from pycallgraph.output import GraphvizOutput    with PyCallGraph(output=GraphvizOutput()):      code_to_profile()  

Either of these will generate a pycallgraph.png file similar to the image below:

Answer by Colonel Panic for How can you profile a Python script?

Ever want to know what the hell that python script is doing? Enter the Inspect Shell. Inspect Shell lets you print/alter globals and run functions without interrupting the running script. Now with auto-complete and command history (only on linux).

Inspect Shell is not a pdb-style debugger.

https://github.com/amoffat/Inspect-Shell

You could use that (and your wristwatch).

Answer by quodlibetor for How can you profile a Python script?

@Maxy's comment on this answer helped me out enough that I think it deserves its own answer: I already had cProfile-generated .pstats files and I didn't want to re-run things with pycallgraph, so I used gprof2dot, and got pretty svgs:

$ sudo apt-get install graphviz  $ git clone https://code.google.com/p/jrfonseca.gprof2dot/ gprof2dot  $ ln -s "$PWD"/gprof2dot/gprof2dot.py ~/bin  $ cd $PROJECT_DIR  $ gprof2dot.py -f pstats profile.pstats | dot -Tsvg -o callgraph.svg  

and BLAM!

It uses dot (the same thing that pycallgraph uses) so output looks similar. I get the impression that gprof2dot loses less information though:

Answer by Mr. Girgitt for How can you profile a Python script?

My way is to use yappi (https://code.google.com/p/yappi/). It's especially useful combined with an RPC server where (even just for debugging) you register method to start, stop and print profiling information, e.g. in this way:

@staticmethod  def startProfiler():      yappi.start()    @staticmethod  def stopProfiler():      yappi.stop()    @staticmethod  def printProfiler():      stats = yappi.get_stats(yappi.SORTTYPE_TTOT, yappi.SORTORDER_DESC, 20)      statPrint = '\n'      namesArr = [len(str(stat[0])) for stat in stats.func_stats]      log.debug("namesArr %s", str(namesArr))      maxNameLen = max(namesArr)      log.debug("maxNameLen: %s", maxNameLen)        for stat in stats.func_stats:          nameAppendSpaces = [' ' for i in range(maxNameLen - len(stat[0]))]          log.debug('nameAppendSpaces: %s', nameAppendSpaces)          blankSpace = ''          for space in nameAppendSpaces:              blankSpace += space            log.debug("adding spaces: %s", len(nameAppendSpaces))          statPrint = statPrint + str(stat[0]) + blankSpace + " " + str(stat[1]).ljust(8) + "\t" + str(              round(stat[2], 2)).ljust(8 - len(str(stat[2]))) + "\t" + str(round(stat[3], 2)) + "\n"        log.log(1000, "\nname" + ''.ljust(maxNameLen - 4) + " ncall \tttot \ttsub")      log.log(1000, statPrint)  

Then when your program work you can start profiler at any time by calling the startProfiler RPC method and dump profiling information to a log file by calling printProfiler (or modify the rpc method to return it to the caller) and get such output:

2014-02-19 16:32:24,128-|SVR-MAIN  |-(Thread-3   )-Level 1000:   name                                                                                                                                      ncall     ttot    tsub  2014-02-19 16:32:24,128-|SVR-MAIN  |-(Thread-3   )-Level 1000:   C:\Python27\lib\sched.py.run:80                                                                                                           22        0.11    0.05  M:\02_documents\_repos\09_aheadRepos\apps\ahdModbusSrv\pyAheadRpcSrv\xmlRpc.py.iterFnc:293                                                22        0.11    0.0  M:\02_documents\_repos\09_aheadRepos\apps\ahdModbusSrv\serverMain.py.makeIteration:515                                                    22        0.11    0.0  M:\02_documents\_repos\09_aheadRepos\apps\ahdModbusSrv\pyAheadRpcSrv\PicklingXMLRPC.py._dispatch:66                                       1         0.0     0.0  C:\Python27\lib\BaseHTTPServer.py.date_time_string:464                                                                                    1         0.0     0.0  c:\users\zasiec~1\appdata\local\temp\easy_install-hwcsr1\psutil-1.1.2-py2.7-win32.egg.tmp\psutil\_psmswindows.py._get_raw_meminfo:243     4         0.0     0.0  C:\Python27\lib\SimpleXMLRPCServer.py.decode_request_content:537                                                                          1         0.0     0.0  c:\users\zasiec~1\appdata\local\temp\easy_install-hwcsr1\psutil-1.1.2-py2.7-win32.egg.tmp\psutil\_psmswindows.py.get_system_cpu_times:148 4         0.0     0.0  .__new__:8                                                                                                                        220       0.0     0.0  C:\Python27\lib\socket.py.close:276                                                                                                       4         0.0     0.0  C:\Python27\lib\threading.py.__init__:558                                                                                                 1         0.0     0.0  .__new__:8                                                                                                                        4         0.0     0.0  C:\Python27\lib\threading.py.notify:372                                                                                                   1         0.0     0.0  C:\Python27\lib\rfc822.py.getheader:285                                                                                                   4         0.0     0.0  C:\Python27\lib\BaseHTTPServer.py.handle_one_request:301                                                                                  1         0.0     0.0  C:\Python27\lib\xmlrpclib.py.end:816                                                                                                      3         0.0     0.0  C:\Python27\lib\SimpleXMLRPCServer.py.do_POST:467                                                                                         1         0.0     0.0  C:\Python27\lib\SimpleXMLRPCServer.py.is_rpc_path_valid:460                                                                               1         0.0     0.0  C:\Python27\lib\SocketServer.py.close_request:475                                                                                         1         0.0     0.0  c:\users\zasiec~1\appdata\local\temp\easy_install-hwcsr1\psutil-1.1.2-py2.7-win32.egg.tmp\psutil\__init__.py.cpu_times:1066               4         0.0     0.0   

It may not be very useful for short scripts but helps to optimize server-type processes especially given the printProfiler method can be called multiple times over time to profile and compare e.g. different program usage scenarios.

Answer by Pete for How can you profile a Python script?

Also worth mentioning is the GUI cProfile dump viewer RunSnakeRun. It allows you to sort and select, thereby zooming in on the relevant parts of the program. The sizes of the rectangles in the picture is proportional to the time taken. If you mouse over a rectangle it highlights that call in the table and everywhere on the map. When you double-click on a rectangle it zooms in on that portion. It will show you who calls that portion and what that portion calls.

The descriptive information is very helpful. It shows you the code for that bit which can be helpful when you are dealing with built-in library calls. It tells you what file and what line to find the code.

Also want to point at that the OP said 'profiling' but it appears he meant 'timing'. Keep in mind programs will run slower when profiled.

Answer by BenC for How can you profile a Python script?

line_profiler (already presented here) also inspired pprofile, which is described as:

Line-granularity, thread-aware deterministic and statistic pure-python profiler

It provides line-granularity as line_profiler, is pure Python, can be used as a standalone command or a module, and can even generate callgrind-format files that can be easily analyzed with [k|q]cachegrind.

Answer by michael for How can you profile a Python script?

To add on to http://stackoverflow.com/a/582337/1070617,

I wrote this module that allows you to use cProfile and view its output easily. More here: https://github.com/ymichael/cprofilev

$ python -m cprofilev /your/python/program  # Go to http://localhost:4000 to view collected statistics.  

Also see: http://ymichael.com/2014/03/08/profiling-python-with-cprofile.html on how to make sense of the collected statistics.

Answer by Datageek for How can you profile a Python script?

cProfile is great for quick profiling but most of the time it was ending for me with the errors. Function runctx solves this problem by initializing correctly the environment and variables, hope it can be useful for someone:

import cProfile  cProfile.runctx('foo()', None, locals())  

Answer by Fabio Zadrozny for How can you profile a Python script?

A new tool to handle profiling in Python is PyVmMonitor: http://www.pyvmmonitor.com/

It has some unique features such as

• Attach profiler to a running (CPython) program
• On demand profiling with Yappi integration
• Profile on a different machine
• Multiple processes support (multiprocessing, django...)
• Live sampling/CPU view (with time range selection)
• Deterministic profiling through cProfile/profile integration
• Analyze existing PStats results
• Open DOT files
• Programatic API access
• Group samples by method or line
• PyDev integration
• PyCharm integration

Note: it's commercial, but free for open source.

Answer by Vivian De Smedt for How can you profile a Python script?

Profilers tends to slow down very much code that are usually already very slow.

The plop profiler at https://github.com/bdarnell/plop is a efficient way to profile code.

Answer by David Mašek for How can you profile a Python script?

There's a lot of great answers but they either use command line or some external program for profiling and/or sorting the results.

I really missed some way I could use in my IDE (eclipse-PyDev) without touching the command line or installing anything. So here it is.

Profiling without command line

def count():      from math import sqrt      for x in range(10**5):          sqrt(x)    if __name__ == '__main__':      import cProfile, pstats      cProfile.run("count()", "{}.profile".format(__file__))      s = pstats.Stats("{}.profile".format(__file__))      s.strip_dirs()      s.sort_stats("time").print_stats(10)  

See docs or other answers for more info.

Answer by z0r for How can you profile a Python script?

There's also a statistical profiler called statprof. It's a sampling profiler, so it adds minimal overhead to your code and gives line-based (not just function-based) timings. It's more suited to soft real-time applications like games, but may be have less precision than cProfile.

The version in pypi is a bit old (e.g. doesn't support Python 3), but you can install it with pip by specifying the git repository:

pip install git+git://github.com/bos/statprof.py@1a33eba91899afe17a8b752c6dfdec6f05dd0c01  

You can run it like this:

import statprof    with statprof.profile():      my_questionable_function()  

See also http://stackoverflow.com/a/10333592/320036

Fatal error: Call to a member function getElementsByTagName() on a non-object in D:\XAMPP INSTALLASTION\xampp\htdocs\endunpratama9i\www-stackoverflow-info-proses.php on line 72