diff --git a/Doc/library/cmdline.rst b/Doc/library/cmdline.rst index c43b10157f9aea..6418706269f1ed 100644 --- a/Doc/library/cmdline.rst +++ b/Doc/library/cmdline.rst @@ -12,7 +12,7 @@ The following modules have a command-line interface. * :ref:`calendar ` * :mod:`code` * :ref:`compileall ` -* :mod:`cProfile`: see :ref:`profile ` +* ``cProfile``: see :ref:`profiling.tracing ` * :ref:`dis ` * :ref:`doctest ` * :mod:`!encodings.rot_13` @@ -31,8 +31,9 @@ The following modules have a command-line interface. * :ref:`pickletools ` * :ref:`platform ` * :mod:`poplib` -* :ref:`profile ` -* :mod:`pstats` +* :ref:`profiling.sampling ` +* :ref:`profiling.tracing ` +* :ref:`pstats ` * :ref:`py_compile ` * :mod:`pyclbr` * :mod:`pydoc` diff --git a/Doc/library/debug.rst b/Doc/library/debug.rst index 60223657a44043..f87c2481fb89cd 100644 --- a/Doc/library/debug.rst +++ b/Doc/library/debug.rst @@ -1,5 +1,5 @@ *********************** -Debugging and Profiling +Debugging and profiling *********************** These libraries help you with Python development: the debugger enables you to @@ -15,7 +15,8 @@ intrusive debugging or patching. bdb.rst faulthandler.rst pdb.rst - profile.rst + profiling.rst + pstats.rst timeit.rst trace.rst tracemalloc.rst diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index 03ad50b2c5eaf8..218aa88bc49d47 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -1,458 +1,64 @@ .. _profile: -******************** -The Python Profilers -******************** +**************************************** +:mod:`!profile` --- Pure Python profiler +**************************************** -**Source code:** :source:`Lib/profile.py`, :source:`Lib/pstats.py`, and :source:`Lib/profile/sample.py` +.. module:: profile + :synopsis: Pure Python profiler (deprecated). + :deprecated: --------------- +**Source code:** :source:`Lib/profile.py` -.. _profiler-introduction: +-------------- -Introduction to the profilers -============================= +.. deprecated-removed:: 3.15 3.17 -.. index:: - single: statistical profiling - single: profiling, statistical - single: deterministic profiling - single: profiling, deterministic +The :mod:`profile` module is deprecated and will be removed in Python 3.17. +Use :mod:`profiling.tracing` instead. -Python provides both :dfn:`statistical profiling` and :dfn:`deterministic profiling` of -Python programs. A :dfn:`profile` is a set of statistics that describes how -often and for how long various parts of the program executed. These statistics -can be formatted into reports via the :mod:`pstats` module. +The :mod:`profile` module provides a pure Python implementation of a +deterministic profiler. While useful for understanding profiler internals or +extending profiler behavior through subclassing, its pure Python implementation +introduces significant overhead compared to the C-based :mod:`profiling.tracing` +module. -The Python standard library provides three different profiling implementations: +For most profiling tasks, use: -**Statistical Profiler:** +- :mod:`profiling.sampling` for production debugging with zero overhead +- :mod:`profiling.tracing` for development and testing -1. :mod:`!profiling.sampling` provides statistical profiling of running Python processes - using periodic stack sampling. It can attach to any running Python process without - requiring code modification or restart, making it ideal for production debugging. -**Deterministic Profilers:** +Migration +========= -2. :mod:`cProfile` is recommended for development and testing; it's a C extension with - reasonable overhead that makes it suitable for profiling long-running - programs. Based on :mod:`lsprof`, contributed by Brett Rosen and Ted - Czotter. +Migrating from :mod:`profile` to :mod:`profiling.tracing` is straightforward. +The APIs are compatible:: -3. :mod:`profile`, a pure Python module whose interface is imitated by - :mod:`cProfile`, but which adds significant overhead to profiled programs. - If you're trying to extend the profiler in some way, the task might be easier - with this module. Originally designed and written by Jim Roskind. + # Old (deprecated) + import profile + profile.run('my_function()') -.. note:: + # New (recommended) + import profiling.tracing + profiling.tracing.run('my_function()') - The profiler modules are designed to provide an execution profile for a given - program, not for benchmarking purposes (for that, there is :mod:`timeit` for - reasonably accurate results). This particularly applies to benchmarking - Python code against C code: the profilers introduce overhead for Python code, - but not for C-level functions, and so the C code would seem faster than any - Python one. - -**Profiler Comparison:** - -+-------------------+--------------------------+----------------------+----------------------+ -| Feature | Statistical | Deterministic | Deterministic | -| | (``profiling.sampling``) | (``cProfile``) | (``profile``) | -+===================+==========================+======================+======================+ -| **Target** | Running process | Code you run | Code you run | -+-------------------+--------------------------+----------------------+----------------------+ -| **Overhead** | Virtually none | Moderate | High | -+-------------------+--------------------------+----------------------+----------------------+ -| **Accuracy** | Statistical approx. | Exact call counts | Exact call counts | -+-------------------+--------------------------+----------------------+----------------------+ -| **Setup** | Attach to any PID | Instrument code | Instrument code | -+-------------------+--------------------------+----------------------+----------------------+ -| **Use Case** | Production debugging | Development/testing | Profiler extension | -+-------------------+--------------------------+----------------------+----------------------+ -| **Implementation**| C extension | C extension | Pure Python | -+-------------------+--------------------------+----------------------+----------------------+ +For most code, replacing ``import profile`` with ``import profiling.tracing`` +(and using ``profiling.tracing`` instead of ``profile`` throughout) provides +a straightforward migration path. .. note:: - The statistical profiler (:mod:`!profiling.sampling`) is recommended for most production - use cases due to its extremely low overhead and ability to profile running processes - without modification. It can attach to any Python process and collect performance - data with minimal impact on execution speed, making it ideal for debugging - performance issues in live applications. - - -.. _statistical-profiling: - -What Is Statistical Profiling? -============================== - -:dfn:`Statistical profiling` works by periodically interrupting a running -program to capture its current call stack. Rather than monitoring every -function entry and exit like deterministic profilers, it takes snapshots at -regular intervals to build a statistical picture of where the program spends -its time. - -The sampling profiler uses process memory reading (via system calls like -``process_vm_readv`` on Linux, ``vm_read`` on macOS, and ``ReadProcessMemory`` on -Windows) to attach to a running Python process and extract stack trace -information without requiring any code modification or restart of the target -process. This approach provides several key advantages over traditional -profiling methods. - -The fundamental principle is that if a function appears frequently in the -collected stack samples, it is likely consuming significant CPU time. By -analyzing thousands of samples, the profiler can accurately estimate the -relative time spent in different parts of the program. The statistical nature -means that while individual measurements may vary, the aggregate results -converge to represent the true performance characteristics of the application. - -Since statistical profiling operates externally to the target process, it -introduces virtually no overhead to the running program. The profiler process -runs separately and reads the target process memory without interrupting its -execution. This makes it suitable for profiling production systems where -performance impact must be minimized. - -The accuracy of statistical profiling improves with the number of samples -collected. Short-lived functions may be missed or underrepresented, while -long-running functions will be captured proportionally to their execution time. -This characteristic makes statistical profiling particularly effective for -identifying the most significant performance bottlenecks rather than providing -exhaustive coverage of all function calls. - -Statistical profiling excels at answering questions like "which functions -consume the most CPU time?" and "where should I focus optimization efforts?" -rather than "exactly how many times was this function called?" The trade-off -between precision and practicality makes it an invaluable tool for performance -analysis in real-world applications. - -.. _profile-instant: - -Instant User's Manual -===================== - -This section is provided for users that "don't want to read the manual." It -provides a very brief overview, and allows a user to rapidly perform profiling -on an existing application. - -**Statistical Profiling (Recommended for Production):** - -To profile an existing running process:: - - python -m profiling.sampling 1234 - -To profile with custom settings:: - - python -m profiling.sampling -i 50 -d 30 1234 - -**Deterministic Profiling (Development/Testing):** - -To profile a function that takes a single argument, you can do:: - - import cProfile - import re - cProfile.run('re.compile("foo|bar")') - -(Use :mod:`profile` instead of :mod:`cProfile` if the latter is not available on -your system.) - -The above action would run :func:`re.compile` and print profile results like -the following:: - - 214 function calls (207 primitive calls) in 0.002 seconds - - Ordered by: cumulative time - - ncalls tottime percall cumtime percall filename:lineno(function) - 1 0.000 0.000 0.002 0.002 {built-in method builtins.exec} - 1 0.000 0.000 0.001 0.001 :1() - 1 0.000 0.000 0.001 0.001 __init__.py:250(compile) - 1 0.000 0.000 0.001 0.001 __init__.py:289(_compile) - 1 0.000 0.000 0.000 0.000 _compiler.py:759(compile) - 1 0.000 0.000 0.000 0.000 _parser.py:937(parse) - 1 0.000 0.000 0.000 0.000 _compiler.py:598(_code) - 1 0.000 0.000 0.000 0.000 _parser.py:435(_parse_sub) - -The first line indicates that 214 calls were monitored. Of those calls, 207 -were :dfn:`primitive`, meaning that the call was not induced via recursion. The -next line: ``Ordered by: cumulative time`` indicates the output is sorted -by the ``cumtime`` values. The column headings include: - -ncalls - for the number of calls. - -tottime - for the total time spent in the given function (and excluding time made in - calls to sub-functions) - -percall - is the quotient of ``tottime`` divided by ``ncalls`` - -cumtime - is the cumulative time spent in this and all subfunctions (from invocation - till exit). This figure is accurate *even* for recursive functions. - -percall - is the quotient of ``cumtime`` divided by primitive calls - -filename:lineno(function) - provides the respective data of each function - -When there are two numbers in the first column (for example ``3/1``), it means -that the function recursed. The second value is the number of primitive calls -and the former is the total number of calls. Note that when the function does -not recurse, these two values are the same, and only the single figure is -printed. - -Instead of printing the output at the end of the profile run, you can save the -results to a file by specifying a filename to the :func:`run` function:: - - import cProfile - import re - cProfile.run('re.compile("foo|bar")', 'restats') - -The :class:`pstats.Stats` class reads profile results from a file and formats -them in various ways. - -.. _sampling-profiler-cli: - -Statistical Profiler Command Line Interface -=========================================== - -.. program:: profiling.sampling - -The :mod:`!profiling.sampling` module can be invoked as a script to profile running processes:: - - python -m profiling.sampling [options] PID - -**Basic Usage Examples:** - -Profile process 1234 for 10 seconds with default settings:: - - python -m profiling.sampling 1234 - -Profile with custom interval and duration, save to file:: - - python -m profiling.sampling -i 50 -d 30 -o profile.stats 1234 - -Generate collapsed stacks to use with tools like `flamegraph.pl -`_:: - - python -m profiling.sampling --collapsed 1234 - -Profile all threads, sort by total time:: - - python -m profiling.sampling -a --sort-tottime 1234 - -Profile with real-time sampling statistics:: - - python -m profiling.sampling --realtime-stats 1234 - -**Command Line Options:** - -.. option:: PID - - Process ID of the Python process to profile (required) - -.. option:: -i, --interval INTERVAL - - Sampling interval in microseconds (default: 100) - -.. option:: -d, --duration DURATION - - Sampling duration in seconds (default: 10) - -.. option:: -a, --all-threads - - Sample all threads in the process instead of just the main thread - -.. option:: --native - - Include artificial ```` frames to denote calls to non-Python code. - -.. option:: --no-gc - - Don't include artificial ```` frames to denote active garbage collection. - -.. option:: --realtime-stats - - Print real-time sampling statistics during profiling - -.. option:: --pstats - - Generate pstats output (default) - -.. option:: --collapsed - - Generate collapsed stack traces for flamegraphs - -.. option:: -o, --outfile OUTFILE - - Save output to a file - -**Sorting Options (pstats format only):** + The ``cProfile`` module remains available as a backward-compatible alias + to :mod:`profiling.tracing`. Existing code using ``import cProfile`` will + continue to work without modification. -.. option:: --sort-nsamples - Sort by number of direct samples +:mod:`!profile` and :mod:`!profiling.tracing` module reference +============================================================== -.. option:: --sort-tottime - - Sort by total time - -.. option:: --sort-cumtime - - Sort by cumulative time (default) - -.. option:: --sort-sample-pct - - Sort by sample percentage - -.. option:: --sort-cumul-pct - - Sort by cumulative sample percentage - -.. option:: --sort-nsamples-cumul - - Sort by cumulative samples - -.. option:: --sort-name - - Sort by function name - -.. option:: -l, --limit LIMIT - - Limit the number of rows in the output (default: 15) - -.. option:: --no-summary - - Disable the summary section in the output - -**Understanding Statistical Profile Output:** - -The statistical profiler produces output similar to deterministic profilers but with different column meanings:: - - Profile Stats: - nsamples sample% tottime (ms) cumul% cumtime (ms) filename:lineno(function) - 45/67 12.5 23.450 18.6 56.780 mymodule.py:42(process_data) - 23/23 6.4 15.230 6.4 15.230 :0(len) - -**Column Meanings:** - -- **nsamples**: ``direct/cumulative`` - Times function was directly executing / on call stack -- **sample%**: Percentage of total samples where function was directly executing -- **tottime**: Estimated time spent directly in this function -- **cumul%**: Percentage of samples where function was anywhere on call stack -- **cumtime**: Estimated cumulative time including called functions -- **filename:lineno(function)**: Location and name of the function - -.. _profile-cli: - -Deterministic Profiler Command Line Interface -============================================= - -.. program:: cProfile - -The files :mod:`cProfile` and :mod:`profile` can also be invoked as a script to -profile another script. For example:: - - python -m cProfile [-o output_file] [-s sort_order] (-m module | myscript.py) - -.. option:: -o - - Writes the profile results to a file instead of to stdout. - -.. option:: -s - - Specifies one of the :func:`~pstats.Stats.sort_stats` sort values - to sort the output by. - This only applies when :option:`-o ` is not supplied. - -.. option:: -m - - Specifies that a module is being profiled instead of a script. - - .. versionadded:: 3.7 - Added the ``-m`` option to :mod:`cProfile`. - - .. versionadded:: 3.8 - Added the ``-m`` option to :mod:`profile`. - -The :mod:`pstats` module's :class:`~pstats.Stats` class has a variety of methods -for manipulating and printing the data saved into a profile results file:: - - import pstats - from pstats import SortKey - p = pstats.Stats('restats') - p.strip_dirs().sort_stats(-1).print_stats() - -The :meth:`~pstats.Stats.strip_dirs` method removed the extraneous path from all -the module names. The :meth:`~pstats.Stats.sort_stats` method sorted all the -entries according to the standard module/line/name string that is printed. The -:meth:`~pstats.Stats.print_stats` method printed out all the statistics. You -might try the following sort calls:: - - p.sort_stats(SortKey.NAME) - p.print_stats() - -The first call will actually sort the list by function name, and the second call -will print out the statistics. The following are some interesting calls to -experiment with:: - - p.sort_stats(SortKey.CUMULATIVE).print_stats(10) - -This sorts the profile by cumulative time in a function, and then only prints -the ten most significant lines. If you want to understand what algorithms are -taking time, the above line is what you would use. - -If you were looking to see what functions were looping a lot, and taking a lot -of time, you would do:: - - p.sort_stats(SortKey.TIME).print_stats(10) - -to sort according to time spent within each function, and then print the -statistics for the top ten functions. - -You might also try:: - - p.sort_stats(SortKey.FILENAME).print_stats('__init__') - -This will sort all the statistics by file name, and then print out statistics -for only the class init methods (since they are spelled with ``__init__`` in -them). As one final example, you could try:: - - p.sort_stats(SortKey.TIME, SortKey.CUMULATIVE).print_stats(.5, 'init') - -This line sorts statistics with a primary key of time, and a secondary key of -cumulative time, and then prints out some of the statistics. To be specific, the -list is first culled down to 50% (re: ``.5``) of its original size, then only -lines containing ``init`` are maintained, and that sub-sub-list is printed. - -If you wondered what functions called the above functions, you could now (``p`` -is still sorted according to the last criteria) do:: - - p.print_callers(.5, 'init') - -and you would get a list of callers for each of the listed functions. - -If you want more functionality, you're going to have to read the manual, or -guess what the following functions do:: - - p.print_callees() - p.add('restats') - -Invoked as a script, the :mod:`pstats` module is a statistics browser for -reading and examining profile dumps. It has a simple line-oriented interface -(implemented using :mod:`cmd`) and interactive help. - -:mod:`profile` and :mod:`cProfile` Module Reference -======================================================= - -.. module:: cProfile -.. module:: profile - :synopsis: Python source profiler. - -Both the :mod:`profile` and :mod:`cProfile` modules provide the following -functions: +Both the :mod:`profile` and :mod:`profiling.tracing` modules provide the +following functions: .. function:: run(command, filename=None, sort=-1) @@ -480,7 +86,7 @@ functions: .. class:: Profile(timer=None, timeunit=0.0, subcalls=True, builtins=True) This class is normally only used if more precise control over profiling is - needed than what the :func:`cProfile.run` function provides. + needed than what the :func:`profiling.tracing.run` function provides. A custom timer can be supplied for measuring how long code takes to run via the *timer* argument. This must be a function that returns a single number @@ -492,9 +98,12 @@ functions: Directly using the :class:`Profile` class allows formatting profile results without writing the profile data to a file:: - import cProfile, pstats, io + import profiling.tracing + import pstats + import io from pstats import SortKey - pr = cProfile.Profile() + + pr = profiling.tracing.Profile() pr.enable() # ... do something ... pr.disable() @@ -505,11 +114,12 @@ functions: print(s.getvalue()) The :class:`Profile` class can also be used as a context manager (supported - only in :mod:`cProfile` module. see :ref:`typecontextmanager`):: + only in :mod:`profiling.tracing`, not in the deprecated :mod:`profile` + module; see :ref:`typecontextmanager`):: - import cProfile + import profiling.tracing - with cProfile.Profile() as pr: + with profiling.tracing.Profile() as pr: # ... do something ... pr.print_stats() @@ -519,11 +129,11 @@ functions: .. method:: enable() - Start collecting profiling data. Only in :mod:`cProfile`. + Start collecting profiling data. Only in :mod:`profiling.tracing`. .. method:: disable() - Stop collecting profiling data. Only in :mod:`cProfile`. + Stop collecting profiling data. Only in :mod:`profiling.tracing`. .. method:: create_stats() @@ -537,7 +147,7 @@ functions: The *sort* parameter specifies the sorting order of the displayed statistics. It accepts a single key or a tuple of keys to enable - multi-level sorting, as in :func:`Stats.sort_stats `. + multi-level sorting, as in :meth:`pstats.Stats.sort_stats`. .. versionadded:: 3.13 :meth:`~Profile.print_stats` now accepts a tuple of keys. @@ -564,237 +174,43 @@ returns. If the interpreter is terminated (e.g. via a :func:`sys.exit` call during the called command/function execution) no profiling results will be printed. -.. _profile-stats: - -The :class:`Stats` Class -======================== -Analysis of the profiler data is done using the :class:`~pstats.Stats` class. +Differences from :mod:`!profiling.tracing` +========================================== -.. module:: pstats - :synopsis: Statistics object for use with the profiler. +The :mod:`profile` module differs from :mod:`profiling.tracing` in several +ways: -.. class:: Stats(*filenames or profile, stream=sys.stdout) +**Higher overhead.** The pure Python implementation is significantly slower +than the C implementation, making it unsuitable for profiling long-running +programs or performance-sensitive code. - This class constructor creates an instance of a "statistics object" from a - *filename* (or list of filenames) or from a :class:`Profile` instance. Output - will be printed to the stream specified by *stream*. +**Calibration support.** The :mod:`profile` module supports calibration to +compensate for profiling overhead. This is not needed in :mod:`profiling.tracing` +because the C implementation has negligible overhead. - The file selected by the above constructor must have been created by the - corresponding version of :mod:`profile` or :mod:`cProfile`. To be specific, - there is *no* file compatibility guaranteed with future versions of this - profiler, and there is no compatibility with files produced by other - profilers, or the same profiler run on a different operating system. If - several files are provided, all the statistics for identical functions will - be coalesced, so that an overall view of several processes can be considered - in a single report. If additional files need to be combined with data in an - existing :class:`~pstats.Stats` object, the :meth:`~pstats.Stats.add` method - can be used. +**Custom timers.** Both modules support custom timers, but :mod:`profile` +accepts timer functions that return tuples (like :func:`os.times`), while +:mod:`profiling.tracing` requires a function returning a single number. - Instead of reading the profile data from a file, a :class:`cProfile.Profile` - or :class:`profile.Profile` object can be used as the profile data source. +**Subclassing.** The pure Python implementation is easier to subclass and +extend for custom profiling behavior. - :class:`Stats` objects have the following methods: - - .. method:: strip_dirs() - - This method for the :class:`Stats` class removes all leading path - information from file names. It is very useful in reducing the size of - the printout to fit within (close to) 80 columns. This method modifies - the object, and the stripped information is lost. After performing a - strip operation, the object is considered to have its entries in a - "random" order, as it was just after object initialization and loading. - If :meth:`~pstats.Stats.strip_dirs` causes two function names to be - indistinguishable (they are on the same line of the same filename, and - have the same function name), then the statistics for these two entries - are accumulated into a single entry. - - - .. method:: add(*filenames) - - This method of the :class:`Stats` class accumulates additional profiling - information into the current profiling object. Its arguments should refer - to filenames created by the corresponding version of :func:`profile.run` - or :func:`cProfile.run`. Statistics for identically named (re: file, line, - name) functions are automatically accumulated into single function - statistics. - - - .. method:: dump_stats(filename) - - Save the data loaded into the :class:`Stats` object to a file named - *filename*. The file is created if it does not exist, and is overwritten - if it already exists. This is equivalent to the method of the same name - on the :class:`profile.Profile` and :class:`cProfile.Profile` classes. - - - .. method:: sort_stats(*keys) - - This method modifies the :class:`Stats` object by sorting it according to - the supplied criteria. The argument can be either a string or a SortKey - enum identifying the basis of a sort (example: ``'time'``, ``'name'``, - ``SortKey.TIME`` or ``SortKey.NAME``). The SortKey enums argument have - advantage over the string argument in that it is more robust and less - error prone. - - When more than one key is provided, then additional keys are used as - secondary criteria when there is equality in all keys selected before - them. For example, ``sort_stats(SortKey.NAME, SortKey.FILE)`` will sort - all the entries according to their function name, and resolve all ties - (identical function names) by sorting by file name. - - For the string argument, abbreviations can be used for any key names, as - long as the abbreviation is unambiguous. - - The following are the valid string and SortKey: - - +------------------+---------------------+----------------------+ - | Valid String Arg | Valid enum Arg | Meaning | - +==================+=====================+======================+ - | ``'calls'`` | SortKey.CALLS | call count | - +------------------+---------------------+----------------------+ - | ``'cumulative'`` | SortKey.CUMULATIVE | cumulative time | - +------------------+---------------------+----------------------+ - | ``'cumtime'`` | N/A | cumulative time | - +------------------+---------------------+----------------------+ - | ``'file'`` | N/A | file name | - +------------------+---------------------+----------------------+ - | ``'filename'`` | SortKey.FILENAME | file name | - +------------------+---------------------+----------------------+ - | ``'module'`` | N/A | file name | - +------------------+---------------------+----------------------+ - | ``'ncalls'`` | N/A | call count | - +------------------+---------------------+----------------------+ - | ``'pcalls'`` | SortKey.PCALLS | primitive call count | - +------------------+---------------------+----------------------+ - | ``'line'`` | SortKey.LINE | line number | - +------------------+---------------------+----------------------+ - | ``'name'`` | SortKey.NAME | function name | - +------------------+---------------------+----------------------+ - | ``'nfl'`` | SortKey.NFL | name/file/line | - +------------------+---------------------+----------------------+ - | ``'stdname'`` | SortKey.STDNAME | standard name | - +------------------+---------------------+----------------------+ - | ``'time'`` | SortKey.TIME | internal time | - +------------------+---------------------+----------------------+ - | ``'tottime'`` | N/A | internal time | - +------------------+---------------------+----------------------+ - - Note that all sorts on statistics are in descending order (placing most - time consuming items first), where as name, file, and line number searches - are in ascending order (alphabetical). The subtle distinction between - ``SortKey.NFL`` and ``SortKey.STDNAME`` is that the standard name is a - sort of the name as printed, which means that the embedded line numbers - get compared in an odd way. For example, lines 3, 20, and 40 would (if - the file names were the same) appear in the string order 20, 3 and 40. - In contrast, ``SortKey.NFL`` does a numeric compare of the line numbers. - In fact, ``sort_stats(SortKey.NFL)`` is the same as - ``sort_stats(SortKey.NAME, SortKey.FILENAME, SortKey.LINE)``. - - For backward-compatibility reasons, the numeric arguments ``-1``, ``0``, - ``1``, and ``2`` are permitted. They are interpreted as ``'stdname'``, - ``'calls'``, ``'time'``, and ``'cumulative'`` respectively. If this old - style format (numeric) is used, only one sort key (the numeric key) will - be used, and additional arguments will be silently ignored. - - .. For compatibility with the old profiler. - - .. versionadded:: 3.7 - Added the SortKey enum. - - .. method:: reverse_order() - - This method for the :class:`Stats` class reverses the ordering of the - basic list within the object. Note that by default ascending vs - descending order is properly selected based on the sort key of choice. - - .. This method is provided primarily for compatibility with the old - profiler. - - - .. method:: print_stats(*restrictions) - - This method for the :class:`Stats` class prints out a report as described - in the :func:`profile.run` definition. - - The order of the printing is based on the last - :meth:`~pstats.Stats.sort_stats` operation done on the object (subject to - caveats in :meth:`~pstats.Stats.add` and - :meth:`~pstats.Stats.strip_dirs`). - - The arguments provided (if any) can be used to limit the list down to the - significant entries. Initially, the list is taken to be the complete set - of profiled functions. Each restriction is either an integer (to select a - count of lines), or a decimal fraction between 0.0 and 1.0 inclusive (to - select a percentage of lines), or a string that will be interpreted as a - regular expression (to pattern match the standard name that is printed). - If several restrictions are provided, then they are applied sequentially. - For example:: - - print_stats(.1, 'foo:') - - would first limit the printing to first 10% of list, and then only print - functions that were part of filename :file:`.\*foo:`. In contrast, the - command:: - - print_stats('foo:', .1) - - would limit the list to all functions having file names :file:`.\*foo:`, - and then proceed to only print the first 10% of them. - - - .. method:: print_callers(*restrictions) - - This method for the :class:`Stats` class prints a list of all functions - that called each function in the profiled database. The ordering is - identical to that provided by :meth:`~pstats.Stats.print_stats`, and the - definition of the restricting argument is also identical. Each caller is - reported on its own line. The format differs slightly depending on the - profiler that produced the stats: - - * With :mod:`profile`, a number is shown in parentheses after each caller - to show how many times this specific call was made. For convenience, a - second non-parenthesized number repeats the cumulative time spent in the - function at the right. - - * With :mod:`cProfile`, each caller is preceded by three numbers: the - number of times this specific call was made, and the total and - cumulative times spent in the current function while it was invoked by - this specific caller. - - - .. method:: print_callees(*restrictions) - - This method for the :class:`Stats` class prints a list of all function - that were called by the indicated function. Aside from this reversal of - direction of calls (re: called vs was called by), the arguments and - ordering are identical to the :meth:`~pstats.Stats.print_callers` method. - - - .. method:: get_stats_profile() - - This method returns an instance of StatsProfile, which contains a mapping - of function names to instances of FunctionProfile. Each FunctionProfile - instance holds information related to the function's profile such as how - long the function took to run, how many times it was called, etc... - - .. versionadded:: 3.9 - Added the following dataclasses: StatsProfile, FunctionProfile. - Added the following function: get_stats_profile. .. _deterministic-profiling: -What Is Deterministic Profiling? +What is deterministic profiling? ================================ :dfn:`Deterministic profiling` is meant to reflect the fact that all *function call*, *function return*, and *exception* events are monitored, and precise timings are made for the intervals between these events (during which time the user's code is executing). In contrast, :dfn:`statistical profiling` (which is -provided by the :mod:`!profiling.sampling` module) periodically samples the effective instruction pointer, and -deduces where time is being spent. The latter technique traditionally involves -less overhead (as the code does not need to be instrumented), but provides only -relative indications of where time is being spent. +provided by the :mod:`profiling.sampling` module) periodically samples the +effective instruction pointer, and deduces where time is being spent. The +latter technique traditionally involves less overhead (as the code does not +need to be instrumented), but provides only relative indications of where time +is being spent. In Python, since there is an interpreter active during execution, the presence of instrumented code is not required in order to do deterministic profiling. @@ -838,15 +254,16 @@ this error. The error that accumulates in this fashion is typically less than the accuracy of the clock (less than one clock tick), but it *can* accumulate and become very significant. -The problem is more important with :mod:`profile` than with the lower-overhead -:mod:`cProfile`. For this reason, :mod:`profile` provides a means of -calibrating itself for a given platform so that this error can be -probabilistically (on the average) removed. After the profiler is calibrated, it -will be more accurate (in a least square sense), but it will sometimes produce -negative numbers (when call counts are exceptionally low, and the gods of -probability work against you :-). ) Do *not* be alarmed by negative numbers in -the profile. They should *only* appear if you have calibrated your profiler, -and the results are actually better than without calibration. +The problem is more important with the deprecated :mod:`profile` module than +with the lower-overhead :mod:`profiling.tracing`. For this reason, +:mod:`profile` provides a means of calibrating itself for a given platform so +that this error can be probabilistically (on the average) removed. After the +profiler is calibrated, it will be more accurate (in a least square sense), but +it will sometimes produce negative numbers (when call counts are exceptionally +low, and the gods of probability work against you :-). ) Do *not* be alarmed +by negative numbers in the profile. They should *only* appear if you have +calibrated your profiler, and the results are actually better than without +calibration. .. _profile-calibration: @@ -892,6 +309,7 @@ When you have a consistent answer, there are three ways you can use it:: If you have a choice, you are better off choosing a smaller constant, and then your results will "less often" show up as negative in profile statistics. + .. _profile-timers: Using a custom timer @@ -904,7 +322,7 @@ to the :class:`Profile` class constructor:: pr = profile.Profile(your_time_func) The resulting profiler will then call ``your_time_func``. Depending on whether -you are using :class:`profile.Profile` or :class:`cProfile.Profile`, +you are using :class:`profile.Profile` or :class:`profiling.tracing.Profile`, ``your_time_func``'s return value will be interpreted differently: :class:`profile.Profile` @@ -923,20 +341,32 @@ you are using :class:`profile.Profile` or :class:`cProfile.Profile`, replacement dispatch method that best handles your timer call, along with the appropriate calibration constant. -:class:`cProfile.Profile` +:class:`profiling.tracing.Profile` ``your_time_func`` should return a single number. If it returns integers, you can also invoke the class constructor with a second argument specifying the real duration of one unit of time. For example, if ``your_integer_time_func`` returns times measured in thousands of seconds, you would construct the :class:`Profile` instance as follows:: - pr = cProfile.Profile(your_integer_time_func, 0.001) + pr = profiling.tracing.Profile(your_integer_time_func, 0.001) - As the :class:`cProfile.Profile` class cannot be calibrated, custom timer - functions should be used with care and should be as fast as possible. For - the best results with a custom timer, it might be necessary to hard-code it - in the C source of the internal :mod:`!_lsprof` module. + As the :class:`profiling.tracing.Profile` class cannot be calibrated, custom + timer functions should be used with care and should be as fast as possible. + For the best results with a custom timer, it might be necessary to hard-code + it in the C source of the internal :mod:`!_lsprof` module. Python 3.3 adds several new functions in :mod:`time` that can be used to make precise measurements of process or wall-clock time. For example, see :func:`time.perf_counter`. + + +.. seealso:: + + :mod:`profiling` + Overview of Python profiling tools. + + :mod:`profiling.tracing` + Recommended replacement for this module. + + :mod:`pstats` + Statistical analysis and formatting for profile data. diff --git a/Doc/library/profiling.rst b/Doc/library/profiling.rst new file mode 100644 index 00000000000000..4b56d9c4b7b1b8 --- /dev/null +++ b/Doc/library/profiling.rst @@ -0,0 +1,270 @@ +.. highlight:: shell-session + +.. _profiling-module: + +*************************************** +:mod:`profiling` --- Python profilers +*************************************** + +.. module:: profiling + :synopsis: Python profiling tools for performance analysis. + +.. versionadded:: 3.15 + +**Source code:** :source:`Lib/profiling/` + +-------------- + +.. index:: + single: statistical profiling + single: profiling, statistical + single: deterministic profiling + single: profiling, deterministic + + +Introduction to profiling +========================= + +A :dfn:`profile` is a set of statistics that describes how often and for how +long various parts of a program execute. These statistics help identify +performance bottlenecks and guide optimization efforts. Python provides two +fundamentally different approaches to collecting this information: statistical +sampling and deterministic tracing. + +The :mod:`profiling` package organizes Python's built-in profiling tools under +a single namespace. It contains two submodules, each implementing a different +profiling methodology: + +:mod:`profiling.sampling` + A statistical profiler that periodically samples the call stack. Run scripts + directly or attach to running processes by PID. Provides multiple output + formats (flame graphs, heatmaps, Firefox Profiler), GIL analysis, GC tracking, + and multiple profiling modes (wall-clock, CPU, GIL) with virtually no overhead. + +:mod:`profiling.tracing` + A deterministic profiler that traces every function call, return, and + exception event. Provides exact call counts and precise timing information, + capturing every invocation including very fast functions. + +.. note:: + + The profiler modules are designed to provide an execution profile for a + given program, not for benchmarking purposes. For benchmarking, use the + :mod:`timeit` module, which provides reasonably accurate timing + measurements. This distinction is particularly important when comparing + Python code against C code: deterministic profilers introduce overhead for + Python code but not for C-level functions, which can skew comparisons. + + +.. _choosing-a-profiler: + +Choosing a profiler +=================== + +For most performance analysis, use the statistical profiler +(:mod:`profiling.sampling`). It has minimal overhead, works for both development +and production, and provides rich visualization options including flamegraphs, +heatmaps, GIL analysis, and more. + +Use the deterministic profiler (:mod:`profiling.tracing`) when you need **exact +call counts** and cannot afford to miss any function calls. Since it instruments +every function call and return, it will capture even very fast functions that +complete between sampling intervals. The tradeoff is higher overhead. + +The following table summarizes the key differences: + ++--------------------+------------------------------+------------------------------+ +| Feature | Statistical sampling | Deterministic | +| | (:mod:`profiling.sampling`) | (:mod:`profiling.tracing`) | ++====================+==============================+==============================+ +| **Overhead** | Virtually none | Moderate | ++--------------------+------------------------------+------------------------------+ +| **Accuracy** | Statistical estimate | Exact call counts | ++--------------------+------------------------------+------------------------------+ +| **Output formats** | pstats, flamegraph, heatmap, | pstats | +| | gecko, collapsed | | ++--------------------+------------------------------+------------------------------+ +| **Profiling modes**| Wall-clock, CPU, GIL | Wall-clock | ++--------------------+------------------------------+------------------------------+ +| **Special frames** | GC, native (C extensions) | N/A | ++--------------------+------------------------------+------------------------------+ +| **Attach to PID** | Yes | No | ++--------------------+------------------------------+------------------------------+ + + +When to use statistical sampling +-------------------------------- + +The statistical profiler (:mod:`profiling.sampling`) is recommended for most +performance analysis tasks. Use it the same way you would use +:mod:`profiling.tracing`:: + + python -m profiling.sampling run script.py + +One of the main strengths of the sampling profiler is its variety of output +formats. Beyond traditional pstats tables, it can generate interactive +flamegraphs that visualize call hierarchies, line-level source heatmaps that +show exactly where time is spent in your code, and Firefox Profiler output for +timeline-based analysis. + +The profiler also provides insight into Python interpreter behavior that +deterministic profiling cannot capture. Use ``--mode gil`` to identify GIL +contention in multi-threaded code, ``--mode cpu`` to measure actual CPU time +excluding I/O waits, or inspect ```` frames to understand garbage collection +overhead. The ``--native`` option reveals time spent in C extensions, helping +distinguish Python overhead from library performance. + +For multi-threaded applications, the ``-a`` option samples all threads +simultaneously, showing how work is distributed. And for production debugging, +the ``attach`` command connects to any running Python process by PID without +requiring a restart or code changes. + + +When to use deterministic tracing +--------------------------------- + +The deterministic profiler (:mod:`profiling.tracing`) instruments every function +call and return. This approach has higher overhead than sampling, but guarantees +complete coverage of program execution. + +The primary reason to choose deterministic tracing is when you need exact call +counts. Statistical profiling estimates frequency based on sampling, which may +undercount short-lived functions that complete between samples. If you need to +verify that an optimization actually reduced the number of function calls, or +if you want to trace the complete call graph to understand caller-callee +relationships, deterministic tracing is the right choice. + +Deterministic tracing also excels at capturing functions that execute in +microseconds. Such functions may not appear frequently enough in statistical +samples, but deterministic tracing records every invocation regardless of +duration. + + +Quick start +=========== + +This section provides the minimal steps needed to start profiling. For complete +documentation, see the dedicated pages for each profiler. + + +Statistical profiling +--------------------- + +To profile a script, use the :mod:`profiling.sampling` module with the ``run`` +command:: + + python -m profiling.sampling run script.py + python -m profiling.sampling run -m mypackage.module + +This runs the script under the profiler and prints a summary of where time was +spent. For an interactive flamegraph:: + + python -m profiling.sampling run --flamegraph script.py + +To profile an already-running process, use the ``attach`` command with the +process ID:: + + python -m profiling.sampling attach 1234 + +For custom settings, specify the sampling interval (in microseconds) and +duration (in seconds):: + + python -m profiling.sampling run -i 50 -d 30 script.py + + +Deterministic profiling +----------------------- + +To profile a script from the command line:: + + python -m profiling.tracing script.py + +To profile a piece of code programmatically: + +.. code-block:: python + + import profiling.tracing + profiling.tracing.run('my_function()') + +This executes the given code under the profiler and prints a summary showing +exact function call counts and timing. + + +.. _profile-output: + +Understanding profile output +============================ + +Both profilers collect function-level statistics, though they present them in +different formats. The sampling profiler offers multiple visualizations +(flamegraphs, heatmaps, Firefox Profiler, pstats tables), while the +deterministic profiler produces pstats-compatible output. Regardless of format, +the underlying concepts are the same. + +Key profiling concepts: + +**Direct time** (also called *self time* or *tottime*) + Time spent executing code in the function itself, excluding time spent in + functions it called. High direct time indicates the function contains + expensive operations. + +**Cumulative time** (also called *total time* or *cumtime*) + Time spent in the function and all functions it called. This measures the + total cost of calling a function, including its entire call subtree. + +**Call count** (also called *ncalls* or *samples*) + How many times the function was called (deterministic) or sampled + (statistical). In deterministic profiling, this is exact. In statistical + profiling, it represents the number of times the function appeared in a + stack sample. + +**Primitive calls** + Calls that are not induced by recursion. When a function recurses, the total + call count includes recursive invocations, but primitive calls counts only + the initial entry. Displayed as ``total/primitive`` (for example, ``3/1`` + means three total calls, one primitive). + +**Caller/Callee relationships** + Which functions called a given function (callers) and which functions it + called (callees). Flamegraphs visualize this as nested rectangles; pstats + can display it via the :meth:`~pstats.Stats.print_callers` and + :meth:`~pstats.Stats.print_callees` methods. + + +Legacy compatibility +==================== + +For backward compatibility, the ``cProfile`` module remains available as an +alias to :mod:`profiling.tracing`. Existing code using ``import cProfile`` will +continue to work without modification in all future Python versions. + +.. deprecated:: 3.15 + + The pure Python :mod:`profile` module is deprecated and will be removed in + Python 3.17. Use :mod:`profiling.tracing` (or its alias ``cProfile``) + instead. See :mod:`profile` for migration guidance. + + +.. seealso:: + + :mod:`profiling.sampling` + Statistical sampling profiler with flamegraphs, heatmaps, and GIL analysis. + Recommended for most users. + + :mod:`profiling.tracing` + Deterministic tracing profiler for exact call counts. + + :mod:`pstats` + Statistics analysis and formatting for profile data. + + :mod:`timeit` + Module for measuring execution time of small code snippets. + + +.. rubric:: Submodules + +.. toctree:: + :maxdepth: 1 + + profiling.tracing.rst + profiling.sampling.rst diff --git a/Doc/library/profiling.sampling.rst b/Doc/library/profiling.sampling.rst new file mode 100644 index 00000000000000..e2e354c3d134fe --- /dev/null +++ b/Doc/library/profiling.sampling.rst @@ -0,0 +1,1035 @@ +.. highlight:: shell-session + +.. _profiling-sampling: + +*************************************************** +:mod:`profiling.sampling` --- Statistical profiler +*************************************************** + +.. module:: profiling.sampling + :synopsis: Statistical sampling profiler for Python processes. + +.. versionadded:: 3.15 + +**Source code:** :source:`Lib/profiling/sampling/` + +-------------- + +.. image:: tachyon-logo.png + :alt: Tachyon logo + :align: center + :width: 300px + +The :mod:`profiling.sampling` module, named **Tachyon**, provides statistical +profiling of Python programs through periodic stack sampling. Tachyon can +run scripts directly or attach to any running Python process without requiring +code changes or restarts. Because sampling occurs externally to the target +process, overhead is virtually zero, making Tachyon suitable for both +development and production environments. + + +What is statistical profiling? +============================== + +Statistical profiling builds a picture of program behavior by periodically +capturing snapshots of the call stack. Rather than instrumenting every function +call and return as deterministic profilers do, Tachyon reads the call stack at +regular intervals to record what code is currently running. + +This approach rests on a simple principle: functions that consume significant +CPU time will appear frequently in the collected samples. By gathering thousands +of samples over a profiling session, Tachyon constructs an accurate statistical +estimate of where time is spent. The more samples collected, the +more precise this estimate becomes. + + +How time is estimated +--------------------- + +The time values shown in Tachyon's output are **estimates derived from sample +counts**, not direct measurements. Tachyon counts how many times each function +appears in the collected samples, then multiplies by the sampling interval to +estimate time. + +For example, with a 100 microsecond sampling interval over a 10-second profile, +Tachyon collects approximately 100,000 samples. If a function appears in 5,000 +samples (5% of total), Tachyon estimates it consumed 5% of the 10-second +duration, or about 500 milliseconds. This is a statistical estimate, not a +precise measurement. + +The accuracy of these estimates depends on sample count. With 100,000 samples, +a function showing 5% has a margin of error of roughly ±0.5%. With only 1,000 +samples, the same 5% measurement could actually represent anywhere from 3% to +7% of real time. + +This is why longer profiling durations and shorter sampling intervals produce +more reliable results---they collect more samples. For most performance +analysis, the default settings provide sufficient accuracy to identify +bottlenecks and guide optimization efforts. + +Because sampling is statistical, results will vary slightly between runs. A +function showing 12% in one run might show 11% or 13% in the next. This is +normal and expected. Focus on the overall pattern rather than exact percentages, +and don't worry about small variations between runs. + + +When to use a different approach +-------------------------------- + +Statistical sampling is not ideal for every situation. + +For very short scripts that complete in under one second, the profiler may not +collect enough samples for reliable results. Use :mod:`profiling.tracing` +instead, or run the script in a loop to extend profiling time. + +When you need exact call counts, sampling cannot provide them. Sampling +estimates frequency from snapshots, so if you need to know precisely how many +times a function was called, use :mod:`profiling.tracing`. + +When comparing two implementations where the difference might be only 1-2%, +sampling noise can obscure real differences. Use :mod:`timeit` for +micro-benchmarks or :mod:`profiling.tracing` for precise measurements. + + +The key difference from :mod:`profiling.tracing` is how measurement happens. +A tracing profiler instruments your code, recording every function call and +return. This provides exact call counts and precise timing but adds overhead +to every function call. A sampling profiler, by contrast, observes the program +from outside at fixed intervals without modifying its execution. Think of the +difference like this: tracing is like having someone follow you and write down +every step you take, while sampling is like taking photographs every second +and inferring your path from those snapshots. + +This external observation model is what makes sampling profiling practical for +production use. The profiled program runs at full speed because there is no +instrumentation code running inside it, and the target process is never stopped +or paused during sampling---Tachyon reads the call stack directly from the +process's memory while it continues to run. You can attach to a live server, +collect data, and detach without the application ever knowing it was observed. +The trade-off is that very short-lived functions may be missed if they happen +to complete between samples. + +Statistical profiling excels at answering the question, "Where is my program +spending time?" It reveals hotspots and bottlenecks in production code where +deterministic profiling overhead would be unacceptable. For exact call counts +and complete call graphs, use :mod:`profiling.tracing` instead. + + +Quick examples +============== + +Profile a script and see the results immediately:: + + python -m profiling.sampling run script.py + +Profile a module with arguments:: + + python -m profiling.sampling run -m mypackage.module arg1 arg2 + +Generate an interactive flame graph:: + + python -m profiling.sampling run --flamegraph -o profile.html script.py + +Attach to a running process by PID:: + + python -m profiling.sampling attach 12345 + +Use live mode for real-time monitoring (press ``q`` to quit):: + + python -m profiling.sampling run --live script.py + +Profile for 60 seconds with a faster sampling rate:: + + python -m profiling.sampling run -d 60 -i 50 script.py + +Generate a line-by-line heatmap:: + + python -m profiling.sampling run --heatmap script.py + + +Commands +======== + +Tachyon operates through two subcommands that determine how to obtain the +target process. + + +The ``run`` command +------------------- + +The ``run`` command launches a Python script or module and profiles it from +startup:: + + python -m profiling.sampling run script.py + python -m profiling.sampling run -m mypackage.module + +When profiling a script, the profiler starts the target in a subprocess, waits +for it to initialize, then begins collecting samples. The ``-m`` flag +indicates that the target should be run as a module (equivalent to +``python -m``). Arguments after the target are passed through to the +profiled program:: + + python -m profiling.sampling run script.py --config settings.yaml + + +The ``attach`` command +---------------------- + +The ``attach`` command connects to an already-running Python process by its +process ID:: + + python -m profiling.sampling attach 12345 + +This command is particularly valuable for investigating performance issues in +production systems. The target process requires no modification and need not +be restarted. The profiler attaches, collects samples for the specified +duration, then detaches and produces output. + +On most systems, attaching to another process requires appropriate permissions. +See :ref:`profiling-permissions` for platform-specific requirements. + + +Profiling in production +----------------------- + +The sampling profiler is designed for production use. It imposes no measurable +overhead on the target process because it reads memory externally rather than +instrumenting code. The target application continues running at full speed and +is unaware it is being profiled. + +When profiling production systems, keep these guidelines in mind: + +Start with shorter durations (10-30 seconds) to get quick results, then extend +if you need more statistical accuracy. The default 10-second duration is usually +sufficient to identify major hotspots. + +If possible, profile during representative load rather than peak traffic. +Profiles collected during normal operation are easier to interpret than those +collected during unusual spikes. + +The profiler itself consumes some CPU on the machine where it runs (not on the +target process). On the same machine, this is typically negligible. When +profiling remote processes, network latency does not affect the target. + +Results from production may differ from development due to different data +sizes, concurrent load, or caching effects. This is expected and is often +exactly what you want to capture. + + +.. _profiling-permissions: + +Platform requirements +--------------------- + +The profiler reads the target process's memory to capture stack traces. This +requires elevated permissions on most operating systems. + +**Linux** + +On Linux, the profiler uses ``ptrace`` or ``process_vm_readv`` to read the +target process's memory. This typically requires one of: + +- Running as root +- Having the ``CAP_SYS_PTRACE`` capability +- Adjusting the Yama ptrace scope: ``/proc/sys/kernel/yama/ptrace_scope`` + +The default ptrace_scope of 1 restricts ptrace to parent processes only. To +allow attaching to any process owned by the same user, set it to 0:: + + echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope + +**macOS** + +On macOS, the profiler uses ``task_for_pid()`` to access the target process. +This requires one of: + +- Running as root +- The profiler binary having the ``com.apple.security.cs.debugger`` entitlement +- System Integrity Protection (SIP) being disabled (not recommended) + +**Windows** + +On Windows, the profiler requires administrative privileges or the +``SeDebugPrivilege`` privilege to read another process's memory. + + +Version compatibility +--------------------- + +The profiler and target process must run the same Python minor version (for +example, both Python 3.15). Attaching from Python 3.14 to a Python 3.15 process +is not supported. + +Additional restrictions apply to pre-release Python versions: if either the +profiler or target is running a pre-release (alpha, beta, or release candidate), +both must run the exact same version. + +On free-threaded Python builds, the profiler cannot attach from a free-threaded +build to a standard build, or vice versa. + + +Sampling configuration +====================== + +Before exploring the various output formats and visualization options, it is +important to understand how to configure the sampling process itself. The +profiler offers several options that control how frequently samples are +collected, how long profiling runs, which threads are observed, and what +additional context is captured in each sample. + +The default configuration works well for most use cases: + +.. list-table:: + :header-rows: 1 + :widths: 25 75 + + * - Option + - Default behavior + * - ``--interval`` / ``-i`` + - 100 µs between samples (~10,000 samples/sec) + * - ``--duration`` / ``-d`` + - Profile for 10 seconds + * - ``--all-threads`` / ``-a`` + - Sample main thread only + * - ``--native`` + - No ```` frames (C code time attributed to caller) + * - ``--no-gc`` + - Include ```` frames when garbage collection is active + * - ``--mode`` + - Wall-clock mode (all samples recorded) + * - ``--realtime-stats`` + - No live statistics display during profiling + + +Sampling interval and duration +------------------------------ + +The two most fundamental parameters are the sampling interval and duration. +Together, these determine how many samples will be collected during a profiling +session. + +The ``--interval`` option (``-i``) sets the time between samples in +microseconds. The default is 100 microseconds, which produces approximately +10,000 samples per second:: + + python -m profiling.sampling run -i 50 script.py + +Lower intervals capture more samples and provide finer-grained data at the +cost of slightly higher profiler CPU usage. Higher intervals reduce profiler +overhead but may miss short-lived functions. For most applications, the +default interval provides a good balance between accuracy and overhead. + +The ``--duration`` option (``-d``) sets how long to profile in seconds. The +default is 10 seconds:: + + python -m profiling.sampling run -d 60 script.py + +Longer durations collect more samples and produce more statistically reliable +results, especially for code paths that execute infrequently. When profiling +a program that runs for a fixed time, you may want to set the duration to +match or exceed the expected runtime. + + +Thread selection +---------------- + +Python programs often use multiple threads, whether explicitly through the +:mod:`threading` module or implicitly through libraries that manage thread +pools. + +By default, the profiler samples only the main thread. The ``--all-threads`` +option (``-a``) enables sampling of all threads in the process:: + + python -m profiling.sampling run -a script.py + +Multi-thread profiling reveals how work is distributed across threads and can +identify threads that are blocked or starved. Each thread's samples are +combined in the output, with the ability to filter by thread in some formats. +This option is particularly useful when investigating concurrency issues or +when work is distributed across a thread pool. + + +Special frames +-------------- + +The profiler can inject artificial frames into the captured stacks to provide +additional context about what the interpreter is doing at the moment each +sample is taken. These synthetic frames help distinguish different types of +execution that would otherwise be invisible. + +The ``--native`` option adds ```` frames to indicate when Python has +called into C code (extension modules, built-in functions, or the interpreter +itself):: + + python -m profiling.sampling run --native script.py + +These frames help distinguish time spent in Python code versus time spent in +native libraries. Without this option, native code execution appears as time +in the Python function that made the call. This is useful when optimizing +code that makes heavy use of C extensions like NumPy or database drivers. + +By default, the profiler includes ```` frames when garbage collection is +active. The ``--no-gc`` option suppresses these frames:: + + python -m profiling.sampling run --no-gc script.py + +GC frames help identify programs where garbage collection consumes significant +time, which may indicate memory allocation patterns worth optimizing. If you +see substantial time in ```` frames, consider investigating object +allocation rates or using object pooling. + + +Real-time statistics +-------------------- + +The ``--realtime-stats`` option displays sampling rate statistics during +profiling:: + + python -m profiling.sampling run --realtime-stats script.py + +This shows the actual achieved sampling rate, which may be lower than requested +if the profiler cannot keep up. The statistics help verify that profiling is +working correctly and that sufficient samples are being collected. See +:ref:`sampling-efficiency` for details on interpreting these metrics. + + +.. _sampling-efficiency: + +Sampling efficiency +------------------- + +Sampling efficiency metrics help assess the quality of the collected data. +These metrics appear in the profiler's terminal output and in the flame graph +sidebar. + +**Sampling efficiency** is the percentage of sample attempts that succeeded. +Each sample attempt reads the target process's call stack from memory. An +attempt can fail if the process is in an inconsistent state at the moment of +reading, such as during a context switch or while the interpreter is updating +its internal structures. A low efficiency may indicate that the profiler could +not keep up with the requested sampling rate, often due to system load or an +overly aggressive interval setting. + +**Missed samples** is the percentage of expected samples that were not +collected. Based on the configured interval and duration, the profiler expects +to collect a certain number of samples. Some samples may be missed if the +profiler falls behind schedule, for example when the system is under heavy +load. A small percentage of missed samples is normal and does not significantly +affect the statistical accuracy of the profile. + +Both metrics are informational. Even with some failed attempts or missed +samples, the profile remains statistically valid as long as enough samples +were collected. The profiler reports the actual number of samples captured, +which you can use to judge whether the data is sufficient for your analysis. + + +Profiling modes +=============== + +The sampling profiler supports three modes that control which samples are +recorded. The mode determines what the profile measures: total elapsed time, +CPU execution time, or time spent holding the global interpreter lock. + + +Wall-clock mode +--------------- + +Wall-clock mode (``--mode=wall``) captures all samples regardless of what the +thread is doing. This is the default mode and provides a complete picture of +where time passes during program execution:: + + python -m profiling.sampling run --mode=wall script.py + +In wall-clock mode, samples are recorded whether the thread is actively +executing Python code, waiting for I/O, blocked on a lock, or sleeping. +This makes wall-clock profiling ideal for understanding the overall time +distribution in your program, including time spent waiting. + +If your program spends significant time in I/O operations, network calls, or +sleep, wall-clock mode will show these waits as time attributed to the calling +function. This is often exactly what you want when optimizing end-to-end +latency. + + +CPU mode +-------- + +CPU mode (``--mode=cpu``) records samples only when the thread is actually +executing on a CPU core:: + + python -m profiling.sampling run --mode=cpu script.py + +Samples taken while the thread is sleeping, blocked on I/O, or waiting for +a lock are discarded. The resulting profile shows where CPU cycles are consumed, +filtering out idle time. + +CPU mode is useful when you want to focus on computational hotspots without +being distracted by I/O waits. If your program alternates between computation +and network calls, CPU mode reveals which computational sections are most +expensive. + + +Comparing wall-clock and CPU profiles +------------------------------------- + +Running both wall-clock and CPU mode profiles can reveal whether a function's +time is spent computing or waiting. + +If a function appears prominently in both profiles, it is a true computational +hotspot---actively using the CPU. Optimization should focus on algorithmic +improvements or more efficient code. + +If a function is high in wall-clock mode but low or absent in CPU mode, it is +I/O-bound or waiting. The function spends most of its time waiting for network, +disk, locks, or sleep. CPU optimization won't help here; consider async I/O, +connection pooling, or reducing wait time instead. + + +GIL mode +-------- + +GIL mode (``--mode=gil``) records samples only when the thread holds Python's +global interpreter lock:: + + python -m profiling.sampling run --mode=gil script.py + +The GIL is held only while executing Python bytecode. When Python calls into +C extensions, performs I/O operations, or executes native code, the GIL is +typically released. This means GIL mode effectively measures time spent +running Python code specifically, filtering out time in native libraries. + +In multi-threaded programs, GIL mode reveals which code is preventing other +threads from running Python bytecode. Since only one thread can hold the GIL +at a time, functions that appear frequently in GIL mode profiles are +monopolizing the interpreter. + +GIL mode helps answer questions like "which functions are monopolizing the +GIL?" and "why are my other threads starving?" It can also be useful in +single-threaded programs to distinguish Python execution time from time spent +in C extensions or I/O. + + +Output formats +============== + +The profiler produces output in several formats, each suited to different +analysis workflows. The format is selected with a command-line flag, and +output goes to stdout, a file, or a directory depending on the format. + + +pstats format +------------- + +The pstats format (``--pstats``) produces a text table similar to what +deterministic profilers generate. This is the default output format:: + + python -m profiling.sampling run script.py + python -m profiling.sampling run --pstats script.py + +Output appears on stdout by default:: + + Profile Stats (Mode: wall): + nsamples sample% tottime (ms) cumul% cumtime (ms) filename:lineno(function) + 234/892 11.7% 234.00 44.6% 892.00 server.py:145(handle_request) + 156/156 7.8% 156.00 7.8% 156.00 :0(socket.recv) + 98/421 4.9% 98.00 21.1% 421.00 parser.py:67(parse_message) + +The columns show sampling counts and estimated times: + +- **nsamples**: Displayed as ``direct/cumulative`` (for example, ``10/50``). + Direct samples are when the function was at the top of the stack, actively + executing. Cumulative samples are when the function appeared anywhere on the + stack, including when it was waiting for functions it called. If a function + shows ``10/50``, it was directly executing in 10 samples and was on the call + stack in 50 samples total. + +- **sample%** and **cumul%**: Percentages of total samples for direct and + cumulative counts respectively. + +- **tottime** and **cumtime**: Estimated wall-clock time based on sample counts + and the profiling duration. Time units are selected automatically based on + the magnitude: seconds for large values, milliseconds for moderate values, + or microseconds for small values. + +The output includes a legend explaining each column and a summary of +interesting functions that highlights: + +- **Hot spots**: Functions with high direct/cumulative sample ratio (ratio + close to 1.0). These functions spend most of their time executing their own + code rather than waiting for callees. High ratios indicate where CPU time + is actually consumed. + +- **Indirect calls**: Functions with large differences between cumulative and + direct samples. These are orchestration functions that delegate work to + other functions. They appear frequently on the stack but rarely at the top. + +- **Call magnification**: Functions where cumulative samples far exceed direct + samples (high cumulative/direct multiplier). These are frequently-nested + functions that appear deep in many call chains. + +Use ``--no-summary`` to suppress both the legend and summary sections. + +To save pstats output to a file instead of stdout:: + + python -m profiling.sampling run -o profile.txt script.py + +The pstats format supports several options for controlling the display. +The ``--sort`` option determines the column used for ordering results:: + + python -m profiling.sampling run --sort=tottime script.py + python -m profiling.sampling run --sort=cumtime script.py + python -m profiling.sampling run --sort=nsamples script.py + +The ``--limit`` option restricts output to the top N entries:: + + python -m profiling.sampling run --limit=30 script.py + +The ``--no-summary`` option suppresses the header summary that precedes the +statistics table. + + +Collapsed stacks format +----------------------- + +Collapsed stacks format (``--collapsed``) produces one line per unique call +stack, with a count of how many times that stack was sampled:: + + python -m profiling.sampling run --collapsed script.py + +The output looks like: + +.. code-block:: text + + main;process_data;parse_json;decode_utf8 42 + main;process_data;parse_json 156 + main;handle_request;send_response 89 + +Each line contains semicolon-separated function names representing the call +stack from bottom to top, followed by a space and the sample count. This +format is designed for compatibility with external flame graph tools, +particularly Brendan Gregg's ``flamegraph.pl`` script. + +To generate a flame graph from collapsed stacks:: + + python -m profiling.sampling run --collapsed script.py > stacks.txt + flamegraph.pl stacks.txt > profile.svg + +The resulting SVG can be viewed in any web browser and provides an interactive +visualization where you can click to zoom into specific call paths. + + +Flame graph format +------------------ + +Flame graph format (``--flamegraph``) produces a self-contained HTML file with +an interactive flame graph visualization:: + + python -m profiling.sampling run --flamegraph script.py + python -m profiling.sampling run --flamegraph -o profile.html script.py + +If no output file is specified, the profiler generates a filename based on +the process ID (for example, ``flamegraph.12345.html``). + +The generated HTML file requires no external dependencies and can be opened +directly in a web browser. The visualization displays call stacks as nested +rectangles, with width proportional to time spent. Hovering over a rectangle +shows details about that function including source code context, and clicking +zooms into that portion of the call tree. + +The flame graph interface includes: + +- A sidebar showing profile summary, thread statistics, sampling efficiency + metrics (see :ref:`sampling-efficiency`), and top hotspot functions +- Search functionality supporting both function name matching and + ``file.py:42`` line patterns +- Per-thread filtering via dropdown +- Dark/light theme toggle (preference saved across sessions) +- SVG export for saving the current view + +The thread statistics section shows runtime behavior metrics: + +- **GIL Held**: percentage of samples where a thread held the global interpreter + lock (actively running Python code) +- **GIL Released**: percentage of samples where no thread held the GIL +- **Waiting GIL**: percentage of samples where a thread was waiting to acquire + the GIL +- **GC**: percentage of samples during garbage collection + +These statistics help identify GIL contention and understand how time is +distributed between Python execution, native code, and waiting. + +Flame graphs are particularly effective for identifying deep call stacks and +understanding the hierarchical structure of time consumption. Wide rectangles +at the top indicate functions that consume significant time either directly +or through their callees. + + +Gecko format +------------ + +Gecko format (``--gecko``) produces JSON output compatible with the Firefox +Profiler:: + + python -m profiling.sampling run --gecko script.py + python -m profiling.sampling run --gecko -o profile.json script.py + +The `Firefox Profiler `__ is a sophisticated +web-based tool originally built for profiling Firefox itself. It provides +features beyond basic flame graphs, including a timeline view, call tree +exploration, and marker visualization. See the +`Firefox Profiler documentation `__ for +detailed usage instructions. + +To use the output, open the Firefox Profiler in your browser and load the +JSON file. The profiler runs entirely client-side, so your profiling data +never leaves your machine. + +Gecko format automatically collects additional metadata about GIL state and +CPU activity, enabling analysis features specific to Python's threading model. +The profiler emits interval markers that appear as colored bands in the +Firefox Profiler timeline: + +- **GIL markers**: show when threads hold or release the global interpreter lock +- **CPU markers**: show when threads are executing on CPU versus idle +- **Code type markers**: distinguish Python code from native (C extension) code +- **GC markers**: indicate garbage collection activity + +For this reason, the ``--mode`` option is not available with Gecko format; +all relevant data is captured automatically. + + +Heatmap format +-------------- + +Heatmap format (``--heatmap``) generates an interactive HTML visualization +showing sample counts at the source line level:: + + python -m profiling.sampling run --heatmap script.py + python -m profiling.sampling run --heatmap -o my_heatmap script.py + +Unlike other formats that produce a single file, heatmap output creates a +directory containing HTML files for each profiled source file. If no output +path is specified, the directory is named ``heatmap_PID``. + +The heatmap visualization displays your source code with a color gradient +indicating how many samples were collected at each line. Hot lines (many +samples) appear in warm colors, while cold lines (few or no samples) appear +in cool colors. This view helps pinpoint exactly which lines of code are +responsible for time consumption. + +The heatmap interface provides several interactive features: + +- **Coloring modes**: toggle between "Self Time" (direct execution) and + "Total Time" (cumulative, including time in called functions) +- **Cold code filtering**: show all lines or only lines with samples +- **Call graph navigation**: each line shows navigation buttons (▲ for callers, + ▼ for callees) that let you trace execution paths through your code. When + multiple functions called or were called from a line, a menu appears showing + all options with their sample counts. +- **Scroll minimap**: a vertical overview showing the heat distribution across + the entire file +- **Hierarchical index**: files organized by type (stdlib, site-packages, + project) with aggregate sample counts per folder +- **Dark/light theme**: toggle with preference saved across sessions +- **Line linking**: click line numbers to create shareable URLs + +Heatmaps are especially useful when you know which file contains a performance +issue but need to identify the specific lines. Many developers prefer this +format because it maps directly to their source code, making it easy to read +and navigate. For smaller scripts and focused analysis, heatmaps provide an +intuitive view that shows exactly where time is spent without requiring +interpretation of hierarchical visualizations. + + +Live mode +========= + +Live mode (``--live``) provides a terminal-based real-time view of profiling +data, similar to the ``top`` command for system processes:: + + python -m profiling.sampling run --live script.py + python -m profiling.sampling attach --live 12345 + +The display updates continuously as new samples arrive, showing the current +hottest functions. This mode requires the :mod:`curses` module, which is +available on Unix-like systems but not on Windows. The terminal must be at +least 60 columns wide and 12 lines tall; larger terminals display more columns. + +The header displays the top 3 hottest functions, sampling efficiency metrics, +and thread status statistics (GIL held percentage, CPU usage, GC time). The +main table shows function statistics with the currently sorted column indicated +by an arrow (▼). + + +Keyboard commands +----------------- + +Within live mode, keyboard commands control the display: + +:kbd:`q` + Quit the profiler and return to the shell. + +:kbd:`s` / :kbd:`S` + Cycle through sort orders forward/backward (sample count, percentage, + total time, cumulative percentage, cumulative time). + +:kbd:`p` + Pause or resume display updates. Sampling continues in the background + while the display is paused, so you can freeze the view to examine results + without stopping data collection. + +:kbd:`r` + Reset all statistics and start fresh. This is disabled after profiling + finishes to prevent accidental data loss. + +:kbd:`/` + Enter filter mode to search for functions by name. The filter uses + case-insensitive substring matching against the filename and function name. + Type a pattern and press Enter to apply, or Escape to cancel. Glob patterns + and regular expressions are not supported. + +:kbd:`c` + Clear the current filter and show all functions again. + +:kbd:`t` + Toggle between viewing all threads combined or per-thread statistics. + In per-thread mode, a thread counter (for example, ``1/4``) appears showing + your position among the available threads. + +:kbd:`←` :kbd:`→` or :kbd:`↑` :kbd:`↓` + In per-thread view, navigate between threads. Navigation wraps around + from the last thread to the first and vice versa. + +:kbd:`+` / :kbd:`-` + Increase or decrease the display refresh rate. The range is 0.05 seconds + (20 Hz, very responsive) to 1.0 second (1 Hz, lower overhead). Faster refresh + rates use more CPU. The default is 0.1 seconds (10 Hz). + +:kbd:`x` + Toggle trend indicators that show whether functions are becoming hotter + or cooler over time. When enabled, increasing metrics appear in green and + decreasing metrics appear in red, comparing each update to the previous one. + +:kbd:`h` or :kbd:`?` + Show the help screen with all available commands. + +When profiling finishes (duration expires or target process exits), the display +shows a "PROFILING COMPLETE" banner and freezes the final results. You can +still navigate, sort, and filter the results before pressing :kbd:`q` to exit. + +Live mode is incompatible with output format options (``--collapsed``, +``--flamegraph``, and so on) because it uses an interactive terminal +interface rather than producing file output. + + +Async-aware profiling +===================== + +For programs using :mod:`asyncio`, the profiler offers async-aware mode +(``--async-aware``) that reconstructs call stacks based on the task structure +rather than the raw Python frames:: + + python -m profiling.sampling run --async-aware async_script.py + +Standard profiling of async code can be confusing because the physical call +stack often shows event loop internals rather than the logical flow of your +coroutines. Async-aware mode addresses this by tracking which task is running +and presenting stacks that reflect the ``await`` chain. + +.. note:: + + Async-aware profiling requires the target process to have the :mod:`asyncio` + module loaded. If you profile a script before it imports asyncio, async-aware + mode will not be able to capture task information. + + +Async modes +----------- + +The ``--async-mode`` option controls which tasks appear in the profile:: + + python -m profiling.sampling run --async-aware --async-mode=running async_script.py + python -m profiling.sampling run --async-aware --async-mode=all async_script.py + +With ``--async-mode=running`` (the default), only the task currently executing +on the CPU is profiled. This shows where your program is actively spending time +and is the typical choice for performance analysis. + +With ``--async-mode=all``, tasks that are suspended (awaiting I/O, locks, or +other tasks) are also included. This mode is useful for understanding what your +program is waiting on, but produces larger profiles since every suspended task +appears in each sample. + + +Task markers and stack reconstruction +------------------------------------- + +In async-aware profiles, you will see ```` frames that mark boundaries +between asyncio tasks. These are synthetic frames inserted by the profiler to +show the task structure. The task name appears as the function name in these +frames. + +When a task awaits another task, the profiler reconstructs the logical call +chain by following the ``await`` relationships. Only "leaf" tasks (tasks that +no other task is currently awaiting) generate their own stack entries. Tasks +being awaited by other tasks appear as part of their awaiter's stack instead. + +If a task has multiple awaiters (a diamond pattern in the task graph), the +profiler deterministically selects one parent and annotates the task marker +with the number of parents, for example ``MyTask (2 parents)``. This indicates +that alternate execution paths exist but are not shown in this particular stack. + + +Option restrictions +------------------- + +Async-aware mode uses a different stack reconstruction mechanism and is +incompatible with: ``--native``, ``--no-gc``, ``--all-threads``, and +``--mode=cpu`` or ``--mode=gil``. + + +Command-line interface +====================== + +.. program:: profiling.sampling + +The complete command-line interface for reference. + + +Global options +-------------- + +.. option:: run + + Run and profile a Python script or module. + +.. option:: attach + + Attach to and profile a running process by PID. + + +Sampling options +---------------- + +.. option:: -i , --interval + + Sampling interval in microseconds. Default: 100. + +.. option:: -d , --duration + + Profiling duration in seconds. Default: 10. + +.. option:: -a, --all-threads + + Sample all threads, not just the main thread. + +.. option:: --realtime-stats + + Display sampling statistics during profiling. + +.. option:: --native + + Include ```` frames for non-Python code. + +.. option:: --no-gc + + Exclude ```` frames for garbage collection. + +.. option:: --async-aware + + Enable async-aware profiling for asyncio programs. + + +Mode options +------------ + +.. option:: --mode + + Sampling mode: ``wall`` (default), ``cpu``, or ``gil``. + The ``cpu`` and ``gil`` modes are incompatible with ``--async-aware``. + +.. option:: --async-mode + + Async profiling mode: ``running`` (default) or ``all``. + Requires ``--async-aware``. + + +Output options +-------------- + +.. option:: --pstats + + Generate text statistics output. This is the default. + +.. option:: --collapsed + + Generate collapsed stack format for external flame graph tools. + +.. option:: --flamegraph + + Generate self-contained HTML flame graph. + +.. option:: --gecko + + Generate Gecko JSON format for Firefox Profiler. + +.. option:: --heatmap + + Generate HTML heatmap with line-level sample counts. + +.. option:: -o , --output + + Output file or directory path. Default behavior varies by format: + ``--pstats`` writes to stdout, ``--flamegraph`` and ``--gecko`` generate + files like ``flamegraph.PID.html``, and ``--heatmap`` creates a directory + named ``heatmap_PID``. + + +pstats display options +---------------------- + +These options apply only to pstats format output. + +.. option:: --sort + + Sort order: ``nsamples``, ``tottime``, ``cumtime``, ``sample-pct``, + ``cumul-pct``, ``nsamples-cumul``, or ``name``. Default: ``nsamples``. + +.. option:: -l , --limit + + Maximum number of entries to display. Default: 15. + +.. option:: --no-summary + + Omit the Legend and Summary of Interesting Functions sections from output. + + +Run command options +------------------- + +.. option:: -m, --module + + Treat the target as a module name rather than a script path. + +.. option:: --live + + Start interactive terminal interface instead of batch profiling. + + +.. seealso:: + + :mod:`profiling` + Overview of Python profiling tools and guidance on choosing a profiler. + + :mod:`profiling.tracing` + Deterministic tracing profiler for exact call counts and timing. + + :mod:`pstats` + Statistics analysis for profile data. + + `Firefox Profiler `__ + Web-based profiler that accepts Gecko format output. See the + `documentation `__ for usage details. + + `FlameGraph `__ + Tools for generating flame graphs from collapsed stack format. diff --git a/Doc/library/profiling.tracing.rst b/Doc/library/profiling.tracing.rst new file mode 100644 index 00000000000000..6e6ba9173a1d2f --- /dev/null +++ b/Doc/library/profiling.tracing.rst @@ -0,0 +1,331 @@ +.. _profiling-tracing: + +**************************************************** +:mod:`profiling.tracing` --- Deterministic profiler +**************************************************** + +.. module:: profiling.tracing + :synopsis: Deterministic tracing profiler for Python programs. + +.. module:: cProfile + :synopsis: Alias for profiling.tracing (backward compatibility). + :noindex: + +.. versionadded:: 3.15 + +**Source code:** :source:`Lib/profiling/tracing/` + +-------------- + +The :mod:`profiling.tracing` module provides deterministic profiling of Python +programs. It monitors every function call, function return, and exception event, +recording precise timing for each. This approach provides exact call counts and +complete visibility into program execution, making it ideal for development and +testing scenarios. + +.. note:: + + This module is also available as ``cProfile`` for backward compatibility. + The ``cProfile`` name will continue to work in all future Python versions. + Use whichever import style suits your codebase:: + + # Preferred (new style) + import profiling.tracing + profiling.tracing.run('my_function()') + + # Also works (backward compatible) + import cProfile + cProfile.run('my_function()') + + +What is deterministic profiling? +================================ + +:dfn:`Deterministic profiling` captures every function call, function return, +and exception event during program execution. The profiler measures the precise +time intervals between these events, providing exact statistics about how the +program behaves. + +In contrast to :ref:`statistical profiling `, which samples +the call stack periodically to estimate where time is spent, deterministic +profiling records every event. This means you get exact call counts rather than +statistical approximations. The trade-off is that instrumenting every event +introduces overhead that can slow down program execution. + +Python's interpreted nature makes deterministic profiling practical. The +interpreter already dispatches events for function calls and returns, so the +profiler can hook into this mechanism without requiring code modification. The +overhead tends to be moderate relative to the inherent cost of interpretation, +making deterministic profiling suitable for most development workflows. + +Deterministic profiling helps answer questions like: + +- How many times was this function called? +- What is the complete call graph of my program? +- Which functions are called by a particular function? +- Are there unexpected function calls happening? + +Call count statistics can identify bugs (surprising counts) and inline +expansion opportunities (high call counts). Internal time statistics reveal +"hot loops" that warrant optimization. Cumulative time statistics help identify +algorithmic inefficiencies. The handling of cumulative times in this profiler +allows direct comparison of recursive and iterative implementations. + + +.. _profiling-tracing-cli: + +Command-line interface +====================== + +.. program:: profiling.tracing + +The :mod:`profiling.tracing` module can be invoked as a script to profile +another script or module: + +.. code-block:: shell-session + + python -m profiling.tracing [-o output_file] [-s sort_order] (-m module | script.py) + +This runs the specified script or module under the profiler and prints the +results to standard output (or saves them to a file). + +.. option:: -o + + Write the profile results to a file instead of standard output. The output + file can be read by the :mod:`pstats` module for later analysis. + +.. option:: -s + + Sort the output by the specified key. This accepts any of the sort keys + recognized by :meth:`pstats.Stats.sort_stats`, such as ``cumulative``, + ``time``, ``calls``, or ``name``. This option only applies when + :option:`-o ` is not specified. + +.. option:: -m + + Profile a module instead of a script. The module is located using the + standard import mechanism. + + .. versionadded:: 3.7 + The ``-m`` option for ``cProfile``. + + .. versionadded:: 3.8 + The ``-m`` option for :mod:`profile`. + + +Programmatic usage examples +=========================== + +For more control over profiling, use the module's functions and classes +directly. + + +Basic profiling +--------------- + +The simplest approach uses the :func:`!run` function:: + + import profiling.tracing + profiling.tracing.run('my_function()') + +This profiles the given code string and prints a summary to standard output. +To save results for later analysis:: + + profiling.tracing.run('my_function()', 'output.prof') + + +Using the :class:`!Profile` class +--------------------------------- + +The :class:`!Profile` class provides fine-grained control:: + + import profiling.tracing + import pstats + from io import StringIO + + pr = profiling.tracing.Profile() + pr.enable() + # ... code to profile ... + pr.disable() + + # Print results + s = StringIO() + ps = pstats.Stats(pr, stream=s).sort_stats(pstats.SortKey.CUMULATIVE) + ps.print_stats() + print(s.getvalue()) + +The :class:`!Profile` class also works as a context manager:: + + import profiling.tracing + + with profiling.tracing.Profile() as pr: + # ... code to profile ... + + pr.print_stats() + + +Module reference +================ + +.. currentmodule:: profiling.tracing + +.. function:: run(command, filename=None, sort=-1) + + Profile execution of a command and print or save the results. + + This function executes the *command* string using :func:`exec` in the + ``__main__`` module's namespace:: + + exec(command, __main__.__dict__, __main__.__dict__) + + If *filename* is not provided, the function creates a :class:`pstats.Stats` + instance and prints a summary to standard output. If *filename* is + provided, the raw profile data is saved to that file for later analysis + with :mod:`pstats`. + + The *sort* argument specifies the sort order for printed output, accepting + any value recognized by :meth:`pstats.Stats.sort_stats`. + + +.. function:: runctx(command, globals, locals, filename=None, sort=-1) + + Profile execution of a command with explicit namespaces. + + Like :func:`run`, but executes the command with the specified *globals* + and *locals* mappings instead of using the ``__main__`` module's namespace:: + + exec(command, globals, locals) + + +.. class:: Profile(timer=None, timeunit=0.0, subcalls=True, builtins=True) + + A profiler object that collects execution statistics. + + The optional *timer* argument specifies a custom timing function. If not + provided, the profiler uses a platform-appropriate default timer. When + supplying a custom timer, it must return a single number representing the + current time. If the timer returns integers, use *timeunit* to specify the + duration of one time unit (for example, ``0.001`` for milliseconds). + + The *subcalls* argument controls whether the profiler tracks call + relationships between functions. The *builtins* argument controls whether + built-in functions are profiled. + + .. versionchanged:: 3.8 + Added context manager support. + + .. method:: enable() + + Start collecting profiling data. + + .. method:: disable() + + Stop collecting profiling data. + + .. method:: create_stats() + + Stop collecting data and record the results internally as the current + profile. + + .. method:: print_stats(sort=-1) + + Create a :class:`pstats.Stats` object from the current profile and print + the results to standard output. + + The *sort* argument specifies the sorting order. It accepts a single + key or a tuple of keys for multi-level sorting, using the same values + as :meth:`pstats.Stats.sort_stats`. + + .. versionadded:: 3.13 + Support for a tuple of sort keys. + + .. method:: dump_stats(filename) + + Write the current profile data to *filename*. The file can be read by + :class:`pstats.Stats` for later analysis. + + .. method:: run(cmd) + + Profile the command string via :func:`exec`. + + .. method:: runctx(cmd, globals, locals) + + Profile the command string via :func:`exec` with the specified + namespaces. + + .. method:: runcall(func, /, *args, **kwargs) + + Profile a function call. Returns whatever *func* returns:: + + result = pr.runcall(my_function, arg1, arg2, keyword=value) + +.. note:: + + Profiling requires that the profiled code returns normally. If the + interpreter terminates (for example, via :func:`sys.exit`) during + profiling, no results will be available. + + +Using a custom timer +==================== + +The :class:`Profile` class accepts a custom timing function, allowing you to +measure different aspects of execution such as wall-clock time or CPU time. +Pass the timing function to the constructor:: + + pr = profiling.tracing.Profile(my_timer_function) + +The timer function must return a single number representing the current time. +If it returns integers, also specify *timeunit* to indicate the duration of +one unit:: + + # Timer returns time in milliseconds + pr = profiling.tracing.Profile(my_ms_timer, 0.001) + +For best performance, the timer function should be as fast as possible. The +profiler calls it frequently, so timer overhead directly affects profiling +overhead. + +The :mod:`time` module provides several functions suitable for use as custom +timers: + +- :func:`time.perf_counter` for high-resolution wall-clock time +- :func:`time.process_time` for CPU time (excluding sleep) +- :func:`time.monotonic` for monotonic clock time + + +Limitations +=========== + +Deterministic profiling has inherent limitations related to timing accuracy. + +The underlying timer typically has a resolution of about one millisecond. +Measurements cannot be more accurate than this resolution. With enough +measurements, timing errors tend to average out, but individual measurements +may be imprecise. + +There is also latency between when an event occurs and when the profiler +captures the timestamp. Similarly, there is latency after reading the +timestamp before user code resumes. Functions called frequently accumulate +this latency, which can make them appear slower than they actually are. This +error is typically less than one clock tick per call but can become +significant for functions called many times. + +The :mod:`profiling.tracing` module (and its ``cProfile`` alias) is +implemented as a C extension with low overhead, so these timing issues are +less pronounced than with the deprecated pure Python :mod:`profile` module. + + +.. seealso:: + + :mod:`profiling` + Overview of Python profiling tools and guidance on choosing a profiler. + + :mod:`profiling.sampling` + Statistical sampling profiler for production use. + + :mod:`pstats` + Statistics analysis and formatting for profile data. + + :mod:`profile` + Deprecated pure Python profiler (includes calibration documentation). diff --git a/Doc/library/pstats.rst b/Doc/library/pstats.rst new file mode 100644 index 00000000000000..ce1cc5c9535ca6 --- /dev/null +++ b/Doc/library/pstats.rst @@ -0,0 +1,362 @@ +.. _pstats-module: + +******************************************** +:mod:`pstats` --- Statistics for profilers +******************************************** + +.. module:: pstats + :synopsis: Statistics object for analyzing profiler output. + +**Source code:** :source:`Lib/pstats.py` + +-------------- + +The :mod:`pstats` module provides tools for reading, manipulating, and +displaying profiling statistics generated by Python's profilers. It reads +output from both :mod:`profiling.tracing` (deterministic profiler) and +:mod:`profiling.sampling` (statistical profiler). + + +Reading and displaying profile data +=================================== + +The :class:`Stats` class is the primary interface for working with profile +data. It can read statistics from files or directly from a +:class:`~profiling.tracing.Profile` object. + +Load statistics from a file and print a basic report:: + + import pstats + + p = pstats.Stats('profile_output.prof') + p.print_stats() + +The :class:`Stats` object provides methods for sorting and filtering the +data before printing. For example, to see the ten functions with the highest +cumulative time:: + + from pstats import SortKey + + p = pstats.Stats('profile_output.prof') + p.sort_stats(SortKey.CUMULATIVE).print_stats(10) + + +Working with statistics +----------------------- + +The :class:`Stats` class supports method chaining, making it convenient to +perform multiple operations:: + + p = pstats.Stats('restats') + p.strip_dirs().sort_stats(-1).print_stats() + +The :meth:`~Stats.strip_dirs` method removes directory paths from filenames, +making the output more compact. The :meth:`~Stats.sort_stats` method accepts +various keys to control the sort order. + +Different sort keys highlight different aspects of performance:: + + from pstats import SortKey + + # Functions that consume the most cumulative time + p.sort_stats(SortKey.CUMULATIVE).print_stats(10) + + # Functions that consume the most time in their own code + p.sort_stats(SortKey.TIME).print_stats(10) + + # Functions sorted by name + p.sort_stats(SortKey.NAME).print_stats() + + +Filtering output +---------------- + +The :meth:`~Stats.print_stats` method accepts restrictions that filter +which functions are displayed. Restrictions can be integers (limiting the +count), floats between 0 and 1 (selecting a percentage), or strings (matching +function names via regular expression). + +Print only the top 10%:: + + p.print_stats(.1) + +Print only functions whose names contain "init":: + + p.print_stats('init') + +Combine restrictions (they apply sequentially):: + + # Top 10%, then only those containing "init" + p.print_stats(.1, 'init') + + # Functions in files matching "foo:", limited to top 50% + p.sort_stats(SortKey.FILENAME).print_stats('foo:', .5) + + +Analyzing call relationships +---------------------------- + +The :meth:`~Stats.print_callers` method shows which functions called each +displayed function:: + + p.print_callers() + +The :meth:`~Stats.print_callees` method shows the opposite relationship, +listing which functions each displayed function called:: + + p.print_callees() + +Both methods accept the same restriction arguments as :meth:`~Stats.print_stats`. + + +Combining multiple profiles +--------------------------- + +Statistics from multiple profiling runs can be combined into a single +:class:`Stats` object:: + + # Load multiple files at once + p = pstats.Stats('run1.prof', 'run2.prof', 'run3.prof') + + # Or add files incrementally + p = pstats.Stats('run1.prof') + p.add('run2.prof') + p.add('run3.prof') + +When files are combined, statistics for identical functions (same file, line, +and name) are accumulated, giving an aggregate view across all profiling runs. + + +The :class:`!Stats` class +========================= + +.. class:: Stats(*filenames_or_profile, stream=sys.stdout) + + Create a statistics object from profile data. + + The arguments can be filenames (strings or path-like objects) or + :class:`~profiling.tracing.Profile` objects. If multiple sources are + provided, their statistics are combined. + + The *stream* argument specifies where output from :meth:`print_stats` and + related methods is written. It defaults to :data:`sys.stdout`. + + The profile data format is specific to the Python version that created it. + There is no compatibility guarantee between Python versions or between + different profilers. + + .. method:: strip_dirs() + + Remove leading path information from all filenames. + + This method modifies the object in place and returns it for method + chaining. After stripping, the statistics are considered to be in + random order. + + If stripping causes two functions to become indistinguishable (same + filename, line number, and function name), their statistics are + combined into a single entry. + + .. method:: add(*filenames) + + Add profiling data from additional files. + + The files must have been created by the same profiler type. Statistics + for identical functions are accumulated. + + .. method:: dump_stats(filename) + + Save the current statistics to a file. + + The file is created if it does not exist and overwritten if it does. + The saved data can be loaded by creating a new :class:`Stats` object. + + .. method:: sort_stats(*keys) + + Sort the statistics according to the specified criteria. + + Each key can be a string or a :class:`SortKey` enum member. When + multiple keys are provided, later keys break ties in earlier keys. + + Using :class:`SortKey` enum members is preferred over strings as it + provides better error checking:: + + from pstats import SortKey + p.sort_stats(SortKey.CUMULATIVE) + + Valid sort keys: + + +------------------+------------------------+----------------------+ + | String | Enum | Meaning | + +==================+========================+======================+ + | ``'calls'`` | ``SortKey.CALLS`` | call count | + +------------------+------------------------+----------------------+ + | ``'cumulative'`` | ``SortKey.CUMULATIVE`` | cumulative time | + +------------------+------------------------+----------------------+ + | ``'cumtime'`` | N/A | cumulative time | + +------------------+------------------------+----------------------+ + | ``'file'`` | N/A | file name | + +------------------+------------------------+----------------------+ + | ``'filename'`` | ``SortKey.FILENAME`` | file name | + +------------------+------------------------+----------------------+ + | ``'module'`` | N/A | file name | + +------------------+------------------------+----------------------+ + | ``'ncalls'`` | N/A | call count | + +------------------+------------------------+----------------------+ + | ``'pcalls'`` | ``SortKey.PCALLS`` | primitive call count | + +------------------+------------------------+----------------------+ + | ``'line'`` | ``SortKey.LINE`` | line number | + +------------------+------------------------+----------------------+ + | ``'name'`` | ``SortKey.NAME`` | function name | + +------------------+------------------------+----------------------+ + | ``'nfl'`` | ``SortKey.NFL`` | name/file/line | + +------------------+------------------------+----------------------+ + | ``'stdname'`` | ``SortKey.STDNAME`` | standard name | + +------------------+------------------------+----------------------+ + | ``'time'`` | ``SortKey.TIME`` | internal time | + +------------------+------------------------+----------------------+ + | ``'tottime'`` | N/A | internal time | + +------------------+------------------------+----------------------+ + + All sorts on statistics are in descending order (most time consuming + first), while name, file, and line number sorts are ascending + (alphabetical). + + The difference between ``SortKey.NFL`` and ``SortKey.STDNAME`` is that + NFL sorts line numbers numerically while STDNAME sorts them as strings. + ``sort_stats(SortKey.NFL)`` is equivalent to + ``sort_stats(SortKey.NAME, SortKey.FILENAME, SortKey.LINE)``. + + For backward compatibility, the numeric arguments ``-1``, ``0``, ``1``, + and ``2`` are also accepted, meaning ``'stdname'``, ``'calls'``, + ``'time'``, and ``'cumulative'`` respectively. + + .. versionadded:: 3.7 + The :class:`SortKey` enum. + + .. method:: reverse_order() + + Reverse the current sort order. + + By default, the sort direction is chosen appropriately for the sort key + (descending for time-based keys, ascending for name-based keys). This + method inverts that choice. + + .. method:: print_stats(*restrictions) + + Print a report of the profiling statistics. + + The output includes a header line summarizing the data, followed by a + table of function statistics sorted according to the last + :meth:`sort_stats` call. + + Restrictions filter the output. Each restriction is either: + + - An integer: limits output to that many entries + - A float between 0.0 and 1.0: selects that fraction of entries + - A string: matches function names via regular expression + + Restrictions are applied sequentially. For example:: + + print_stats(.1, 'foo:') + + First limits to the top 10%, then filters to functions matching 'foo:'. + + .. method:: print_callers(*restrictions) + + Print the callers of each function in the statistics. + + For each function in the filtered results, shows which functions called + it and how often. + + With :mod:`profiling.tracing` (or ``cProfile``), each caller line + shows three numbers: the number of calls from that caller, and the + total and cumulative times for those specific calls. + + Accepts the same restriction arguments as :meth:`print_stats`. + + .. method:: print_callees(*restrictions) + + Print the functions called by each function in the statistics. + + This is the inverse of :meth:`print_callers`, showing which functions + each listed function called. + + Accepts the same restriction arguments as :meth:`print_stats`. + + .. method:: get_stats_profile() + + Return a ``StatsProfile`` object containing the statistics. + + The returned object provides programmatic access to the profile data, + with function names mapped to ``FunctionProfile`` objects + containing timing and call count information. + + .. versionadded:: 3.9 + + +.. class:: SortKey + + An enumeration of valid sort keys for :meth:`Stats.sort_stats`. + + .. attribute:: CALLS + + Sort by call count. + + .. attribute:: CUMULATIVE + + Sort by cumulative time. + + .. attribute:: FILENAME + + Sort by file name. + + .. attribute:: LINE + + Sort by line number. + + .. attribute:: NAME + + Sort by function name. + + .. attribute:: NFL + + Sort by name, then file, then line number (numeric line sort). + + .. attribute:: PCALLS + + Sort by primitive (non-recursive) call count. + + .. attribute:: STDNAME + + Sort by standard name (string-based line sort). + + .. attribute:: TIME + + Sort by internal time (time in function excluding subcalls). + + +.. _pstats-cli: + +Command-line interface +====================== + +The :mod:`pstats` module can be invoked as a script to interactively browse +profile data:: + + python -m pstats profile_output.prof + +This opens a line-oriented interface (built on :mod:`cmd`) for examining the +statistics. Type ``help`` at the prompt for available commands. + + +.. seealso:: + + :mod:`profiling` + Overview of Python profiling tools. + + :mod:`profiling.tracing` + Deterministic tracing profiler. + + :mod:`profiling.sampling` + Statistical sampling profiler. diff --git a/Doc/library/superseded.rst b/Doc/library/superseded.rst index d120c6acf621e3..b6b2f28d067299 100644 --- a/Doc/library/superseded.rst +++ b/Doc/library/superseded.rst @@ -1,7 +1,7 @@ .. _superseded: ****************** -Superseded Modules +Superseded modules ****************** The modules described in this chapter have been superseded by other modules @@ -24,3 +24,4 @@ currently no modules in this latter category. :maxdepth: 1 getopt.rst + profile.rst diff --git a/Doc/library/tachyon-logo.png b/Doc/library/tachyon-logo.png new file mode 100644 index 00000000000000..fddeaafe7e09a5 Binary files /dev/null and b/Doc/library/tachyon-logo.png differ diff --git a/Doc/tools/.nitignore b/Doc/tools/.nitignore index c41c70f0ed3306..54b92f1172e80c 100644 --- a/Doc/tools/.nitignore +++ b/Doc/tools/.nitignore @@ -23,7 +23,6 @@ Doc/library/multiprocessing.rst Doc/library/optparse.rst Doc/library/os.rst Doc/library/pickletools.rst -Doc/library/profile.rst Doc/library/pyexpat.rst Doc/library/select.rst Doc/library/socket.rst diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst index e195d9d462dda9..9b8f36862c1335 100644 --- a/Doc/whatsnew/2.5.rst +++ b/Doc/whatsnew/2.5.rst @@ -1277,12 +1277,12 @@ complete list of changes, or look through the SVN logs for all the details. with the new ':keyword:`with`' statement. See section :ref:`contextlibmod` for more about this module. -* New module: The :mod:`cProfile` module is a C implementation of the existing - :mod:`profile` module that has much lower overhead. The module's interface is - the same as :mod:`profile`: you run ``cProfile.run('main()')`` to profile a +* New module: The :mod:`!cProfile` module is a C implementation of the existing + :mod:`!profile` module that has much lower overhead. The module's interface is + the same as :mod:`!profile`: you run ``cProfile.run('main()')`` to profile a function, can save profile data to a file, etc. It's not yet known if the Hotshot profiler, which is also written in C but doesn't match the - :mod:`profile` module's interface, will continue to be maintained in future + :mod:`!profile` module's interface, will continue to be maintained in future versions of Python. (Contributed by Armin Rigo.) Also, the :mod:`pstats` module for analyzing the data measured by the profiler diff --git a/Doc/whatsnew/3.15.rst b/Doc/whatsnew/3.15.rst index 0c892e63393ad1..853c47d4402f20 100644 --- a/Doc/whatsnew/3.15.rst +++ b/Doc/whatsnew/3.15.rst @@ -67,6 +67,8 @@ Summary -- Release highlights * :pep:`799`: :ref:`A dedicated profiling package for organizing Python profiling tools ` +* :pep:`799`: :ref:`Tachyon: High frequency statistical sampling profiler + profiling tools ` * :pep:`686`: :ref:`Python now uses UTF-8 as the default encoding ` * :pep:`782`: :ref:`A new PyBytesWriter C API to create a Python bytes object @@ -82,14 +84,14 @@ New features :pep:`799`: A dedicated profiling package ----------------------------------------- -A new :mod:`!profiling` module has been added to organize Python's built-in +A new :mod:`profiling` module has been added to organize Python's built-in profiling tools under a single, coherent namespace. This module contains: -* :mod:`!profiling.tracing`: deterministic function-call tracing (relocated from - :mod:`cProfile`). -* :mod:`!profiling.sampling`: a new statistical sampling profiler (named Tachyon). +* :mod:`profiling.tracing`: deterministic function-call tracing (relocated from + ``cProfile``). +* :mod:`profiling.sampling`: a new statistical sampling profiler (named Tachyon). -The :mod:`cProfile` module remains as an alias for backwards compatibility. +The ``cProfile`` module remains as an alias for backwards compatibility. The :mod:`profile` module is deprecated and will be removed in Python 3.17. .. seealso:: :pep:`799` for further details. @@ -102,11 +104,16 @@ The :mod:`profile` module is deprecated and will be removed in Python 3.17. Tachyon: High frequency statistical sampling profiler ----------------------------------------------------- +.. image:: ../library/tachyon-logo.png + :alt: Tachyon profiler logo + :align: center + :width: 200px + A new statistical sampling profiler (Tachyon) has been added as -:mod:`!profiling.sampling`. This profiler enables low-overhead performance analysis of +:mod:`profiling.sampling`. This profiler enables low-overhead performance analysis of running Python processes without requiring code modification or process restart. -Unlike deterministic profilers (:mod:`cProfile` and :mod:`profile`) that instrument +Unlike deterministic profilers (such as :mod:`profiling.tracing`) that instrument every function call, the sampling profiler periodically captures stack traces from running processes. This approach provides virtually zero overhead while achieving sampling rates of **up to 1,000,000 Hz**, making it the fastest sampling profiler @@ -168,6 +175,9 @@ Key features include: (``--async-aware``). See which coroutines are consuming time, with options to show only running tasks or all tasks including those waiting. +See :mod:`profiling.sampling` for the complete documentation, including all +available output formats, profiling modes, and configuration options. + (Contributed by Pablo Galindo and László Kiss Kollár in :gh:`135953` and :gh:`138122`.) diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 9ac3bf53833f93..32ca965b7d0ed0 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -841,7 +841,7 @@ and by Alexander Mohr and Ilya Kulakov in :issue:`29302`.) cProfile -------- -The :mod:`cProfile` command line now accepts ``-m module_name`` as an +The :mod:`!cProfile` command line now accepts ``-m module_name`` as an alternative to script path. (Contributed by Sanyam Khurana in :issue:`21862`.) diff --git a/Lib/profiling/sampling/_assets/tachyon-logo.png b/Lib/profiling/sampling/_assets/tachyon-logo.png new file mode 100644 index 00000000000000..fddeaafe7e09a5 Binary files /dev/null and b/Lib/profiling/sampling/_assets/tachyon-logo.png differ diff --git a/Lib/profiling/sampling/_flamegraph_assets/flamegraph.css b/Lib/profiling/sampling/_flamegraph_assets/flamegraph.css index e1cc3a8a58cf1d..a6d5a3b90416fd 100644 --- a/Lib/profiling/sampling/_flamegraph_assets/flamegraph.css +++ b/Lib/profiling/sampling/_flamegraph_assets/flamegraph.css @@ -241,8 +241,8 @@ body.resizing-sidebar { } .sidebar-logo-img { - width: 90px; - height: 90px; + width: 220px; + height: 180px; display: flex; align-items: center; justify-content: center; diff --git a/Lib/profiling/sampling/_heatmap_assets/heatmap_index_template.html b/Lib/profiling/sampling/_heatmap_assets/heatmap_index_template.html index b71bd94c66166a..98996bdbf5ffb1 100644 --- a/Lib/profiling/sampling/_heatmap_assets/heatmap_index_template.html +++ b/Lib/profiling/sampling/_heatmap_assets/heatmap_index_template.html @@ -11,6 +11,7 @@
+ Tachyon Heatmap Report diff --git a/Lib/profiling/sampling/_heatmap_assets/heatmap_pyfile_template.html b/Lib/profiling/sampling/_heatmap_assets/heatmap_pyfile_template.html index d8b26adfb0243f..fc85b570984b98 100644 --- a/Lib/profiling/sampling/_heatmap_assets/heatmap_pyfile_template.html +++ b/Lib/profiling/sampling/_heatmap_assets/heatmap_pyfile_template.html @@ -11,6 +11,7 @@
+ Tachyon diff --git a/Lib/profiling/sampling/_shared_assets/base.css b/Lib/profiling/sampling/_shared_assets/base.css index 54e3d78f8ebf34..4815dae3feae19 100644 --- a/Lib/profiling/sampling/_shared_assets/base.css +++ b/Lib/profiling/sampling/_shared_assets/base.css @@ -191,16 +191,16 @@ body { display: flex; align-items: center; justify-content: center; - width: 28px; - height: 28px; + width: 48px; + height: 40px; flex-shrink: 0; } /* Style the inlined SVG/img inside brand-logo */ .brand-logo svg, .brand-logo img { - width: 28px; - height: 28px; + width: 48px; + height: 40px; display: block; object-fit: contain; filter: drop-shadow(0 1px 2px rgba(0, 0, 0, 0.2)); diff --git a/Lib/profiling/sampling/heatmap_collector.py b/Lib/profiling/sampling/heatmap_collector.py index 8a8ba9628df573..e1454f0663a439 100644 --- a/Lib/profiling/sampling/heatmap_collector.py +++ b/Lib/profiling/sampling/heatmap_collector.py @@ -204,15 +204,15 @@ def _load_templates(self): self.index_js = f"{shared_js}\n{(assets_dir / 'heatmap_index.js').read_text(encoding='utf-8')}" self.file_js = f"{shared_js}\n{(assets_dir / 'heatmap.js').read_text(encoding='utf-8')}" - # Load Python logo + # Load Tachyon logo logo_dir = template_dir / "_assets" try: - png_path = logo_dir / "python-logo-only.png" + png_path = logo_dir / "tachyon-logo.png" b64_logo = base64.b64encode(png_path.read_bytes()).decode("ascii") - self.logo_html = f'' + self.logo_html = f'' except (FileNotFoundError, IOError) as e: self.logo_html = '
' - print(f"Warning: Could not load Python logo: {e}") + print(f"Warning: Could not load Tachyon logo: {e}") except (FileNotFoundError, IOError) as e: raise RuntimeError(f"Failed to load heatmap template files: {e}") from e @@ -804,6 +804,7 @@ def _generate_file_html(self, output_path: Path, filename: str, "": ''.join(code_lines_html), "": f"", "": f"", + "": self._template_loader.logo_html, } html_content = self._template_loader.file_template diff --git a/Lib/profiling/sampling/stack_collector.py b/Lib/profiling/sampling/stack_collector.py index 1f766682858d45..aa9cdf2468f683 100644 --- a/Lib/profiling/sampling/stack_collector.py +++ b/Lib/profiling/sampling/stack_collector.py @@ -352,11 +352,11 @@ def _create_flamegraph_html(self, data): "", f"" ) - png_path = assets_dir / "python-logo-only.png" + png_path = assets_dir / "tachyon-logo.png" b64_logo = base64.b64encode(png_path.read_bytes()).decode("ascii") # Let CSS control size; keep markup simple - logo_html = f'Python logo' + logo_html = f'Tachyon logo' html_template = html_template.replace("", logo_html) d3_js = d3_path.read_text(encoding="utf-8") diff --git a/Misc/NEWS.d/3.10.0a2.rst b/Misc/NEWS.d/3.10.0a2.rst index 3e82de9ef266d6..263b8cb9762e10 100644 --- a/Misc/NEWS.d/3.10.0a2.rst +++ b/Misc/NEWS.d/3.10.0a2.rst @@ -593,7 +593,7 @@ of treating them as a match. .. nonce: ONk9Na .. section: Library -Fix ``--outfile`` for :mod:`cProfile` / :mod:`profile` not writing the +Fix ``--outfile`` for :mod:`!cProfile` / :mod:`!profile` not writing the output file in the original directory when the program being profiled changes the working directory. PR by Anthony Sottile. diff --git a/Misc/NEWS.d/3.10.0a5.rst b/Misc/NEWS.d/3.10.0a5.rst index a85ea1ff1c2817..d2bac5d13e989b 100644 --- a/Misc/NEWS.d/3.10.0a5.rst +++ b/Misc/NEWS.d/3.10.0a5.rst @@ -426,7 +426,7 @@ specified using a relative path and the current directory changed. .. nonce: Jq6Az- .. section: Library -Fix CLI of :mod:`cProfile` and :mod:`profile` to catch +Fix CLI of :mod:`!cProfile` and :mod:`!profile` to catch :exc:`BrokenPipeError`. .. diff --git a/Misc/NEWS.d/3.12.0b1.rst b/Misc/NEWS.d/3.12.0b1.rst index 3a3870ac9fe621..b867a4f623006f 100644 --- a/Misc/NEWS.d/3.12.0b1.rst +++ b/Misc/NEWS.d/3.12.0b1.rst @@ -1257,7 +1257,7 @@ defined by any build system or documented for manual use. .. nonce: n_AfcS .. section: Library -Update :mod:`cProfile` to use PEP 669 API +Update :mod:`!cProfile` to use PEP 669 API .. diff --git a/Misc/NEWS.d/3.13.0a1.rst b/Misc/NEWS.d/3.13.0a1.rst index 1391d670575746..0b7b69bbcb2478 100644 --- a/Misc/NEWS.d/3.13.0a1.rst +++ b/Misc/NEWS.d/3.13.0a1.rst @@ -3476,7 +3476,7 @@ Fix rare concurrency bug in lock acquisition by the logging package. .. nonce: ya5jBT .. section: Library -Added PY_THROW event hook for :mod:`cProfile` for generators +Added PY_THROW event hook for :mod:`!cProfile` for generators .. diff --git a/Misc/NEWS.d/3.14.0a1.rst b/Misc/NEWS.d/3.14.0a1.rst index 1938976fa4226a..2f76cbe0a037f4 100644 --- a/Misc/NEWS.d/3.14.0a1.rst +++ b/Misc/NEWS.d/3.14.0a1.rst @@ -2623,7 +2623,7 @@ Fix :func:`unittest.mock.patch` to not read attributes of the target when .. nonce: s4HXR0 .. section: Library -Fixed the use-after-free issue in :mod:`cProfile` by disallowing +Fixed the use-after-free issue in :mod:`!cProfile` by disallowing ``disable()`` and ``clear()`` in external timers. .. diff --git a/Misc/NEWS.d/3.14.0b1.rst b/Misc/NEWS.d/3.14.0b1.rst index 5d03d429f9ee14..df61f04d9be71e 100644 --- a/Misc/NEWS.d/3.14.0b1.rst +++ b/Misc/NEWS.d/3.14.0b1.rst @@ -747,7 +747,7 @@ caches. .. nonce: APBFCw .. section: Library -Fixed the :exc:`SystemError` in :mod:`cProfile` when locating the actual C +Fixed the :exc:`SystemError` in :mod:`!cProfile` when locating the actual C function of a method raises an exception. .. diff --git a/Misc/NEWS.d/3.15.0a1.rst b/Misc/NEWS.d/3.15.0a1.rst index 5331f270162710..273731f2535f70 100644 --- a/Misc/NEWS.d/3.15.0a1.rst +++ b/Misc/NEWS.d/3.15.0a1.rst @@ -4554,7 +4554,7 @@ auto-completion of imports. .. nonce: yLZJpV .. section: Core and Builtins -Make :mod:`cProfile` thread-safe on the :term:`free threaded ` build. .. diff --git a/Misc/NEWS.d/3.7.0a1.rst b/Misc/NEWS.d/3.7.0a1.rst index fd6ba07b53a617..69a0c09bda4145 100644 --- a/Misc/NEWS.d/3.7.0a1.rst +++ b/Misc/NEWS.d/3.7.0a1.rst @@ -1042,8 +1042,8 @@ method of other descriptors. Remove the ``PyEval_GetCallStats()`` function and deprecate the untested and undocumented ``sys.callstats()`` function. Remove the ``CALL_PROFILE`` -special build: use the :func:`sys.setprofile` function, :mod:`cProfile` or -:mod:`profile` to profile function calls. +special build: use the :func:`sys.setprofile` function, :mod:`!cProfile` or +:mod:`!profile` to profile function calls. ..