| **************************** |
| What's New in Python 2.3 |
| **************************** |
| |
| :Author: A.M. Kuchling |
| |
| .. |release| replace:: 1.01 |
| |
| .. $Id: whatsnew23.tex 54631 2007-03-31 11:58:36Z georg.brandl $ |
| |
| This article explains the new features in Python 2.3. Python 2.3 was released |
| on July 29, 2003. |
| |
| The main themes for Python 2.3 are polishing some of the features added in 2.2, |
| adding various small but useful enhancements to the core language, and expanding |
| the standard library. The new object model introduced in the previous version |
| has benefited from 18 months of bugfixes and from optimization efforts that have |
| improved the performance of new-style classes. A few new built-in functions |
| have been added such as :func:`sum` and :func:`enumerate`. The :keyword:`in` |
| operator can now be used for substring searches (e.g. ``"ab" in "abc"`` returns |
| :const:`True`). |
| |
| Some of the many new library features include Boolean, set, heap, and date/time |
| data types, the ability to import modules from ZIP-format archives, metadata |
| support for the long-awaited Python catalog, an updated version of IDLE, and |
| modules for logging messages, wrapping text, parsing CSV files, processing |
| command-line options, using BerkeleyDB databases... the list of new and |
| enhanced modules is lengthy. |
| |
| This article doesn't attempt to provide a complete specification of the new |
| features, but instead provides a convenient overview. For full details, you |
| should refer to the documentation for Python 2.3, such as the Python Library |
| Reference and the Python Reference Manual. If you want to understand the |
| complete implementation and design rationale, refer to the PEP for a particular |
| new feature. |
| |
| .. ====================================================================== |
| |
| |
| PEP 218: A Standard Set Datatype |
| ================================ |
| |
| The new :mod:`sets` module contains an implementation of a set datatype. The |
| :class:`Set` class is for mutable sets, sets that can have members added and |
| removed. The :class:`ImmutableSet` class is for sets that can't be modified, |
| and instances of :class:`ImmutableSet` can therefore be used as dictionary keys. |
| Sets are built on top of dictionaries, so the elements within a set must be |
| hashable. |
| |
| Here's a simple example:: |
| |
| >>> import sets |
| >>> S = sets.Set([1,2,3]) |
| >>> S |
| Set([1, 2, 3]) |
| >>> 1 in S |
| True |
| >>> 0 in S |
| False |
| >>> S.add(5) |
| >>> S.remove(3) |
| >>> S |
| Set([1, 2, 5]) |
| >>> |
| |
| The union and intersection of sets can be computed with the :meth:`union` and |
| :meth:`intersection` methods; an alternative notation uses the bitwise operators |
| ``&`` and ``|``. Mutable sets also have in-place versions of these methods, |
| :meth:`union_update` and :meth:`intersection_update`. :: |
| |
| >>> S1 = sets.Set([1,2,3]) |
| >>> S2 = sets.Set([4,5,6]) |
| >>> S1.union(S2) |
| Set([1, 2, 3, 4, 5, 6]) |
| >>> S1 | S2 # Alternative notation |
| Set([1, 2, 3, 4, 5, 6]) |
| >>> S1.intersection(S2) |
| Set([]) |
| >>> S1 & S2 # Alternative notation |
| Set([]) |
| >>> S1.union_update(S2) |
| >>> S1 |
| Set([1, 2, 3, 4, 5, 6]) |
| >>> |
| |
| It's also possible to take the symmetric difference of two sets. This is the |
| set of all elements in the union that aren't in the intersection. Another way |
| of putting it is that the symmetric difference contains all elements that are in |
| exactly one set. Again, there's an alternative notation (``^``), and an in- |
| place version with the ungainly name :meth:`symmetric_difference_update`. :: |
| |
| >>> S1 = sets.Set([1,2,3,4]) |
| >>> S2 = sets.Set([3,4,5,6]) |
| >>> S1.symmetric_difference(S2) |
| Set([1, 2, 5, 6]) |
| >>> S1 ^ S2 |
| Set([1, 2, 5, 6]) |
| >>> |
| |
| There are also :meth:`issubset` and :meth:`issuperset` methods for checking |
| whether one set is a subset or superset of another:: |
| |
| >>> S1 = sets.Set([1,2,3]) |
| >>> S2 = sets.Set([2,3]) |
| >>> S2.issubset(S1) |
| True |
| >>> S1.issubset(S2) |
| False |
| >>> S1.issuperset(S2) |
| True |
| >>> |
| |
| |
| .. seealso:: |
| |
| :pep:`218` - Adding a Built-In Set Object Type |
| PEP written by Greg V. Wilson. Implemented by Greg V. Wilson, Alex Martelli, and |
| GvR. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-generators: |
| |
| PEP 255: Simple Generators |
| ========================== |
| |
| In Python 2.2, generators were added as an optional feature, to be enabled by a |
| ``from __future__ import generators`` directive. In 2.3 generators no longer |
| need to be specially enabled, and are now always present; this means that |
| :keyword:`yield` is now always a keyword. The rest of this section is a copy of |
| the description of generators from the "What's New in Python 2.2" document; if |
| you read it back when Python 2.2 came out, you can skip the rest of this |
| section. |
| |
| You're doubtless familiar with how function calls work in Python or C. When you |
| call a function, it gets a private namespace where its local variables are |
| created. When the function reaches a :keyword:`return` statement, the local |
| variables are destroyed and the resulting value is returned to the caller. A |
| later call to the same function will get a fresh new set of local variables. |
| But, what if the local variables weren't thrown away on exiting a function? |
| What if you could later resume the function where it left off? This is what |
| generators provide; they can be thought of as resumable functions. |
| |
| Here's the simplest example of a generator function:: |
| |
| def generate_ints(N): |
| for i in range(N): |
| yield i |
| |
| A new keyword, :keyword:`yield`, was introduced for generators. Any function |
| containing a :keyword:`yield` statement is a generator function; this is |
| detected by Python's bytecode compiler which compiles the function specially as |
| a result. |
| |
| When you call a generator function, it doesn't return a single value; instead it |
| returns a generator object that supports the iterator protocol. On executing |
| the :keyword:`yield` statement, the generator outputs the value of ``i``, |
| similar to a :keyword:`return` statement. The big difference between |
| :keyword:`yield` and a :keyword:`return` statement is that on reaching a |
| :keyword:`yield` the generator's state of execution is suspended and local |
| variables are preserved. On the next call to the generator's ``.next()`` |
| method, the function will resume executing immediately after the |
| :keyword:`yield` statement. (For complicated reasons, the :keyword:`yield` |
| statement isn't allowed inside the :keyword:`try` block of a :keyword:`try`...\ |
| :keyword:`finally` statement; read :pep:`255` for a full explanation of the |
| interaction between :keyword:`yield` and exceptions.) |
| |
| Here's a sample usage of the :func:`generate_ints` generator:: |
| |
| >>> gen = generate_ints(3) |
| >>> gen |
| <generator object at 0x8117f90> |
| >>> gen.next() |
| 0 |
| >>> gen.next() |
| 1 |
| >>> gen.next() |
| 2 |
| >>> gen.next() |
| Traceback (most recent call last): |
| File "stdin", line 1, in ? |
| File "stdin", line 2, in generate_ints |
| StopIteration |
| |
| You could equally write ``for i in generate_ints(5)``, or ``a,b,c = |
| generate_ints(3)``. |
| |
| Inside a generator function, the :keyword:`return` statement can only be used |
| without a value, and signals the end of the procession of values; afterwards the |
| generator cannot return any further values. :keyword:`return` with a value, such |
| as ``return 5``, is a syntax error inside a generator function. The end of the |
| generator's results can also be indicated by raising :exc:`StopIteration` |
| manually, or by just letting the flow of execution fall off the bottom of the |
| function. |
| |
| You could achieve the effect of generators manually by writing your own class |
| and storing all the local variables of the generator as instance variables. For |
| example, returning a list of integers could be done by setting ``self.count`` to |
| 0, and having the :meth:`next` method increment ``self.count`` and return it. |
| However, for a moderately complicated generator, writing a corresponding class |
| would be much messier. :file:`Lib/test/test_generators.py` contains a number of |
| more interesting examples. The simplest one implements an in-order traversal of |
| a tree using generators recursively. :: |
| |
| # A recursive generator that generates Tree leaves in in-order. |
| def inorder(t): |
| if t: |
| for x in inorder(t.left): |
| yield x |
| yield t.label |
| for x in inorder(t.right): |
| yield x |
| |
| Two other examples in :file:`Lib/test/test_generators.py` produce solutions for |
| the N-Queens problem (placing $N$ queens on an $NxN$ chess board so that no |
| queen threatens another) and the Knight's Tour (a route that takes a knight to |
| every square of an $NxN$ chessboard without visiting any square twice). |
| |
| The idea of generators comes from other programming languages, especially Icon |
| (http://www.cs.arizona.edu/icon/), where the idea of generators is central. In |
| Icon, every expression and function call behaves like a generator. One example |
| from "An Overview of the Icon Programming Language" at |
| http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks |
| like:: |
| |
| sentence := "Store it in the neighboring harbor" |
| if (i := find("or", sentence)) > 5 then write(i) |
| |
| In Icon the :func:`find` function returns the indexes at which the substring |
| "or" is found: 3, 23, 33. In the :keyword:`if` statement, ``i`` is first |
| assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon |
| retries it with the second value of 23. 23 is greater than 5, so the comparison |
| now succeeds, and the code prints the value 23 to the screen. |
| |
| Python doesn't go nearly as far as Icon in adopting generators as a central |
| concept. Generators are considered part of the core Python language, but |
| learning or using them isn't compulsory; if they don't solve any problems that |
| you have, feel free to ignore them. One novel feature of Python's interface as |
| compared to Icon's is that a generator's state is represented as a concrete |
| object (the iterator) that can be passed around to other functions or stored in |
| a data structure. |
| |
| |
| .. seealso:: |
| |
| :pep:`255` - Simple Generators |
| Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland. Implemented mostly |
| by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-encodings: |
| |
| PEP 263: Source Code Encodings |
| ============================== |
| |
| Python source files can now be declared as being in different character set |
| encodings. Encodings are declared by including a specially formatted comment in |
| the first or second line of the source file. For example, a UTF-8 file can be |
| declared with:: |
| |
| #!/usr/bin/env python |
| # -*- coding: UTF-8 -*- |
| |
| Without such an encoding declaration, the default encoding used is 7-bit ASCII. |
| Executing or importing modules that contain string literals with 8-bit |
| characters and have no encoding declaration will result in a |
| :exc:`DeprecationWarning` being signalled by Python 2.3; in 2.4 this will be a |
| syntax error. |
| |
| The encoding declaration only affects Unicode string literals, which will be |
| converted to Unicode using the specified encoding. Note that Python identifiers |
| are still restricted to ASCII characters, so you can't have variable names that |
| use characters outside of the usual alphanumerics. |
| |
| |
| .. seealso:: |
| |
| :pep:`263` - Defining Python Source Code Encodings |
| Written by Marc-André Lemburg and Martin von Löwis; implemented by Suzuki Hisao |
| and Martin von Löwis. |
| |
| .. ====================================================================== |
| |
| |
| PEP 273: Importing Modules from ZIP Archives |
| ============================================ |
| |
| The new :mod:`zipimport` module adds support for importing modules from a ZIP- |
| format archive. You don't need to import the module explicitly; it will be |
| automatically imported if a ZIP archive's filename is added to ``sys.path``. |
| For example:: |
| |
| amk@nyman:~/src/python$ unzip -l /tmp/example.zip |
| Archive: /tmp/example.zip |
| Length Date Time Name |
| -------- ---- ---- ---- |
| 8467 11-26-02 22:30 jwzthreading.py |
| -------- ------- |
| 8467 1 file |
| amk@nyman:~/src/python$ ./python |
| Python 2.3 (#1, Aug 1 2003, 19:54:32) |
| >>> import sys |
| >>> sys.path.insert(0, '/tmp/example.zip') # Add .zip file to front of path |
| >>> import jwzthreading |
| >>> jwzthreading.__file__ |
| '/tmp/example.zip/jwzthreading.py' |
| >>> |
| |
| An entry in ``sys.path`` can now be the filename of a ZIP archive. The ZIP |
| archive can contain any kind of files, but only files named :file:`\*.py`, |
| :file:`\*.pyc`, or :file:`\*.pyo` can be imported. If an archive only contains |
| :file:`\*.py` files, Python will not attempt to modify the archive by adding the |
| corresponding :file:`\*.pyc` file, meaning that if a ZIP archive doesn't contain |
| :file:`\*.pyc` files, importing may be rather slow. |
| |
| A path within the archive can also be specified to only import from a |
| subdirectory; for example, the path :file:`/tmp/example.zip/lib/` would only |
| import from the :file:`lib/` subdirectory within the archive. |
| |
| |
| .. seealso:: |
| |
| :pep:`273` - Import Modules from Zip Archives |
| Written by James C. Ahlstrom, who also provided an implementation. Python 2.3 |
| follows the specification in :pep:`273`, but uses an implementation written by |
| Just van Rossum that uses the import hooks described in :pep:`302`. See section |
| :ref:`section-pep302` for a description of the new import hooks. |
| |
| .. ====================================================================== |
| |
| |
| PEP 277: Unicode file name support for Windows NT |
| ================================================= |
| |
| On Windows NT, 2000, and XP, the system stores file names as Unicode strings. |
| Traditionally, Python has represented file names as byte strings, which is |
| inadequate because it renders some file names inaccessible. |
| |
| Python now allows using arbitrary Unicode strings (within the limitations of the |
| file system) for all functions that expect file names, most notably the |
| :func:`open` built-in function. If a Unicode string is passed to |
| :func:`os.listdir`, Python now returns a list of Unicode strings. A new |
| function, :func:`os.getcwdu`, returns the current directory as a Unicode string. |
| |
| Byte strings still work as file names, and on Windows Python will transparently |
| convert them to Unicode using the ``mbcs`` encoding. |
| |
| Other systems also allow Unicode strings as file names but convert them to byte |
| strings before passing them to the system, which can cause a :exc:`UnicodeError` |
| to be raised. Applications can test whether arbitrary Unicode strings are |
| supported as file names by checking :attr:`os.path.supports_unicode_filenames`, |
| a Boolean value. |
| |
| Under MacOS, :func:`os.listdir` may now return Unicode filenames. |
| |
| |
| .. seealso:: |
| |
| :pep:`277` - Unicode file name support for Windows NT |
| Written by Neil Hodgson; implemented by Neil Hodgson, Martin von Löwis, and Mark |
| Hammond. |
| |
| .. ====================================================================== |
| |
| |
| PEP 278: Universal Newline Support |
| ================================== |
| |
| The three major operating systems used today are Microsoft Windows, Apple's |
| Macintosh OS, and the various Unix derivatives. A minor irritation of cross- |
| platform work is that these three platforms all use different characters to |
| mark the ends of lines in text files. Unix uses the linefeed (ASCII character |
| 10), MacOS uses the carriage return (ASCII character 13), and Windows uses a |
| two-character sequence of a carriage return plus a newline. |
| |
| Python's file objects can now support end of line conventions other than the one |
| followed by the platform on which Python is running. Opening a file with the |
| mode ``'U'`` or ``'rU'`` will open a file for reading in universal newline mode. |
| All three line ending conventions will be translated to a ``'\n'`` in the |
| strings returned by the various file methods such as :meth:`read` and |
| :meth:`readline`. |
| |
| Universal newline support is also used when importing modules and when executing |
| a file with the :func:`execfile` function. This means that Python modules can |
| be shared between all three operating systems without needing to convert the |
| line-endings. |
| |
| This feature can be disabled when compiling Python by specifying the |
| :option:`--without-universal-newlines` switch when running Python's |
| :program:`configure` script. |
| |
| |
| .. seealso:: |
| |
| :pep:`278` - Universal Newline Support |
| Written and implemented by Jack Jansen. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-enumerate: |
| |
| PEP 279: enumerate() |
| ==================== |
| |
| A new built-in function, :func:`enumerate`, will make certain loops a bit |
| clearer. ``enumerate(thing)``, where *thing* is either an iterator or a |
| sequence, returns a iterator that will return ``(0, thing[0])``, ``(1, |
| thing[1])``, ``(2, thing[2])``, and so forth. |
| |
| A common idiom to change every element of a list looks like this:: |
| |
| for i in range(len(L)): |
| item = L[i] |
| # ... compute some result based on item ... |
| L[i] = result |
| |
| This can be rewritten using :func:`enumerate` as:: |
| |
| for i, item in enumerate(L): |
| # ... compute some result based on item ... |
| L[i] = result |
| |
| |
| .. seealso:: |
| |
| :pep:`279` - The enumerate() built-in function |
| Written and implemented by Raymond D. Hettinger. |
| |
| .. ====================================================================== |
| |
| |
| PEP 282: The logging Package |
| ============================ |
| |
| A standard package for writing logs, :mod:`logging`, has been added to Python |
| 2.3. It provides a powerful and flexible mechanism for generating logging |
| output which can then be filtered and processed in various ways. A |
| configuration file written in a standard format can be used to control the |
| logging behavior of a program. Python includes handlers that will write log |
| records to standard error or to a file or socket, send them to the system log, |
| or even e-mail them to a particular address; of course, it's also possible to |
| write your own handler classes. |
| |
| The :class:`Logger` class is the primary class. Most application code will deal |
| with one or more :class:`Logger` objects, each one used by a particular |
| subsystem of the application. Each :class:`Logger` is identified by a name, and |
| names are organized into a hierarchy using ``.`` as the component separator. |
| For example, you might have :class:`Logger` instances named ``server``, |
| ``server.auth`` and ``server.network``. The latter two instances are below |
| ``server`` in the hierarchy. This means that if you turn up the verbosity for |
| ``server`` or direct ``server`` messages to a different handler, the changes |
| will also apply to records logged to ``server.auth`` and ``server.network``. |
| There's also a root :class:`Logger` that's the parent of all other loggers. |
| |
| For simple uses, the :mod:`logging` package contains some convenience functions |
| that always use the root log:: |
| |
| import logging |
| |
| logging.debug('Debugging information') |
| logging.info('Informational message') |
| logging.warning('Warning:config file %s not found', 'server.conf') |
| logging.error('Error occurred') |
| logging.critical('Critical error -- shutting down') |
| |
| This produces the following output:: |
| |
| WARNING:root:Warning:config file server.conf not found |
| ERROR:root:Error occurred |
| CRITICAL:root:Critical error -- shutting down |
| |
| In the default configuration, informational and debugging messages are |
| suppressed and the output is sent to standard error. You can enable the display |
| of informational and debugging messages by calling the :meth:`setLevel` method |
| on the root logger. |
| |
| Notice the :func:`warning` call's use of string formatting operators; all of the |
| functions for logging messages take the arguments ``(msg, arg1, arg2, ...)`` and |
| log the string resulting from ``msg % (arg1, arg2, ...)``. |
| |
| There's also an :func:`exception` function that records the most recent |
| traceback. Any of the other functions will also record the traceback if you |
| specify a true value for the keyword argument *exc_info*. :: |
| |
| def f(): |
| try: 1/0 |
| except: logging.exception('Problem recorded') |
| |
| f() |
| |
| This produces the following output:: |
| |
| ERROR:root:Problem recorded |
| Traceback (most recent call last): |
| File "t.py", line 6, in f |
| 1/0 |
| ZeroDivisionError: integer division or modulo by zero |
| |
| Slightly more advanced programs will use a logger other than the root logger. |
| The :func:`getLogger(name)` function is used to get a particular log, creating |
| it if it doesn't exist yet. :func:`getLogger(None)` returns the root logger. :: |
| |
| log = logging.getLogger('server') |
| ... |
| log.info('Listening on port %i', port) |
| ... |
| log.critical('Disk full') |
| ... |
| |
| Log records are usually propagated up the hierarchy, so a message logged to |
| ``server.auth`` is also seen by ``server`` and ``root``, but a :class:`Logger` |
| can prevent this by setting its :attr:`propagate` attribute to :const:`False`. |
| |
| There are more classes provided by the :mod:`logging` package that can be |
| customized. When a :class:`Logger` instance is told to log a message, it |
| creates a :class:`LogRecord` instance that is sent to any number of different |
| :class:`Handler` instances. Loggers and handlers can also have an attached list |
| of filters, and each filter can cause the :class:`LogRecord` to be ignored or |
| can modify the record before passing it along. When they're finally output, |
| :class:`LogRecord` instances are converted to text by a :class:`Formatter` |
| class. All of these classes can be replaced by your own specially-written |
| classes. |
| |
| With all of these features the :mod:`logging` package should provide enough |
| flexibility for even the most complicated applications. This is only an |
| incomplete overview of its features, so please see the package's reference |
| documentation for all of the details. Reading :pep:`282` will also be helpful. |
| |
| |
| .. seealso:: |
| |
| :pep:`282` - A Logging System |
| Written by Vinay Sajip and Trent Mick; implemented by Vinay Sajip. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-bool: |
| |
| PEP 285: A Boolean Type |
| ======================= |
| |
| A Boolean type was added to Python 2.3. Two new constants were added to the |
| :mod:`__builtin__` module, :const:`True` and :const:`False`. (:const:`True` and |
| :const:`False` constants were added to the built-ins in Python 2.2.1, but the |
| 2.2.1 versions are simply set to integer values of 1 and 0 and aren't a |
| different type.) |
| |
| The type object for this new type is named :class:`bool`; the constructor for it |
| takes any Python value and converts it to :const:`True` or :const:`False`. :: |
| |
| >>> bool(1) |
| True |
| >>> bool(0) |
| False |
| >>> bool([]) |
| False |
| >>> bool( (1,) ) |
| True |
| |
| Most of the standard library modules and built-in functions have been changed to |
| return Booleans. :: |
| |
| >>> obj = [] |
| >>> hasattr(obj, 'append') |
| True |
| >>> isinstance(obj, list) |
| True |
| >>> isinstance(obj, tuple) |
| False |
| |
| Python's Booleans were added with the primary goal of making code clearer. For |
| example, if you're reading a function and encounter the statement ``return 1``, |
| you might wonder whether the ``1`` represents a Boolean truth value, an index, |
| or a coefficient that multiplies some other quantity. If the statement is |
| ``return True``, however, the meaning of the return value is quite clear. |
| |
| Python's Booleans were *not* added for the sake of strict type-checking. A very |
| strict language such as Pascal would also prevent you performing arithmetic with |
| Booleans, and would require that the expression in an :keyword:`if` statement |
| always evaluate to a Boolean result. Python is not this strict and never will |
| be, as :pep:`285` explicitly says. This means you can still use any expression |
| in an :keyword:`if` statement, even ones that evaluate to a list or tuple or |
| some random object. The Boolean type is a subclass of the :class:`int` class so |
| that arithmetic using a Boolean still works. :: |
| |
| >>> True + 1 |
| 2 |
| >>> False + 1 |
| 1 |
| >>> False * 75 |
| 0 |
| >>> True * 75 |
| 75 |
| |
| To sum up :const:`True` and :const:`False` in a sentence: they're alternative |
| ways to spell the integer values 1 and 0, with the single difference that |
| :func:`str` and :func:`repr` return the strings ``'True'`` and ``'False'`` |
| instead of ``'1'`` and ``'0'``. |
| |
| |
| .. seealso:: |
| |
| :pep:`285` - Adding a bool type |
| Written and implemented by GvR. |
| |
| .. ====================================================================== |
| |
| |
| PEP 293: Codec Error Handling Callbacks |
| ======================================= |
| |
| When encoding a Unicode string into a byte string, unencodable characters may be |
| encountered. So far, Python has allowed specifying the error processing as |
| either "strict" (raising :exc:`UnicodeError`), "ignore" (skipping the |
| character), or "replace" (using a question mark in the output string), with |
| "strict" being the default behavior. It may be desirable to specify alternative |
| processing of such errors, such as inserting an XML character reference or HTML |
| entity reference into the converted string. |
| |
| Python now has a flexible framework to add different processing strategies. New |
| error handlers can be added with :func:`codecs.register_error`, and codecs then |
| can access the error handler with :func:`codecs.lookup_error`. An equivalent C |
| API has been added for codecs written in C. The error handler gets the necessary |
| state information such as the string being converted, the position in the string |
| where the error was detected, and the target encoding. The handler can then |
| either raise an exception or return a replacement string. |
| |
| Two additional error handlers have been implemented using this framework: |
| "backslashreplace" uses Python backslash quoting to represent unencodable |
| characters and "xmlcharrefreplace" emits XML character references. |
| |
| |
| .. seealso:: |
| |
| :pep:`293` - Codec Error Handling Callbacks |
| Written and implemented by Walter Dörwald. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-pep301: |
| |
| PEP 301: Package Index and Metadata for Distutils |
| ================================================= |
| |
| Support for the long-requested Python catalog makes its first appearance in 2.3. |
| |
| The heart of the catalog is the new Distutils :command:`register` command. |
| Running ``python setup.py register`` will collect the metadata describing a |
| package, such as its name, version, maintainer, description, &c., and send it to |
| a central catalog server. The resulting catalog is available from |
| http://www.python.org/pypi. |
| |
| To make the catalog a bit more useful, a new optional *classifiers* keyword |
| argument has been added to the Distutils :func:`setup` function. A list of |
| `Trove <http://catb.org/~esr/trove/>`_-style strings can be supplied to help |
| classify the software. |
| |
| Here's an example :file:`setup.py` with classifiers, written to be compatible |
| with older versions of the Distutils:: |
| |
| from distutils import core |
| kw = {'name': "Quixote", |
| 'version': "0.5.1", |
| 'description': "A highly Pythonic Web application framework", |
| # ... |
| } |
| |
| if (hasattr(core, 'setup_keywords') and |
| 'classifiers' in core.setup_keywords): |
| kw['classifiers'] = \ |
| ['Topic :: Internet :: WWW/HTTP :: Dynamic Content', |
| 'Environment :: No Input/Output (Daemon)', |
| 'Intended Audience :: Developers'], |
| |
| core.setup(**kw) |
| |
| The full list of classifiers can be obtained by running ``python setup.py |
| register --list-classifiers``. |
| |
| |
| .. seealso:: |
| |
| :pep:`301` - Package Index and Metadata for Distutils |
| Written and implemented by Richard Jones. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-pep302: |
| |
| PEP 302: New Import Hooks |
| ========================= |
| |
| While it's been possible to write custom import hooks ever since the |
| :mod:`ihooks` module was introduced in Python 1.3, no one has ever been really |
| happy with it because writing new import hooks is difficult and messy. There |
| have been various proposed alternatives such as the :mod:`imputil` and :mod:`iu` |
| modules, but none of them has ever gained much acceptance, and none of them were |
| easily usable from C code. |
| |
| :pep:`302` borrows ideas from its predecessors, especially from Gordon |
| McMillan's :mod:`iu` module. Three new items are added to the :mod:`sys` |
| module: |
| |
| * ``sys.path_hooks`` is a list of callable objects; most often they'll be |
| classes. Each callable takes a string containing a path and either returns an |
| importer object that will handle imports from this path or raises an |
| :exc:`ImportError` exception if it can't handle this path. |
| |
| * ``sys.path_importer_cache`` caches importer objects for each path, so |
| ``sys.path_hooks`` will only need to be traversed once for each path. |
| |
| * ``sys.meta_path`` is a list of importer objects that will be traversed before |
| ``sys.path`` is checked. This list is initially empty, but user code can add |
| objects to it. Additional built-in and frozen modules can be imported by an |
| object added to this list. |
| |
| Importer objects must have a single method, :meth:`find_module(fullname, |
| path=None)`. *fullname* will be a module or package name, e.g. ``string`` or |
| ``distutils.core``. :meth:`find_module` must return a loader object that has a |
| single method, :meth:`load_module(fullname)`, that creates and returns the |
| corresponding module object. |
| |
| Pseudo-code for Python's new import logic, therefore, looks something like this |
| (simplified a bit; see :pep:`302` for the full details):: |
| |
| for mp in sys.meta_path: |
| loader = mp(fullname) |
| if loader is not None: |
| <module> = loader.load_module(fullname) |
| |
| for path in sys.path: |
| for hook in sys.path_hooks: |
| try: |
| importer = hook(path) |
| except ImportError: |
| # ImportError, so try the other path hooks |
| pass |
| else: |
| loader = importer.find_module(fullname) |
| <module> = loader.load_module(fullname) |
| |
| # Not found! |
| raise ImportError |
| |
| |
| .. seealso:: |
| |
| :pep:`302` - New Import Hooks |
| Written by Just van Rossum and Paul Moore. Implemented by Just van Rossum. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-pep305: |
| |
| PEP 305: Comma-separated Files |
| ============================== |
| |
| Comma-separated files are a format frequently used for exporting data from |
| databases and spreadsheets. Python 2.3 adds a parser for comma-separated files. |
| |
| Comma-separated format is deceptively simple at first glance:: |
| |
| Costs,150,200,3.95 |
| |
| Read a line and call ``line.split(',')``: what could be simpler? But toss in |
| string data that can contain commas, and things get more complicated:: |
| |
| "Costs",150,200,3.95,"Includes taxes, shipping, and sundry items" |
| |
| A big ugly regular expression can parse this, but using the new :mod:`csv` |
| package is much simpler:: |
| |
| import csv |
| |
| input = open('datafile', 'rb') |
| reader = csv.reader(input) |
| for line in reader: |
| print line |
| |
| The :func:`reader` function takes a number of different options. The field |
| separator isn't limited to the comma and can be changed to any character, and so |
| can the quoting and line-ending characters. |
| |
| Different dialects of comma-separated files can be defined and registered; |
| currently there are two dialects, both used by Microsoft Excel. A separate |
| :class:`csv.writer` class will generate comma-separated files from a succession |
| of tuples or lists, quoting strings that contain the delimiter. |
| |
| |
| .. seealso:: |
| |
| :pep:`305` - CSV File API |
| Written and implemented by Kevin Altis, Dave Cole, Andrew McNamara, Skip |
| Montanaro, Cliff Wells. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-pep307: |
| |
| PEP 307: Pickle Enhancements |
| ============================ |
| |
| The :mod:`pickle` and :mod:`cPickle` modules received some attention during the |
| 2.3 development cycle. In 2.2, new-style classes could be pickled without |
| difficulty, but they weren't pickled very compactly; :pep:`307` quotes a trivial |
| example where a new-style class results in a pickled string three times longer |
| than that for a classic class. |
| |
| The solution was to invent a new pickle protocol. The :func:`pickle.dumps` |
| function has supported a text-or-binary flag for a long time. In 2.3, this |
| flag is redefined from a Boolean to an integer: 0 is the old text-mode pickle |
| format, 1 is the old binary format, and now 2 is a new 2.3-specific format. A |
| new constant, :const:`pickle.HIGHEST_PROTOCOL`, can be used to select the |
| fanciest protocol available. |
| |
| Unpickling is no longer considered a safe operation. 2.2's :mod:`pickle` |
| provided hooks for trying to prevent unsafe classes from being unpickled |
| (specifically, a :attr:`__safe_for_unpickling__` attribute), but none of this |
| code was ever audited and therefore it's all been ripped out in 2.3. You should |
| not unpickle untrusted data in any version of Python. |
| |
| To reduce the pickling overhead for new-style classes, a new interface for |
| customizing pickling was added using three special methods: |
| :meth:`__getstate__`, :meth:`__setstate__`, and :meth:`__getnewargs__`. Consult |
| :pep:`307` for the full semantics of these methods. |
| |
| As a way to compress pickles yet further, it's now possible to use integer codes |
| instead of long strings to identify pickled classes. The Python Software |
| Foundation will maintain a list of standardized codes; there's also a range of |
| codes for private use. Currently no codes have been specified. |
| |
| |
| .. seealso:: |
| |
| :pep:`307` - Extensions to the pickle protocol |
| Written and implemented by Guido van Rossum and Tim Peters. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-slices: |
| |
| Extended Slices |
| =============== |
| |
| Ever since Python 1.4, the slicing syntax has supported an optional third "step" |
| or "stride" argument. For example, these are all legal Python syntax: |
| ``L[1:10:2]``, ``L[:-1:1]``, ``L[::-1]``. This was added to Python at the |
| request of the developers of Numerical Python, which uses the third argument |
| extensively. However, Python's built-in list, tuple, and string sequence types |
| have never supported this feature, raising a :exc:`TypeError` if you tried it. |
| Michael Hudson contributed a patch to fix this shortcoming. |
| |
| For example, you can now easily extract the elements of a list that have even |
| indexes:: |
| |
| >>> L = range(10) |
| >>> L[::2] |
| [0, 2, 4, 6, 8] |
| |
| Negative values also work to make a copy of the same list in reverse order:: |
| |
| >>> L[::-1] |
| [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] |
| |
| This also works for tuples, arrays, and strings:: |
| |
| >>> s='abcd' |
| >>> s[::2] |
| 'ac' |
| >>> s[::-1] |
| 'dcba' |
| |
| If you have a mutable sequence such as a list or an array you can assign to or |
| delete an extended slice, but there are some differences between assignment to |
| extended and regular slices. Assignment to a regular slice can be used to |
| change the length of the sequence:: |
| |
| >>> a = range(3) |
| >>> a |
| [0, 1, 2] |
| >>> a[1:3] = [4, 5, 6] |
| >>> a |
| [0, 4, 5, 6] |
| |
| Extended slices aren't this flexible. When assigning to an extended slice, the |
| list on the right hand side of the statement must contain the same number of |
| items as the slice it is replacing:: |
| |
| >>> a = range(4) |
| >>> a |
| [0, 1, 2, 3] |
| >>> a[::2] |
| [0, 2] |
| >>> a[::2] = [0, -1] |
| >>> a |
| [0, 1, -1, 3] |
| >>> a[::2] = [0,1,2] |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in ? |
| ValueError: attempt to assign sequence of size 3 to extended slice of size 2 |
| |
| Deletion is more straightforward:: |
| |
| >>> a = range(4) |
| >>> a |
| [0, 1, 2, 3] |
| >>> a[::2] |
| [0, 2] |
| >>> del a[::2] |
| >>> a |
| [1, 3] |
| |
| One can also now pass slice objects to the :meth:`__getitem__` methods of the |
| built-in sequences:: |
| |
| >>> range(10).__getitem__(slice(0, 5, 2)) |
| [0, 2, 4] |
| |
| Or use slice objects directly in subscripts:: |
| |
| >>> range(10)[slice(0, 5, 2)] |
| [0, 2, 4] |
| |
| To simplify implementing sequences that support extended slicing, slice objects |
| now have a method :meth:`indices(length)` which, given the length of a sequence, |
| returns a ``(start, stop, step)`` tuple that can be passed directly to |
| :func:`range`. :meth:`indices` handles omitted and out-of-bounds indices in a |
| manner consistent with regular slices (and this innocuous phrase hides a welter |
| of confusing details!). The method is intended to be used like this:: |
| |
| class FakeSeq: |
| ... |
| def calc_item(self, i): |
| ... |
| def __getitem__(self, item): |
| if isinstance(item, slice): |
| indices = item.indices(len(self)) |
| return FakeSeq([self.calc_item(i) for i in range(*indices)]) |
| else: |
| return self.calc_item(i) |
| |
| From this example you can also see that the built-in :class:`slice` object is |
| now the type object for the slice type, and is no longer a function. This is |
| consistent with Python 2.2, where :class:`int`, :class:`str`, etc., underwent |
| the same change. |
| |
| .. ====================================================================== |
| |
| |
| Other Language Changes |
| ====================== |
| |
| Here are all of the changes that Python 2.3 makes to the core Python language. |
| |
| * The :keyword:`yield` statement is now always a keyword, as described in |
| section :ref:`section-generators` of this document. |
| |
| * A new built-in function :func:`enumerate` was added, as described in section |
| :ref:`section-enumerate` of this document. |
| |
| * Two new constants, :const:`True` and :const:`False` were added along with the |
| built-in :class:`bool` type, as described in section :ref:`section-bool` of this |
| document. |
| |
| * The :func:`int` type constructor will now return a long integer instead of |
| raising an :exc:`OverflowError` when a string or floating-point number is too |
| large to fit into an integer. This can lead to the paradoxical result that |
| ``isinstance(int(expression), int)`` is false, but that seems unlikely to cause |
| problems in practice. |
| |
| * Built-in types now support the extended slicing syntax, as described in |
| section :ref:`section-slices` of this document. |
| |
| * A new built-in function, :func:`sum(iterable, start=0)`, adds up the numeric |
| items in the iterable object and returns their sum. :func:`sum` only accepts |
| numbers, meaning that you can't use it to concatenate a bunch of strings. |
| (Contributed by Alex Martelli.) |
| |
| * ``list.insert(pos, value)`` used to insert *value* at the front of the list |
| when *pos* was negative. The behaviour has now been changed to be consistent |
| with slice indexing, so when *pos* is -1 the value will be inserted before the |
| last element, and so forth. |
| |
| * ``list.index(value)``, which searches for *value* within the list and returns |
| its index, now takes optional *start* and *stop* arguments to limit the search |
| to only part of the list. |
| |
| * Dictionaries have a new method, :meth:`pop(key[, *default*])`, that returns |
| the value corresponding to *key* and removes that key/value pair from the |
| dictionary. If the requested key isn't present in the dictionary, *default* is |
| returned if it's specified and :exc:`KeyError` raised if it isn't. :: |
| |
| >>> d = {1:2} |
| >>> d |
| {1: 2} |
| >>> d.pop(4) |
| Traceback (most recent call last): |
| File "stdin", line 1, in ? |
| KeyError: 4 |
| >>> d.pop(1) |
| 2 |
| >>> d.pop(1) |
| Traceback (most recent call last): |
| File "stdin", line 1, in ? |
| KeyError: 'pop(): dictionary is empty' |
| >>> d |
| {} |
| >>> |
| |
| There's also a new class method, :meth:`dict.fromkeys(iterable, value)`, that |
| creates a dictionary with keys taken from the supplied iterator *iterable* and |
| all values set to *value*, defaulting to ``None``. |
| |
| (Patches contributed by Raymond Hettinger.) |
| |
| Also, the :func:`dict` constructor now accepts keyword arguments to simplify |
| creating small dictionaries:: |
| |
| >>> dict(red=1, blue=2, green=3, black=4) |
| {'blue': 2, 'black': 4, 'green': 3, 'red': 1} |
| |
| (Contributed by Just van Rossum.) |
| |
| * The :keyword:`assert` statement no longer checks the ``__debug__`` flag, so |
| you can no longer disable assertions by assigning to ``__debug__``. Running |
| Python with the :option:`-O` switch will still generate code that doesn't |
| execute any assertions. |
| |
| * Most type objects are now callable, so you can use them to create new objects |
| such as functions, classes, and modules. (This means that the :mod:`new` module |
| can be deprecated in a future Python version, because you can now use the type |
| objects available in the :mod:`types` module.) For example, you can create a new |
| module object with the following code: |
| |
| :: |
| |
| >>> import types |
| >>> m = types.ModuleType('abc','docstring') |
| >>> m |
| <module 'abc' (built-in)> |
| >>> m.__doc__ |
| 'docstring' |
| |
| * A new warning, :exc:`PendingDeprecationWarning` was added to indicate features |
| which are in the process of being deprecated. The warning will *not* be printed |
| by default. To check for use of features that will be deprecated in the future, |
| supply :option:`-Walways::PendingDeprecationWarning::` on the command line or |
| use :func:`warnings.filterwarnings`. |
| |
| * The process of deprecating string-based exceptions, as in ``raise "Error |
| occurred"``, has begun. Raising a string will now trigger |
| :exc:`PendingDeprecationWarning`. |
| |
| * Using ``None`` as a variable name will now result in a :exc:`SyntaxWarning` |
| warning. In a future version of Python, ``None`` may finally become a keyword. |
| |
| * The :meth:`xreadlines` method of file objects, introduced in Python 2.1, is no |
| longer necessary because files now behave as their own iterator. |
| :meth:`xreadlines` was originally introduced as a faster way to loop over all |
| the lines in a file, but now you can simply write ``for line in file_obj``. |
| File objects also have a new read-only :attr:`encoding` attribute that gives the |
| encoding used by the file; Unicode strings written to the file will be |
| automatically converted to bytes using the given encoding. |
| |
| * The method resolution order used by new-style classes has changed, though |
| you'll only notice the difference if you have a really complicated inheritance |
| hierarchy. Classic classes are unaffected by this change. Python 2.2 |
| originally used a topological sort of a class's ancestors, but 2.3 now uses the |
| C3 algorithm as described in the paper `"A Monotonic Superclass Linearization |
| for Dylan" <http://www.webcom.com/haahr/dylan/linearization-oopsla96.html>`_. To |
| understand the motivation for this change, read Michele Simionato's article |
| `"Python 2.3 Method Resolution Order" <http://www.python.org/2.3/mro.html>`_, or |
| read the thread on python-dev starting with the message at |
| http://mail.python.org/pipermail/python-dev/2002-October/029035.html. Samuele |
| Pedroni first pointed out the problem and also implemented the fix by coding the |
| C3 algorithm. |
| |
| * Python runs multithreaded programs by switching between threads after |
| executing N bytecodes. The default value for N has been increased from 10 to |
| 100 bytecodes, speeding up single-threaded applications by reducing the |
| switching overhead. Some multithreaded applications may suffer slower response |
| time, but that's easily fixed by setting the limit back to a lower number using |
| :func:`sys.setcheckinterval(N)`. The limit can be retrieved with the new |
| :func:`sys.getcheckinterval` function. |
| |
| * One minor but far-reaching change is that the names of extension types defined |
| by the modules included with Python now contain the module and a ``'.'`` in |
| front of the type name. For example, in Python 2.2, if you created a socket and |
| printed its :attr:`__class__`, you'd get this output:: |
| |
| >>> s = socket.socket() |
| >>> s.__class__ |
| <type 'socket'> |
| |
| In 2.3, you get this:: |
| |
| >>> s.__class__ |
| <type '_socket.socket'> |
| |
| * One of the noted incompatibilities between old- and new-style classes has been |
| removed: you can now assign to the :attr:`__name__` and :attr:`__bases__` |
| attributes of new-style classes. There are some restrictions on what can be |
| assigned to :attr:`__bases__` along the lines of those relating to assigning to |
| an instance's :attr:`__class__` attribute. |
| |
| .. ====================================================================== |
| |
| |
| String Changes |
| -------------- |
| |
| * The :keyword:`in` operator now works differently for strings. Previously, when |
| evaluating ``X in Y`` where *X* and *Y* are strings, *X* could only be a single |
| character. That's now changed; *X* can be a string of any length, and ``X in Y`` |
| will return :const:`True` if *X* is a substring of *Y*. If *X* is the empty |
| string, the result is always :const:`True`. :: |
| |
| >>> 'ab' in 'abcd' |
| True |
| >>> 'ad' in 'abcd' |
| False |
| >>> '' in 'abcd' |
| True |
| |
| Note that this doesn't tell you where the substring starts; if you need that |
| information, use the :meth:`find` string method. |
| |
| * The :meth:`strip`, :meth:`lstrip`, and :meth:`rstrip` string methods now have |
| an optional argument for specifying the characters to strip. The default is |
| still to remove all whitespace characters:: |
| |
| >>> ' abc '.strip() |
| 'abc' |
| >>> '><><abc<><><>'.strip('<>') |
| 'abc' |
| >>> '><><abc<><><>\n'.strip('<>') |
| 'abc<><><>\n' |
| >>> u'\u4000\u4001abc\u4000'.strip(u'\u4000') |
| u'\u4001abc' |
| >>> |
| |
| (Suggested by Simon Brunning and implemented by Walter Dörwald.) |
| |
| * The :meth:`startswith` and :meth:`endswith` string methods now accept negative |
| numbers for the *start* and *end* parameters. |
| |
| * Another new string method is :meth:`zfill`, originally a function in the |
| :mod:`string` module. :meth:`zfill` pads a numeric string with zeros on the |
| left until it's the specified width. Note that the ``%`` operator is still more |
| flexible and powerful than :meth:`zfill`. :: |
| |
| >>> '45'.zfill(4) |
| '0045' |
| >>> '12345'.zfill(4) |
| '12345' |
| >>> 'goofy'.zfill(6) |
| '0goofy' |
| |
| (Contributed by Walter Dörwald.) |
| |
| * A new type object, :class:`basestring`, has been added. Both 8-bit strings and |
| Unicode strings inherit from this type, so ``isinstance(obj, basestring)`` will |
| return :const:`True` for either kind of string. It's a completely abstract |
| type, so you can't create :class:`basestring` instances. |
| |
| * Interned strings are no longer immortal and will now be garbage-collected in |
| the usual way when the only reference to them is from the internal dictionary of |
| interned strings. (Implemented by Oren Tirosh.) |
| |
| .. ====================================================================== |
| |
| |
| Optimizations |
| ------------- |
| |
| * The creation of new-style class instances has been made much faster; they're |
| now faster than classic classes! |
| |
| * The :meth:`sort` method of list objects has been extensively rewritten by Tim |
| Peters, and the implementation is significantly faster. |
| |
| * Multiplication of large long integers is now much faster thanks to an |
| implementation of Karatsuba multiplication, an algorithm that scales better than |
| the O(n\*n) required for the grade-school multiplication algorithm. (Original |
| patch by Christopher A. Craig, and significantly reworked by Tim Peters.) |
| |
| * The ``SET_LINENO`` opcode is now gone. This may provide a small speed |
| increase, depending on your compiler's idiosyncrasies. See section |
| :ref:`section-other` for a longer explanation. (Removed by Michael Hudson.) |
| |
| * :func:`xrange` objects now have their own iterator, making ``for i in |
| xrange(n)`` slightly faster than ``for i in range(n)``. (Patch by Raymond |
| Hettinger.) |
| |
| * A number of small rearrangements have been made in various hotspots to improve |
| performance, such as inlining a function or removing some code. (Implemented |
| mostly by GvR, but lots of people have contributed single changes.) |
| |
| The net result of the 2.3 optimizations is that Python 2.3 runs the pystone |
| benchmark around 25% faster than Python 2.2. |
| |
| .. ====================================================================== |
| |
| |
| New, Improved, and Deprecated Modules |
| ===================================== |
| |
| As usual, Python's standard library received a number of enhancements and bug |
| fixes. Here's a partial list of the most notable changes, sorted alphabetically |
| by module name. Consult the :file:`Misc/NEWS` file in the source tree for a more |
| complete list of changes, or look through the CVS logs for all the details. |
| |
| * The :mod:`array` module now supports arrays of Unicode characters using the |
| ``'u'`` format character. Arrays also now support using the ``+=`` assignment |
| operator to add another array's contents, and the ``*=`` assignment operator to |
| repeat an array. (Contributed by Jason Orendorff.) |
| |
| * The :mod:`bsddb` module has been replaced by version 4.1.6 of the `PyBSDDB |
| <http://pybsddb.sourceforge.net>`_ package, providing a more complete interface |
| to the transactional features of the BerkeleyDB library. |
| |
| The old version of the module has been renamed to :mod:`bsddb185` and is no |
| longer built automatically; you'll have to edit :file:`Modules/Setup` to enable |
| it. Note that the new :mod:`bsddb` package is intended to be compatible with |
| the old module, so be sure to file bugs if you discover any incompatibilities. |
| When upgrading to Python 2.3, if the new interpreter is compiled with a new |
| version of the underlying BerkeleyDB library, you will almost certainly have to |
| convert your database files to the new version. You can do this fairly easily |
| with the new scripts :file:`db2pickle.py` and :file:`pickle2db.py` which you |
| will find in the distribution's :file:`Tools/scripts` directory. If you've |
| already been using the PyBSDDB package and importing it as :mod:`bsddb3`, you |
| will have to change your ``import`` statements to import it as :mod:`bsddb`. |
| |
| * The new :mod:`bz2` module is an interface to the bz2 data compression library. |
| bz2-compressed data is usually smaller than corresponding :mod:`zlib`\ |
| -compressed data. (Contributed by Gustavo Niemeyer.) |
| |
| * A set of standard date/time types has been added in the new :mod:`datetime` |
| module. See the following section for more details. |
| |
| * The Distutils :class:`Extension` class now supports an extra constructor |
| argument named *depends* for listing additional source files that an extension |
| depends on. This lets Distutils recompile the module if any of the dependency |
| files are modified. For example, if :file:`sampmodule.c` includes the header |
| file :file:`sample.h`, you would create the :class:`Extension` object like |
| this:: |
| |
| ext = Extension("samp", |
| sources=["sampmodule.c"], |
| depends=["sample.h"]) |
| |
| Modifying :file:`sample.h` would then cause the module to be recompiled. |
| (Contributed by Jeremy Hylton.) |
| |
| * Other minor changes to Distutils: it now checks for the :envvar:`CC`, |
| :envvar:`CFLAGS`, :envvar:`CPP`, :envvar:`LDFLAGS`, and :envvar:`CPPFLAGS` |
| environment variables, using them to override the settings in Python's |
| configuration (contributed by Robert Weber). |
| |
| * Previously the :mod:`doctest` module would only search the docstrings of |
| public methods and functions for test cases, but it now also examines private |
| ones as well. The :func:`DocTestSuite(` function creates a |
| :class:`unittest.TestSuite` object from a set of :mod:`doctest` tests. |
| |
| * The new :func:`gc.get_referents(object)` function returns a list of all the |
| objects referenced by *object*. |
| |
| * The :mod:`getopt` module gained a new function, :func:`gnu_getopt`, that |
| supports the same arguments as the existing :func:`getopt` function but uses |
| GNU-style scanning mode. The existing :func:`getopt` stops processing options as |
| soon as a non-option argument is encountered, but in GNU-style mode processing |
| continues, meaning that options and arguments can be mixed. For example:: |
| |
| >>> getopt.getopt(['-f', 'filename', 'output', '-v'], 'f:v') |
| ([('-f', 'filename')], ['output', '-v']) |
| >>> getopt.gnu_getopt(['-f', 'filename', 'output', '-v'], 'f:v') |
| ([('-f', 'filename'), ('-v', '')], ['output']) |
| |
| (Contributed by Peter Ă…strand.) |
| |
| * The :mod:`grp`, :mod:`pwd`, and :mod:`resource` modules now return enhanced |
| tuples:: |
| |
| >>> import grp |
| >>> g = grp.getgrnam('amk') |
| >>> g.gr_name, g.gr_gid |
| ('amk', 500) |
| |
| * The :mod:`gzip` module can now handle files exceeding 2 GiB. |
| |
| * The new :mod:`heapq` module contains an implementation of a heap queue |
| algorithm. A heap is an array-like data structure that keeps items in a |
| partially sorted order such that, for every index *k*, ``heap[k] <= |
| heap[2*k+1]`` and ``heap[k] <= heap[2*k+2]``. This makes it quick to remove the |
| smallest item, and inserting a new item while maintaining the heap property is |
| O(lg n). (See http://www.nist.gov/dads/HTML/priorityque.html for more |
| information about the priority queue data structure.) |
| |
| The :mod:`heapq` module provides :func:`heappush` and :func:`heappop` functions |
| for adding and removing items while maintaining the heap property on top of some |
| other mutable Python sequence type. Here's an example that uses a Python list:: |
| |
| >>> import heapq |
| >>> heap = [] |
| >>> for item in [3, 7, 5, 11, 1]: |
| ... heapq.heappush(heap, item) |
| ... |
| >>> heap |
| [1, 3, 5, 11, 7] |
| >>> heapq.heappop(heap) |
| 1 |
| >>> heapq.heappop(heap) |
| 3 |
| >>> heap |
| [5, 7, 11] |
| |
| (Contributed by Kevin O'Connor.) |
| |
| * The IDLE integrated development environment has been updated using the code |
| from the IDLEfork project (http://idlefork.sf.net). The most notable feature is |
| that the code being developed is now executed in a subprocess, meaning that |
| there's no longer any need for manual ``reload()`` operations. IDLE's core code |
| has been incorporated into the standard library as the :mod:`idlelib` package. |
| |
| * The :mod:`imaplib` module now supports IMAP over SSL. (Contributed by Piers |
| Lauder and Tino Lange.) |
| |
| * The :mod:`itertools` contains a number of useful functions for use with |
| iterators, inspired by various functions provided by the ML and Haskell |
| languages. For example, ``itertools.ifilter(predicate, iterator)`` returns all |
| elements in the iterator for which the function :func:`predicate` returns |
| :const:`True`, and ``itertools.repeat(obj, N)`` returns ``obj`` *N* times. |
| There are a number of other functions in the module; see the package's reference |
| documentation for details. |
| (Contributed by Raymond Hettinger.) |
| |
| * Two new functions in the :mod:`math` module, :func:`degrees(rads)` and |
| :func:`radians(degs)`, convert between radians and degrees. Other functions in |
| the :mod:`math` module such as :func:`math.sin` and :func:`math.cos` have always |
| required input values measured in radians. Also, an optional *base* argument |
| was added to :func:`math.log` to make it easier to compute logarithms for bases |
| other than ``e`` and ``10``. (Contributed by Raymond Hettinger.) |
| |
| * Several new POSIX functions (:func:`getpgid`, :func:`killpg`, :func:`lchown`, |
| :func:`loadavg`, :func:`major`, :func:`makedev`, :func:`minor`, and |
| :func:`mknod`) were added to the :mod:`posix` module that underlies the |
| :mod:`os` module. (Contributed by Gustavo Niemeyer, Geert Jansen, and Denis S. |
| Otkidach.) |
| |
| * In the :mod:`os` module, the :func:`\*stat` family of functions can now report |
| fractions of a second in a timestamp. Such time stamps are represented as |
| floats, similar to the value returned by :func:`time.time`. |
| |
| During testing, it was found that some applications will break if time stamps |
| are floats. For compatibility, when using the tuple interface of the |
| :class:`stat_result` time stamps will be represented as integers. When using |
| named fields (a feature first introduced in Python 2.2), time stamps are still |
| represented as integers, unless :func:`os.stat_float_times` is invoked to enable |
| float return values:: |
| |
| >>> os.stat("/tmp").st_mtime |
| 1034791200 |
| >>> os.stat_float_times(True) |
| >>> os.stat("/tmp").st_mtime |
| 1034791200.6335014 |
| |
| In Python 2.4, the default will change to always returning floats. |
| |
| Application developers should enable this feature only if all their libraries |
| work properly when confronted with floating point time stamps, or if they use |
| the tuple API. If used, the feature should be activated on an application level |
| instead of trying to enable it on a per-use basis. |
| |
| * The :mod:`optparse` module contains a new parser for command-line arguments |
| that can convert option values to a particular Python type and will |
| automatically generate a usage message. See the following section for more |
| details. |
| |
| * The old and never-documented :mod:`linuxaudiodev` module has been deprecated, |
| and a new version named :mod:`ossaudiodev` has been added. The module was |
| renamed because the OSS sound drivers can be used on platforms other than Linux, |
| and the interface has also been tidied and brought up to date in various ways. |
| (Contributed by Greg Ward and Nicholas FitzRoy-Dale.) |
| |
| * The new :mod:`platform` module contains a number of functions that try to |
| determine various properties of the platform you're running on. There are |
| functions for getting the architecture, CPU type, the Windows OS version, and |
| even the Linux distribution version. (Contributed by Marc-André Lemburg.) |
| |
| * The parser objects provided by the :mod:`pyexpat` module can now optionally |
| buffer character data, resulting in fewer calls to your character data handler |
| and therefore faster performance. Setting the parser object's |
| :attr:`buffer_text` attribute to :const:`True` will enable buffering. |
| |
| * The :func:`sample(population, k)` function was added to the :mod:`random` |
| module. *population* is a sequence or :class:`xrange` object containing the |
| elements of a population, and :func:`sample` chooses *k* elements from the |
| population without replacing chosen elements. *k* can be any value up to |
| ``len(population)``. For example:: |
| |
| >>> days = ['Mo', 'Tu', 'We', 'Th', 'Fr', 'St', 'Sn'] |
| >>> random.sample(days, 3) # Choose 3 elements |
| ['St', 'Sn', 'Th'] |
| >>> random.sample(days, 7) # Choose 7 elements |
| ['Tu', 'Th', 'Mo', 'We', 'St', 'Fr', 'Sn'] |
| >>> random.sample(days, 7) # Choose 7 again |
| ['We', 'Mo', 'Sn', 'Fr', 'Tu', 'St', 'Th'] |
| >>> random.sample(days, 8) # Can't choose eight |
| Traceback (most recent call last): |
| File "<stdin>", line 1, in ? |
| File "random.py", line 414, in sample |
| raise ValueError, "sample larger than population" |
| ValueError: sample larger than population |
| >>> random.sample(xrange(1,10000,2), 10) # Choose ten odd nos. under 10000 |
| [3407, 3805, 1505, 7023, 2401, 2267, 9733, 3151, 8083, 9195] |
| |
| The :mod:`random` module now uses a new algorithm, the Mersenne Twister, |
| implemented in C. It's faster and more extensively studied than the previous |
| algorithm. |
| |
| (All changes contributed by Raymond Hettinger.) |
| |
| * The :mod:`readline` module also gained a number of new functions: |
| :func:`get_history_item`, :func:`get_current_history_length`, and |
| :func:`redisplay`. |
| |
| * The :mod:`rexec` and :mod:`Bastion` modules have been declared dead, and |
| attempts to import them will fail with a :exc:`RuntimeError`. New-style classes |
| provide new ways to break out of the restricted execution environment provided |
| by :mod:`rexec`, and no one has interest in fixing them or time to do so. If |
| you have applications using :mod:`rexec`, rewrite them to use something else. |
| |
| (Sticking with Python 2.2 or 2.1 will not make your applications any safer |
| because there are known bugs in the :mod:`rexec` module in those versions. To |
| repeat: if you're using :mod:`rexec`, stop using it immediately.) |
| |
| * The :mod:`rotor` module has been deprecated because the algorithm it uses for |
| encryption is not believed to be secure. If you need encryption, use one of the |
| several AES Python modules that are available separately. |
| |
| * The :mod:`shutil` module gained a :func:`move(src, dest)` function that |
| recursively moves a file or directory to a new location. |
| |
| * Support for more advanced POSIX signal handling was added to the :mod:`signal` |
| but then removed again as it proved impossible to make it work reliably across |
| platforms. |
| |
| * The :mod:`socket` module now supports timeouts. You can call the |
| :meth:`settimeout(t)` method on a socket object to set a timeout of *t* seconds. |
| Subsequent socket operations that take longer than *t* seconds to complete will |
| abort and raise a :exc:`socket.timeout` exception. |
| |
| The original timeout implementation was by Tim O'Malley. Michael Gilfix |
| integrated it into the Python :mod:`socket` module and shepherded it through a |
| lengthy review. After the code was checked in, Guido van Rossum rewrote parts |
| of it. (This is a good example of a collaborative development process in |
| action.) |
| |
| * On Windows, the :mod:`socket` module now ships with Secure Sockets Layer |
| (SSL) support. |
| |
| * The value of the C :const:`PYTHON_API_VERSION` macro is now exposed at the |
| Python level as ``sys.api_version``. The current exception can be cleared by |
| calling the new :func:`sys.exc_clear` function. |
| |
| * The new :mod:`tarfile` module allows reading from and writing to |
| :program:`tar`\ -format archive files. (Contributed by Lars Gustäbel.) |
| |
| * The new :mod:`textwrap` module contains functions for wrapping strings |
| containing paragraphs of text. The :func:`wrap(text, width)` function takes a |
| string and returns a list containing the text split into lines of no more than |
| the chosen width. The :func:`fill(text, width)` function returns a single |
| string, reformatted to fit into lines no longer than the chosen width. (As you |
| can guess, :func:`fill` is built on top of :func:`wrap`. For example:: |
| |
| >>> import textwrap |
| >>> paragraph = "Not a whit, we defy augury: ... more text ..." |
| >>> textwrap.wrap(paragraph, 60) |
| ["Not a whit, we defy augury: there's a special providence in", |
| "the fall of a sparrow. If it be now, 'tis not to come; if it", |
| ...] |
| >>> print textwrap.fill(paragraph, 35) |
| Not a whit, we defy augury: there's |
| a special providence in the fall of |
| a sparrow. If it be now, 'tis not |
| to come; if it be not to come, it |
| will be now; if it be not now, yet |
| it will come: the readiness is all. |
| >>> |
| |
| The module also contains a :class:`TextWrapper` class that actually implements |
| the text wrapping strategy. Both the :class:`TextWrapper` class and the |
| :func:`wrap` and :func:`fill` functions support a number of additional keyword |
| arguments for fine-tuning the formatting; consult the module's documentation |
| for details. (Contributed by Greg Ward.) |
| |
| * The :mod:`thread` and :mod:`threading` modules now have companion modules, |
| :mod:`dummy_thread` and :mod:`dummy_threading`, that provide a do-nothing |
| implementation of the :mod:`thread` module's interface for platforms where |
| threads are not supported. The intention is to simplify thread-aware modules |
| (ones that *don't* rely on threads to run) by putting the following code at the |
| top:: |
| |
| try: |
| import threading as _threading |
| except ImportError: |
| import dummy_threading as _threading |
| |
| In this example, :mod:`_threading` is used as the module name to make it clear |
| that the module being used is not necessarily the actual :mod:`threading` |
| module. Code can call functions and use classes in :mod:`_threading` whether or |
| not threads are supported, avoiding an :keyword:`if` statement and making the |
| code slightly clearer. This module will not magically make multithreaded code |
| run without threads; code that waits for another thread to return or to do |
| something will simply hang forever. |
| |
| * The :mod:`time` module's :func:`strptime` function has long been an annoyance |
| because it uses the platform C library's :func:`strptime` implementation, and |
| different platforms sometimes have odd bugs. Brett Cannon contributed a |
| portable implementation that's written in pure Python and should behave |
| identically on all platforms. |
| |
| * The new :mod:`timeit` module helps measure how long snippets of Python code |
| take to execute. The :file:`timeit.py` file can be run directly from the |
| command line, or the module's :class:`Timer` class can be imported and used |
| directly. Here's a short example that figures out whether it's faster to |
| convert an 8-bit string to Unicode by appending an empty Unicode string to it or |
| by using the :func:`unicode` function:: |
| |
| import timeit |
| |
| timer1 = timeit.Timer('unicode("abc")') |
| timer2 = timeit.Timer('"abc" + u""') |
| |
| # Run three trials |
| print timer1.repeat(repeat=3, number=100000) |
| print timer2.repeat(repeat=3, number=100000) |
| |
| # On my laptop this outputs: |
| # [0.36831796169281006, 0.37441694736480713, 0.35304892063140869] |
| # [0.17574405670166016, 0.18193507194519043, 0.17565798759460449] |
| |
| * The :mod:`Tix` module has received various bug fixes and updates for the |
| current version of the Tix package. |
| |
| * The :mod:`Tkinter` module now works with a thread-enabled version of Tcl. |
| Tcl's threading model requires that widgets only be accessed from the thread in |
| which they're created; accesses from another thread can cause Tcl to panic. For |
| certain Tcl interfaces, :mod:`Tkinter` will now automatically avoid this when a |
| widget is accessed from a different thread by marshalling a command, passing it |
| to the correct thread, and waiting for the results. Other interfaces can't be |
| handled automatically but :mod:`Tkinter` will now raise an exception on such an |
| access so that you can at least find out about the problem. See |
| http://mail.python.org/pipermail/python-dev/2002-December/031107.html for a more |
| detailed explanation of this change. (Implemented by Martin von Löwis.) |
| |
| * Calling Tcl methods through :mod:`_tkinter` no longer returns only strings. |
| Instead, if Tcl returns other objects those objects are converted to their |
| Python equivalent, if one exists, or wrapped with a :class:`_tkinter.Tcl_Obj` |
| object if no Python equivalent exists. This behavior can be controlled through |
| the :meth:`wantobjects` method of :class:`tkapp` objects. |
| |
| When using :mod:`_tkinter` through the :mod:`Tkinter` module (as most Tkinter |
| applications will), this feature is always activated. It should not cause |
| compatibility problems, since Tkinter would always convert string results to |
| Python types where possible. |
| |
| If any incompatibilities are found, the old behavior can be restored by setting |
| the :attr:`wantobjects` variable in the :mod:`Tkinter` module to false before |
| creating the first :class:`tkapp` object. :: |
| |
| import Tkinter |
| Tkinter.wantobjects = 0 |
| |
| Any breakage caused by this change should be reported as a bug. |
| |
| * The :mod:`UserDict` module has a new :class:`DictMixin` class which defines |
| all dictionary methods for classes that already have a minimum mapping |
| interface. This greatly simplifies writing classes that need to be |
| substitutable for dictionaries, such as the classes in the :mod:`shelve` |
| module. |
| |
| Adding the mix-in as a superclass provides the full dictionary interface |
| whenever the class defines :meth:`__getitem__`, :meth:`__setitem__`, |
| :meth:`__delitem__`, and :meth:`keys`. For example:: |
| |
| >>> import UserDict |
| >>> class SeqDict(UserDict.DictMixin): |
| ... """Dictionary lookalike implemented with lists.""" |
| ... def __init__(self): |
| ... self.keylist = [] |
| ... self.valuelist = [] |
| ... def __getitem__(self, key): |
| ... try: |
| ... i = self.keylist.index(key) |
| ... except ValueError: |
| ... raise KeyError |
| ... return self.valuelist[i] |
| ... def __setitem__(self, key, value): |
| ... try: |
| ... i = self.keylist.index(key) |
| ... self.valuelist[i] = value |
| ... except ValueError: |
| ... self.keylist.append(key) |
| ... self.valuelist.append(value) |
| ... def __delitem__(self, key): |
| ... try: |
| ... i = self.keylist.index(key) |
| ... except ValueError: |
| ... raise KeyError |
| ... self.keylist.pop(i) |
| ... self.valuelist.pop(i) |
| ... def keys(self): |
| ... return list(self.keylist) |
| ... |
| >>> s = SeqDict() |
| >>> dir(s) # See that other dictionary methods are implemented |
| ['__cmp__', '__contains__', '__delitem__', '__doc__', '__getitem__', |
| '__init__', '__iter__', '__len__', '__module__', '__repr__', |
| '__setitem__', 'clear', 'get', 'has_key', 'items', 'iteritems', |
| 'iterkeys', 'itervalues', 'keylist', 'keys', 'pop', 'popitem', |
| 'setdefault', 'update', 'valuelist', 'values'] |
| |
| (Contributed by Raymond Hettinger.) |
| |
| * The DOM implementation in :mod:`xml.dom.minidom` can now generate XML output |
| in a particular encoding by providing an optional encoding argument to the |
| :meth:`toxml` and :meth:`toprettyxml` methods of DOM nodes. |
| |
| * The :mod:`xmlrpclib` module now supports an XML-RPC extension for handling nil |
| data values such as Python's ``None``. Nil values are always supported on |
| unmarshalling an XML-RPC response. To generate requests containing ``None``, |
| you must supply a true value for the *allow_none* parameter when creating a |
| :class:`Marshaller` instance. |
| |
| * The new :mod:`DocXMLRPCServer` module allows writing self-documenting XML-RPC |
| servers. Run it in demo mode (as a program) to see it in action. Pointing the |
| Web browser to the RPC server produces pydoc-style documentation; pointing |
| xmlrpclib to the server allows invoking the actual methods. (Contributed by |
| Brian Quinlan.) |
| |
| * Support for internationalized domain names (RFCs 3454, 3490, 3491, and 3492) |
| has been added. The "idna" encoding can be used to convert between a Unicode |
| domain name and the ASCII-compatible encoding (ACE) of that name. :: |
| |
| >{}>{}> u"www.Alliancefrançaise.nu".encode("idna") |
| 'www.xn--alliancefranaise-npb.nu' |
| |
| The :mod:`socket` module has also been extended to transparently convert |
| Unicode hostnames to the ACE version before passing them to the C library. |
| Modules that deal with hostnames such as :mod:`httplib` and :mod:`ftplib`) |
| also support Unicode host names; :mod:`httplib` also sends HTTP ``Host`` |
| headers using the ACE version of the domain name. :mod:`urllib` supports |
| Unicode URLs with non-ASCII host names as long as the ``path`` part of the URL |
| is ASCII only. |
| |
| To implement this change, the :mod:`stringprep` module, the ``mkstringprep`` |
| tool and the ``punycode`` encoding have been added. |
| |
| .. ====================================================================== |
| |
| |
| Date/Time Type |
| -------------- |
| |
| Date and time types suitable for expressing timestamps were added as the |
| :mod:`datetime` module. The types don't support different calendars or many |
| fancy features, and just stick to the basics of representing time. |
| |
| The three primary types are: :class:`date`, representing a day, month, and year; |
| :class:`time`, consisting of hour, minute, and second; and :class:`datetime`, |
| which contains all the attributes of both :class:`date` and :class:`time`. |
| There's also a :class:`timedelta` class representing differences between two |
| points in time, and time zone logic is implemented by classes inheriting from |
| the abstract :class:`tzinfo` class. |
| |
| You can create instances of :class:`date` and :class:`time` by either supplying |
| keyword arguments to the appropriate constructor, e.g. |
| ``datetime.date(year=1972, month=10, day=15)``, or by using one of a number of |
| class methods. For example, the :meth:`date.today` class method returns the |
| current local date. |
| |
| Once created, instances of the date/time classes are all immutable. There are a |
| number of methods for producing formatted strings from objects:: |
| |
| >>> import datetime |
| >>> now = datetime.datetime.now() |
| >>> now.isoformat() |
| '2002-12-30T21:27:03.994956' |
| >>> now.ctime() # Only available on date, datetime |
| 'Mon Dec 30 21:27:03 2002' |
| >>> now.strftime('%Y %d %b') |
| '2002 30 Dec' |
| |
| The :meth:`replace` method allows modifying one or more fields of a |
| :class:`date` or :class:`datetime` instance, returning a new instance:: |
| |
| >>> d = datetime.datetime.now() |
| >>> d |
| datetime.datetime(2002, 12, 30, 22, 15, 38, 827738) |
| >>> d.replace(year=2001, hour = 12) |
| datetime.datetime(2001, 12, 30, 12, 15, 38, 827738) |
| >>> |
| |
| Instances can be compared, hashed, and converted to strings (the result is the |
| same as that of :meth:`isoformat`). :class:`date` and :class:`datetime` |
| instances can be subtracted from each other, and added to :class:`timedelta` |
| instances. The largest missing feature is that there's no standard library |
| support for parsing strings and getting back a :class:`date` or |
| :class:`datetime`. |
| |
| For more information, refer to the module's reference documentation. |
| (Contributed by Tim Peters.) |
| |
| .. ====================================================================== |
| |
| |
| The optparse Module |
| ------------------- |
| |
| The :mod:`getopt` module provides simple parsing of command-line arguments. The |
| new :mod:`optparse` module (originally named Optik) provides more elaborate |
| command-line parsing that follows the Unix conventions, automatically creates |
| the output for :option:`--help`, and can perform different actions for different |
| options. |
| |
| You start by creating an instance of :class:`OptionParser` and telling it what |
| your program's options are. :: |
| |
| import sys |
| from optparse import OptionParser |
| |
| op = OptionParser() |
| op.add_option('-i', '--input', |
| action='store', type='string', dest='input', |
| help='set input filename') |
| op.add_option('-l', '--length', |
| action='store', type='int', dest='length', |
| help='set maximum length of output') |
| |
| Parsing a command line is then done by calling the :meth:`parse_args` method. :: |
| |
| options, args = op.parse_args(sys.argv[1:]) |
| print options |
| print args |
| |
| This returns an object containing all of the option values, and a list of |
| strings containing the remaining arguments. |
| |
| Invoking the script with the various arguments now works as you'd expect it to. |
| Note that the length argument is automatically converted to an integer. :: |
| |
| $ ./python opt.py -i data arg1 |
| <Values at 0x400cad4c: {'input': 'data', 'length': None}> |
| ['arg1'] |
| $ ./python opt.py --input=data --length=4 |
| <Values at 0x400cad2c: {'input': 'data', 'length': 4}> |
| [] |
| $ |
| |
| The help message is automatically generated for you:: |
| |
| $ ./python opt.py --help |
| usage: opt.py [options] |
| |
| options: |
| -h, --help show this help message and exit |
| -iINPUT, --input=INPUT |
| set input filename |
| -lLENGTH, --length=LENGTH |
| set maximum length of output |
| $ |
| |
| See the module's documentation for more details. |
| |
| |
| Optik was written by Greg Ward, with suggestions from the readers of the Getopt |
| SIG. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-pymalloc: |
| |
| Pymalloc: A Specialized Object Allocator |
| ======================================== |
| |
| Pymalloc, a specialized object allocator written by Vladimir Marangozov, was a |
| feature added to Python 2.1. Pymalloc is intended to be faster than the system |
| :cfunc:`malloc` and to have less memory overhead for allocation patterns typical |
| of Python programs. The allocator uses C's :cfunc:`malloc` function to get large |
| pools of memory and then fulfills smaller memory requests from these pools. |
| |
| In 2.1 and 2.2, pymalloc was an experimental feature and wasn't enabled by |
| default; you had to explicitly enable it when compiling Python by providing the |
| :option:`--with-pymalloc` option to the :program:`configure` script. In 2.3, |
| pymalloc has had further enhancements and is now enabled by default; you'll have |
| to supply :option:`--without-pymalloc` to disable it. |
| |
| This change is transparent to code written in Python; however, pymalloc may |
| expose bugs in C extensions. Authors of C extension modules should test their |
| code with pymalloc enabled, because some incorrect code may cause core dumps at |
| runtime. |
| |
| There's one particularly common error that causes problems. There are a number |
| of memory allocation functions in Python's C API that have previously just been |
| aliases for the C library's :cfunc:`malloc` and :cfunc:`free`, meaning that if |
| you accidentally called mismatched functions the error wouldn't be noticeable. |
| When the object allocator is enabled, these functions aren't aliases of |
| :cfunc:`malloc` and :cfunc:`free` any more, and calling the wrong function to |
| free memory may get you a core dump. For example, if memory was allocated using |
| :cfunc:`PyObject_Malloc`, it has to be freed using :cfunc:`PyObject_Free`, not |
| :cfunc:`free`. A few modules included with Python fell afoul of this and had to |
| be fixed; doubtless there are more third-party modules that will have the same |
| problem. |
| |
| As part of this change, the confusing multiple interfaces for allocating memory |
| have been consolidated down into two API families. Memory allocated with one |
| family must not be manipulated with functions from the other family. There is |
| one family for allocating chunks of memory and another family of functions |
| specifically for allocating Python objects. |
| |
| * To allocate and free an undistinguished chunk of memory use the "raw memory" |
| family: :cfunc:`PyMem_Malloc`, :cfunc:`PyMem_Realloc`, and :cfunc:`PyMem_Free`. |
| |
| * The "object memory" family is the interface to the pymalloc facility described |
| above and is biased towards a large number of "small" allocations: |
| :cfunc:`PyObject_Malloc`, :cfunc:`PyObject_Realloc`, and :cfunc:`PyObject_Free`. |
| |
| * To allocate and free Python objects, use the "object" family |
| :cfunc:`PyObject_New`, :cfunc:`PyObject_NewVar`, and :cfunc:`PyObject_Del`. |
| |
| Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides debugging |
| features to catch memory overwrites and doubled frees in both extension modules |
| and in the interpreter itself. To enable this support, compile a debugging |
| version of the Python interpreter by running :program:`configure` with |
| :option:`--with-pydebug`. |
| |
| To aid extension writers, a header file :file:`Misc/pymemcompat.h` is |
| distributed with the source to Python 2.3 that allows Python extensions to use |
| the 2.3 interfaces to memory allocation while compiling against any version of |
| Python since 1.5.2. You would copy the file from Python's source distribution |
| and bundle it with the source of your extension. |
| |
| |
| .. seealso:: |
| |
| http://svn.python.org/view/python/trunk/Objects/obmalloc.c |
| For the full details of the pymalloc implementation, see the comments at |
| the top of the file :file:`Objects/obmalloc.c` in the Python source code. |
| The above link points to the file within the python.org SVN browser. |
| |
| .. ====================================================================== |
| |
| |
| Build and C API Changes |
| ======================= |
| |
| Changes to Python's build process and to the C API include: |
| |
| * The cycle detection implementation used by the garbage collection has proven |
| to be stable, so it's now been made mandatory. You can no longer compile Python |
| without it, and the :option:`--with-cycle-gc` switch to :program:`configure` has |
| been removed. |
| |
| * Python can now optionally be built as a shared library |
| (:file:`libpython2.3.so`) by supplying :option:`--enable-shared` when running |
| Python's :program:`configure` script. (Contributed by Ondrej Palkovsky.) |
| |
| * The :cmacro:`DL_EXPORT` and :cmacro:`DL_IMPORT` macros are now deprecated. |
| Initialization functions for Python extension modules should now be declared |
| using the new macro :cmacro:`PyMODINIT_FUNC`, while the Python core will |
| generally use the :cmacro:`PyAPI_FUNC` and :cmacro:`PyAPI_DATA` macros. |
| |
| * The interpreter can be compiled without any docstrings for the built-in |
| functions and modules by supplying :option:`--without-doc-strings` to the |
| :program:`configure` script. This makes the Python executable about 10% smaller, |
| but will also mean that you can't get help for Python's built-ins. (Contributed |
| by Gustavo Niemeyer.) |
| |
| * The :cfunc:`PyArg_NoArgs` macro is now deprecated, and code that uses it |
| should be changed. For Python 2.2 and later, the method definition table can |
| specify the :const:`METH_NOARGS` flag, signalling that there are no arguments, |
| and the argument checking can then be removed. If compatibility with pre-2.2 |
| versions of Python is important, the code could use ``PyArg_ParseTuple(args, |
| "")`` instead, but this will be slower than using :const:`METH_NOARGS`. |
| |
| * :cfunc:`PyArg_ParseTuple` accepts new format characters for various sizes of |
| unsigned integers: ``B`` for :ctype:`unsigned char`, ``H`` for :ctype:`unsigned |
| short int`, ``I`` for :ctype:`unsigned int`, and ``K`` for :ctype:`unsigned |
| long long`. |
| |
| * A new function, :cfunc:`PyObject_DelItemString(mapping, char \*key)` was added |
| as shorthand for ``PyObject_DelItem(mapping, PyString_New(key))``. |
| |
| * File objects now manage their internal string buffer differently, increasing |
| it exponentially when needed. This results in the benchmark tests in |
| :file:`Lib/test/test_bufio.py` speeding up considerably (from 57 seconds to 1.7 |
| seconds, according to one measurement). |
| |
| * It's now possible to define class and static methods for a C extension type by |
| setting either the :const:`METH_CLASS` or :const:`METH_STATIC` flags in a |
| method's :ctype:`PyMethodDef` structure. |
| |
| * Python now includes a copy of the Expat XML parser's source code, removing any |
| dependence on a system version or local installation of Expat. |
| |
| * If you dynamically allocate type objects in your extension, you should be |
| aware of a change in the rules relating to the :attr:`__module__` and |
| :attr:`__name__` attributes. In summary, you will want to ensure the type's |
| dictionary contains a ``'__module__'`` key; making the module name the part of |
| the type name leading up to the final period will no longer have the desired |
| effect. For more detail, read the API reference documentation or the source. |
| |
| .. ====================================================================== |
| |
| |
| Port-Specific Changes |
| --------------------- |
| |
| Support for a port to IBM's OS/2 using the EMX runtime environment was merged |
| into the main Python source tree. EMX is a POSIX emulation layer over the OS/2 |
| system APIs. The Python port for EMX tries to support all the POSIX-like |
| capability exposed by the EMX runtime, and mostly succeeds; :func:`fork` and |
| :func:`fcntl` are restricted by the limitations of the underlying emulation |
| layer. The standard OS/2 port, which uses IBM's Visual Age compiler, also |
| gained support for case-sensitive import semantics as part of the integration of |
| the EMX port into CVS. (Contributed by Andrew MacIntyre.) |
| |
| On MacOS, most toolbox modules have been weaklinked to improve backward |
| compatibility. This means that modules will no longer fail to load if a single |
| routine is missing on the current OS version. Instead calling the missing |
| routine will raise an exception. (Contributed by Jack Jansen.) |
| |
| The RPM spec files, found in the :file:`Misc/RPM/` directory in the Python |
| source distribution, were updated for 2.3. (Contributed by Sean Reifschneider.) |
| |
| Other new platforms now supported by Python include AtheOS |
| (http://www.atheos.cx/), GNU/Hurd, and OpenVMS. |
| |
| .. ====================================================================== |
| |
| |
| .. _section-other: |
| |
| Other Changes and Fixes |
| ======================= |
| |
| As usual, there were a bunch of other improvements and bugfixes scattered |
| throughout the source tree. A search through the CVS change logs finds there |
| were 523 patches applied and 514 bugs fixed between Python 2.2 and 2.3. Both |
| figures are likely to be underestimates. |
| |
| Some of the more notable changes are: |
| |
| * If the :envvar:`PYTHONINSPECT` environment variable is set, the Python |
| interpreter will enter the interactive prompt after running a Python program, as |
| if Python had been invoked with the :option:`-i` option. The environment |
| variable can be set before running the Python interpreter, or it can be set by |
| the Python program as part of its execution. |
| |
| * The :file:`regrtest.py` script now provides a way to allow "all resources |
| except *foo*." A resource name passed to the :option:`-u` option can now be |
| prefixed with a hyphen (``'-'``) to mean "remove this resource." For example, |
| the option '``-uall,-bsddb``' could be used to enable the use of all resources |
| except ``bsddb``. |
| |
| * The tools used to build the documentation now work under Cygwin as well as |
| Unix. |
| |
| * The ``SET_LINENO`` opcode has been removed. Back in the mists of time, this |
| opcode was needed to produce line numbers in tracebacks and support trace |
| functions (for, e.g., :mod:`pdb`). Since Python 1.5, the line numbers in |
| tracebacks have been computed using a different mechanism that works with |
| "python -O". For Python 2.3 Michael Hudson implemented a similar scheme to |
| determine when to call the trace function, removing the need for ``SET_LINENO`` |
| entirely. |
| |
| It would be difficult to detect any resulting difference from Python code, apart |
| from a slight speed up when Python is run without :option:`-O`. |
| |
| C extensions that access the :attr:`f_lineno` field of frame objects should |
| instead call ``PyCode_Addr2Line(f->f_code, f->f_lasti)``. This will have the |
| added effect of making the code work as desired under "python -O" in earlier |
| versions of Python. |
| |
| A nifty new feature is that trace functions can now assign to the |
| :attr:`f_lineno` attribute of frame objects, changing the line that will be |
| executed next. A ``jump`` command has been added to the :mod:`pdb` debugger |
| taking advantage of this new feature. (Implemented by Richie Hindle.) |
| |
| .. ====================================================================== |
| |
| |
| Porting to Python 2.3 |
| ===================== |
| |
| This section lists previously described changes that may require changes to your |
| code: |
| |
| * :keyword:`yield` is now always a keyword; if it's used as a variable name in |
| your code, a different name must be chosen. |
| |
| * For strings *X* and *Y*, ``X in Y`` now works if *X* is more than one |
| character long. |
| |
| * The :func:`int` type constructor will now return a long integer instead of |
| raising an :exc:`OverflowError` when a string or floating-point number is too |
| large to fit into an integer. |
| |
| * If you have Unicode strings that contain 8-bit characters, you must declare |
| the file's encoding (UTF-8, Latin-1, or whatever) by adding a comment to the top |
| of the file. See section :ref:`section-encodings` for more information. |
| |
| * Calling Tcl methods through :mod:`_tkinter` no longer returns only strings. |
| Instead, if Tcl returns other objects those objects are converted to their |
| Python equivalent, if one exists, or wrapped with a :class:`_tkinter.Tcl_Obj` |
| object if no Python equivalent exists. |
| |
| * Large octal and hex literals such as ``0xffffffff`` now trigger a |
| :exc:`FutureWarning`. Currently they're stored as 32-bit numbers and result in a |
| negative value, but in Python 2.4 they'll become positive long integers. |
| |
| There are a few ways to fix this warning. If you really need a positive number, |
| just add an ``L`` to the end of the literal. If you're trying to get a 32-bit |
| integer with low bits set and have previously used an expression such as ``~(1 |
| << 31)``, it's probably clearest to start with all bits set and clear the |
| desired upper bits. For example, to clear just the top bit (bit 31), you could |
| write ``0xffffffffL &~(1L<<31)``. |
| |
| * You can no longer disable assertions by assigning to ``__debug__``. |
| |
| * The Distutils :func:`setup` function has gained various new keyword arguments |
| such as *depends*. Old versions of the Distutils will abort if passed unknown |
| keywords. A solution is to check for the presence of the new |
| :func:`get_distutil_options` function in your :file:`setup.py` and only uses the |
| new keywords with a version of the Distutils that supports them:: |
| |
| from distutils import core |
| |
| kw = {'sources': 'foo.c', ...} |
| if hasattr(core, 'get_distutil_options'): |
| kw['depends'] = ['foo.h'] |
| ext = Extension(**kw) |
| |
| * Using ``None`` as a variable name will now result in a :exc:`SyntaxWarning` |
| warning. |
| |
| * Names of extension types defined by the modules included with Python now |
| contain the module and a ``'.'`` in front of the type name. |
| |
| .. ====================================================================== |
| |
| |
| .. _23acks: |
| |
| Acknowledgements |
| ================ |
| |
| The author would like to thank the following people for offering suggestions, |
| corrections and assistance with various drafts of this article: Jeff Bauer, |
| Simon Brunning, Brett Cannon, Michael Chermside, Andrew Dalke, Scott David |
| Daniels, Fred L. Drake, Jr., David Fraser, Kelly Gerber, Raymond Hettinger, |
| Michael Hudson, Chris Lambert, Detlef Lannert, Martin von Löwis, Andrew |
| MacIntyre, Lalo Martins, Chad Netzer, Gustavo Niemeyer, Neal Norwitz, Hans |
| Nowak, Chris Reedy, Francesco Ricciardi, Vinay Sajip, Neil Schemenauer, Roman |
| Suzi, Jason Tishler, Just van Rossum. |
| |
